Fiddling while the data burns

Posted on 15 October 2012 by Keith Barrington

Two recent articles highlight one of the adverse consequences of our fixation on p<0.05. It became common during the 20th century to state that if your statistical test shows that the results you found are unlikely to be due to chance, at a level of less than 1 time in 20, then they were significant. The scientific community could easily have decided on 1 chance in 18 or 1 chance in 22, but 1 in 20 is a nice round number on a base 10 number system. (If we had evolved to have 11 fingers it would probably be 1 in 22!)

Anyway when you spend a lot of time and energy doing a research project, and you think you data are great, but the statistical test comes out as p=0.058 what do you do? There is an often unconscious feeling that maybe there was an outlier that you should eliminate, maybe the data should be analyzed by a different test, or you should transform it before doing the test. Low and behold, after several tries at fiddling with the data, the p value is 0.048 and you can use the hallowed word “significant”.

A recently published study provides some evidence that this actually happens. Masicampo EJ, Lalande DR: A peculiar prevalence of p values just below .05. The Quarterly Journal of Experimental Psychology 2012:1-9. They examined the main p-values reported in psychology journals, what they showed was something that shouldn’t happen, that p-values just slightly below 0.05 were more common than they should be.

That little circle just below .05 is way higher than it should be, people have been fiddling with their results! Imagine if we did have 11 fingers, a critical p value of .045 would make all those results non-significant.

Another new publication has suggested a way to show this, the authors reckoned that if people are fiddling, then there should be fewer than expected p values which are just above 0.05, as they will have been “adjusted” downwards. Gadbury GL, Allison DB: Inappropriate fiddling with statistical analyses to obtain a desirable p-value: Tests to detect its presence in published literature. PLoS ONE 2012, 7(10):e46363. So they developed methods to compare the proportions of p-values in different ranges, just above 0.05 and just below 0.1. Unfortunately their method only works on large data sets of p values, and can’t be used for an individual study.

The main reason this sort of thing happens, which distorts the medical literature, is the bias in publication of significant results, and the relative difficulty of publishing results if the p value is a bit too big. It was suggested years ago (Gardner MJ, Altman DG: Confidence intervals rather than p values: Estimation rather than hypothesis testing. Brit Med J 1986, 292:746-750, among others.) that we should stop publishing p values, and instead publish estimates of effect and confidence intervals. I think we could take this further, journals should review only the methods of a paper, if the methodology is sound, they could then commit to publishing, and afterwards see the results and discussion, for editing and formatting, and ensuring that the discussion is appropriate. They could demand that authors present estimates of effect and confidence intervals, and that any calculation of the statistical likelihood that the results are due to chance alone (significance testing) be presented without reference to any particular threshold.

Posted in Neonatal Research | Leave a comment

Weekly updates #10

Posted on 13 October 2012 by Keith Barrington

Vanderbilt DL, Schrager SM, Llanes A, Chmait RH: Prevalence and risk factors of cerebral lesions in neonates after laser surgery for twin-twin transfusion syndrome. American Journal of Obstetrics and Gynecology 2012, 207(4):320.e321-320.e326. The authors followed children born after laser surgery, but only performed CNS imaging in those born before 32 weeks, and those more mature who had symptoms. So although described as a study of prevalence it is only a partial prevalence. There also wasn’t a standard protocol for imaging, so a 36 week infant with an cerebral infarct, who would be unlikely to have symptoms, which is not necessarily the case at all, might only have had an ultrasound, with a poor sensitivity for this kind of lesion. So not surprisingly the major risk factor for a cerebral lesion was prematurity.

Wheeler B, Broadbent R, Reith D: Premedication for neonatal intubation in Australia and New Zealand: A survey of current practice. Journal of Paediatrics and Child Health 2012:online. All Australian NICUs and transport teams give premedication prior to endotracheal intubation. Hooray! 80% of them have a written protocol. Hooray! More than half of them use morphine. Boo! (Morphine is the one opiate shown to not improve physiologic stability during intubation, its onset of action is 10 to 15 minutes, too slow for an acute procedure.)  Come on Aussies, you need to switch to something that works fast enough to reduce pain and improve stability during intubation.

Slater L, Asmerom Y, Boskovic DS, Bahjri K, Plank MS, Angeles KR, Phillips R, Deming D, Ashwal S, Hougland K et al: Procedural Pain and Oxidative Stress in Premature Neonates. The Journal of Pain 2012, 13(6):590-597. The authors measured indicators of oxidative stress before and after removing tape securing a dressing of a central line or arterial catheter. Signs of pain correlated significantly with indicators of oxidative stress.

Mann PC, Cooper ME, Ryckman KK, Comas B, Gili J, Crumley S, Bream EN, Byers HM, Piester T, Schaefer A et al: Polymorphisms in the fetal progesterone receptor and a calcium-activated potassium channel isoform are associated with preterm birth in an Argentinian population. J Perinatol 2012. We know that progesterone therapy reduces recurrence of preterm birth, this study suggests that prematurity is associated with an increased risk of having particular isoforms of the progesterone receptor, as well as a potassium channel. Also ethnic differences in these associations.

Grobman WA, Thom EA, Spong CY, Iams JD, Saade GR, Mercer BM, Tita ATN, Rouse DJ, Sorokin Y, Wapner RJ et al: 17 alpha-hydroxyprogesterone caproate to prevent prematurity in nulliparas with cervical length less than 30 mm. American journal of obstetrics and gynecology 2012. In this particular high-risk group without a previous delivery, (placebo controlled randomized trial in 660 pregnant women) progesterone didn’t work. About 25% had a preterm delivery in each group, confirming that they were indeed high risk.

Posted in Neonatal Research | Leave a comment

The Benefits of Chocolate (not quite neonatology, but relevant in the middle of a night on call)

Posted on 12 October 2012 by Keith Barrington

A new publication in the Prestigious New England Journal of Medicine (that is how the news media always refer to it, I think they should change the abbreviation to PNEJM) Messerli FH: Chocolate consumption, cognitive function, and nobel laureates. New England Journal of Medicine, online first October 10, 2012,  shows that chocolate consumption per capita is very strongly correlated with number of Nobel Prize winners as expressed as a proportion of the population. The correlation is much better than you would usually find in an epidemiology publication. While the authors are appropriately cautious about implying causality, they do suggest that either chocolate makes you smarter, or that smarter people choose chocolate. Readily testable hypotheses, with large simple RCTs. I just don’t want to be randomized to placebo, thanks.

Posted in Neonatal Research | 1 Comment

Fresh! New! but not any better.

Posted on 11 October 2012 by Keith Barrington

A new multicenter RCT in preterm infants has just been published. The ARIPI trial which was led by Dean Fergusson from Ottawa and included our NICU as one of the sites (Christian Lachance was our local lead investigator and is a co-author). ARIPI stands for Age of Red cells In Preterm Infants. The study was a response to many observational publications in older children and adults, that showed worse outcomes in intensive care patients when they receive older red blood cell transfusions (we rarely give actual blood any more, blood cells are divided up before we transfuse, so what you might call a blood transfusion is a suspension of red cells in a mix of plasma and some special solution to preserve the red cell function and stop the blood from clotting) compared to if they receive freshly donated red cells.

The trial compared what has become the norm in NICU; which is to give the first transfusion that a baby needs from a relatively freshly donated unit of red cells, then to give subsequent transfusions from the same unit until it is very old (often as much as 42 days); the investigators compared this to giving relatively fresh blood for every transfusion.

Very old red cells look weird under the microscope (see image below taken from an article in ‘Transfusion’ in 2006) and also don’t work very well as oxygen transporters. So the idea in older patients has been that these weird looking red cells actually induce inflammation without having a real benefit, But all of the adult and older child data is observational stuff, not a single prospective RCT. The idea was that giving fresh blood each time might improve all of those diseases that are related to inflammation and reactive oxygen species, but would have the downside of increasing donor exposure. I don’t know if anyone non-medical will have read this far 😉 but each new donor increases the risk of a viral infection such as hepatitis or HIV. But, and it is a big but, blood transfusions, or should I say red cell transfusions, are now incredibly safe. The risk of HIV is less than 1 in 1,000,000 transfusions. Blood transfusions are one of the safest things that we do in medicine, (despite the hysteria generated by politicians that means we now, in Canada, have to get individual signed informed parental consent on government approved forms prior to giving any kind of blood product) much safer than other routine therapy such as giving antibiotics, or intubating patients….

Anyway, to return to the new study, the idea was that the fresh blood arm might well have fewer complications than the ‘old’ blood arm of the study, and that this might overcome the very tiny extra risks of being exposed to more donors. The study was powered to show a 15% difference in a combined outcome of bad stuff death or one of 4 major morbidities : bronchopulmonary dysplasia, retinopathy of prematurity stage 3 or more, necrotizing enterocolitis stage 2 or more, and intraventricular hemorrhage grade 3 or 4.

What they found was…. (drum roll)…. nothing. That is right not any whisper of a difference in outcomes between the groups. The primary outcome was found in 52.9% versus 52.7%, BPD was as you would expect the commonest of the components of the primary outcome, 33% vs 32%, IVH was slightly higher, and retinopathy slightly lower with fresh red cells, neither remotely significant. Now you could argue that the study was underpowered. That a sample size of 377 infants doesn’t give you enough power to say with confidence that there is no effect. That is true, but we can say with confidence that there is no very large difference in outcomes, and that there is no difference in outcomes anywhere near as large as the observational studies suggested. You could also criticize the composite primary outcome variable, as lumping together a number of things with different pathophysiology that could go wrong with a preterm baby. But the fact remains that this was a very difficult study to get organized (but once the blood banks were interested, the logistics were easier than I thought they were going to be), and a difficult study to design and get approved and get funded.

So congratulations to the investigators. Now we know that our standard practice is not substantially worse than giving fresh red blood cells; even though we still don’t really know when to give transfusions! The study does show, yet again, that observational data are of very limited value, they can give you a pointer as to what we should study, but then the studies that we need to direct therapy, are large RCTs. Preferably really large. simple, multinational, RCTs.

Posted in Neonatal Research | Tagged , | Leave a comment

Probiotics: enough is enough

Posted on 9 October 2012 by Keith Barrington

At the EAPS meeting in Istanbul, the Proprems trial results were just presented. Now this has not yet been subject to full peer-review, but as a report of a large multicenter RCT it is not likely that their main findings will change. The study was powered for a reduction in late-onset sepsis, which I would maintain was the wrong outcome, we already had good data that there is no effect of probiotics on this outcome from the 20 previous trials. However that was the decision of the investigators, so they had close to 550 babies per group: 1. placebo, 2. A mix of two Bifidobacteria,infantis and lactis, and Strep thermophilus.

The investigators had less definite NEC (grade 2 or more) than expected. 4.4% in the placebo group. Nevertheless they found even less NEC in the probiotic group (2.2%). Now I don’t have all the details or the exact sample sizes, but that reduction is pretty close to being significant at 0.05, and to be honest if it is 0.051 or 0.049 doesn’t matter a ha’penny (as an Mancunian might say).

The effect size of the Proprems trial on NEC, expressed as a relative risk, is well within the confidence limits of our expectations from prior systematic review (0.5 compared to 0.4; CI 0.29 to 0.55).  Now I don’t have all of the info about the trial and I don’t have the exact numbers, but slightly different exact numbers have almost no effect on the conclusions of an updated meta-analysis; the Proprems trial serves to narrow the confidence interval of the meta-analysis slightly. Shown below is an updated Forest plot using all of the available studies (Girish Deshpande is already working on doing this correctly, and most likely Khalid AlFaleh as well) but this is a very quick initial run at it, don’t quote me!Probiotics dec 2012

One of the interesting things that will no doubt be addressed by the authors is why the control group rate of NEC was so low? The ANZNN database reports previously showed that infants under 28 weeks (to be in Proprems you had to be under 1500 g and under 32 weeks) had a NEC incidence much higher than that, at around 10%. So either a lower risk group was enrolled, which is a possibility, or some other study characteristic decreased NEC in both groups. My first guess is that to be in Proprems you had to entered before 72 hours of age, so maybe infants who were more unstable and at higher risk tended not to be approached or did not consent.

In any case, one of the endless criticisms of previous trials was that they were in high risk groups. The new trial shows that probiotics work in a low risk situation also.

What now? All potential participants in other placebo controlled trials should be informed, MUST be informed, of this information. In a non-biased manner.

Are there still physicians who could honestly continue to randomize children to placebo? It has been many years since I would have been prepared to do so, and we have been giving probiotics routinely in my NICU now for 16 months. Our results, a prospective cohort study, are being prepared for presentation and publication as I speak, but they are not inconsistent with the systematic reviews.

Importantly NEC still exists, we might reduce it by about half with probiotics, but some babies, too many babies, still get this devastating disease. In fact it seems that the cases that we still get occur earlier than the mean age in the past (early analysis of my cohort study, to be confirmed), early onset NEC may not be affected by probiotics, perhaps the pathophysiology is different. Or maybe we need to be giving probiotics to the mothers before delivery to try and ensure normal early gut colonization, rather than waiting till they start feeds (which is our current policy) when they already have an abnormal microbiome, and it may take a while to replace the nasty little buggers with good friendly germs.

As for on-going and future studies? Enough is enough! We have more than enough data to stop randomizing babies to placebo. We need to start doing other studies to answer the important remaining questions:

  • Is one probiotic organism better than another?
  • Are 2 germs better than 1?
  • Do prebiotics make a difference? Giving probiotics with oligosaccharides that promote their growth might make them more effective, and help colonization to be more rapid.
  • Does the combination of lactoferrin and probiotics further enhance gut protection, and also reduce nosocomial sepsis, compared to probiotics alone?

Answering these questions will need very large trials, but current models of performing large trials make them very expensive, 3 million dollars for a trial of 1000 babies would not be unusual, our HIP trial of hypotension treatment in the preterm will cost about 5.6 million Euros for 800 babies. So we need to set up large permanent neonatal research networks that can perform trials that are simpler than in the past, on an ongoing basis, that are multinational, not just multicenter, that are freed from some of the excessive regulation and constraints that currently impede important research questions from being answered, but that still have mechanisms in place to protect babies from unexpected harms.

I can dream, can’t I?

Posted in Neonatal Research | Tagged , , | 2 Comments

The Nobel Prize in Physiology or Medicine

Posted on 8 October 2012 by Keith Barrington

Alfred Nobel made his fortune, which he used to endow the Nobel prizes, from dynamite. He invented a way to stabilize nitroglycerine, which is very unstable and tends to explode unexpectedly. Towards the end of his life he was prescribed nitroglycerine as a treatment for his angina, which he found rather ironic; as he noted in a letter to a friend ‘

“It sounds like the irony of fate that I have been prescribed nitroglycerine internally. They have named it Trinitrin in order not to upset pharmacists and the public.Your affectionate friend,

A. Nobel

Nitroglycerine works by releasing nitric oxide in the tissues, so when Furchgott, Ignarro and Murad won the Nobel prize in 1998 for the discovery of the importance of nitric oxide in blood vessels and, among other things, how nitroglycerine helps angina by releasing nitric oxide, the irony was compounded.

The newest Nobel prize doesn’t have that same resonance for me, I use nitric oxide for my patients, but I don’t know if I will ever use pluripotent stem cells derived from adult cells. Nevertheless one thing which is impressive is how fast an insititution can use the announcement of a Nobel prize to set up a new website! The same day as the announcement of the prize a web page has sprung up, from the insitution of one of the winners, I guess there must be money in there somewhere. The only other comment I have is that the other winner, John Gurdon from Cambridge UK has really bad hair.

Posted in Neonatal Research | Leave a comment

Saving Babies’ Lives, for Pennies a day.

Posted on 8 October 2012 by Keith Barrington

I am talking about Canada, not the third world!

Probiotics are cheap and reduce mortality.

Florababy(TM) costs about $25 per 60 gram tub.

One tub is enough for 120 days of prophylaxis, that is enough for the entire hospitalization of an extremely preterm infant.

Among babes who survive the first few days of life, the common causes of delayed death are sepsis, severe respiratory failure, and Necrotizing Enterocolitis. Probiotics decrease NEC frequency by about 60%. Probiotics decrease  delayed deaths by nearly half. For about 20 cents per day per baby.

(I have no conflicts of Interest: I have received no money or rewards of any kind from pharmaceutical or probiotic companies).

Posted in Neonatal Research | Tagged , , | Leave a comment

Not neonatology: Brian Cox

Posted on 7 October 2012 by Keith Barrington

This week an American congressman, member of their science committee, stated that the earth was about 9000 years old, and that the big bang was a lie from the pits of hell. That might not touch me directly, but clearly he should not be in a position of authority over scientific matters.

One perfect response to him would be to make him watch, over and over again the acceptance speech of Brian Cox for the medal of the Institute of Physics in the UK. His speech is a wonderful brief description of the nature of science and his closing quote from Humphrey Davy is great. “Nothing is so fatal to the progress of the human mind as to suppose that our views of science are ultimate; that there are no mysteries in nature; that our triumphs are complete, and that there are no new worlds to conquer.” Brian Cox and the nature of science

I an a great fan of Brian Cox, a physicist from the University of Manchester. He is a wonderful communicator, he gives lectures with a clarity and a fluency that I can only envy, and in addition he has a strong Manchester accent, which is an added benefit (I still have a weak Manchester accent, more than 48 years after leaving the area). For a wonderful brief introduction to the recent history of physics and why it is important, you could do much worse than watching Brian Cox and the history of the universe.

Posted in Neonatal Research | 1 Comment

Who benefits from whole genome sequencing in the NICU? Who suffers?

Posted on 7 October 2012 by Keith Barrington

Annie and I recently published a letter to the editor of Pediatrics and Child Health about the ethical implications of using Complete Genome Hybridization (CGH) as the default test for possibly genetically determined disorders in pediatrics. CGH is a very high resolution way of examining the DNA of an individual, and finding abnormalities that do not hybridize to the standard DNA of the chip. Although much higher resolution than a standard high resolution karyotype there are several risks that come along with that increased resolution. Those risks are (as enumerated in our published letter):

  • CGH often reveals minor DNA anomalies that are of uncertain significance, leaving a child labelled with a DNA anomaly, but with no idea whether it is of medical importance.
  • CGH may reveal significant DNA anomalies that are not related to the condition being investigated; these unrelated DNA anomalies may have implications for the future health of the child.
  • Finding DNA anomalies in the infant will often lead to the parents being tested. This testing may reveal that they also carry the DNA anomaly, which may have implications for the parents’ future health or may impact their insurability.
  • Because CGH is often performed when the diagnosis is uncertain, it is impossible to counsel parents regarding potential implications of the hundreds of possible diagnoses that might follow, or the potential future health impacts.
  • CGH may reveal DNA anomalies that could affect (or will, in the future, be shown to affect) the health status of the infant including risks of cardiovascular disease or cancer risks.
  • CGH may reveal DNA anomalies that are associated with an increased risk of future mental health or behaviour problems, but with what is referred to as ‘reduced penetrance’, ie, they are not very specific. How can we (and do we currently) counsel parents that the CGH test we are performing might reveal that their child has an increased risk of future antisocial behaviour, bipolar disorder and schizophrenia,  (This is not theoretical, this has actually been demonstrated!) but that even if their child has the relevant microdeletion, the occurrence of these problems is not certain, and we cannot do anything to prevent them anyway?
  • CGH results provide a permanent record of DNA anomalies, and the test is performed on an infant who is unable to understand or consent.

Now that is for CGH, imagine the next step up in resolution, sequencing the entire genome.  Some of you may have seen press releases from Children’s Mercy Hospital in Kansas City about whole genome sequencing, available for newborn babies within 3 days. The hype from that center states things like ‘one third of babies in the NICU have genetic disorders’ and ‘physicians can make practical use of diagnostic results to tailor treatments to individual infants and children’

Well I must live on another planet. The large majority of babies in my unit are there for prematurity. Now unless you really want to hype your own research and stretch the meaning of the phrase ‘genetic disorders’ to include possible potential SNP’s in the mother that might in the future be shown to maybe increase the risk that she will deliver prematurely, then that is a downright lie. A lie that will disturb parents and helps no-one. The only people who will be helped by that are those making profits from the test, which costs “only” 13,500$. The entirely spurious comparison of the cost of the test to the cost of neonatal intensive care made during interviews by the authors is equally inane and self-serving.

It is very likely that there are genetic variants that increase the chances that you will develop BPD if you are born prematurely,  or variants that make you more susceptible to RoP or Group B Strep sepsis etc. etc. But that is entirely different to calling them genetic disorders. It is also entirely different from being able to tailor our treatment to the condition. Even in the case of what have previously considered genetic disorders, there are a minority that can be directly treated. The main advantage of a diagnostic label for many children is to be able to give a prognosis.

Lets imagine a case, a very realistic case that will happen one day soon. A baby in the NICU has a club foot. The genetics consultant isn’t sure if there is an underlying genetic cause (even though there was oligohydramnios and the neonatologist was quite happy that the cause was in utero constraint, but nevertheless the resident had filled out a consultation form for the geneticist before I could stop them) so they decide to sequence the genome of the baby. The lab reports 300 different unknown SNP’s none of which have previously been noted to be associated with club foot. They also find that the baby is heterozygous for a beta glucosidase mutation that increases risk of Parkinson’s disease to 16%.

Now what the hell do you do?

As far as we know there is nothing you can do about this. There is no prospect in the near future of being able to do anything about this. So do you keep quiet about it? Do you lie to the parents and say we didn’t find anything? Or do you leave the knowledge hanging over his head for the next 50 years?

If you actually read the article that this hype is based on you will probably be profoundly unimpressed. They actually report 4 babies in the NICU with possible genetic disorders. Three of the babies were critically ill and had active care discontinued, the 4th is having cardiac surgery. For the 3 babies that died 1 had a condition that could have been diagnosed from his clinical presentation, 1 they didn’t figure out, even with whole genome sequencing, one turned out to have an abnormality in a gene only reported once before as a cause of their problems. The baby with heart disease clearly had an autosomal recessive condition, and the authors may have discovered a new gene causing visceral heterotaxy.

So who benefitted from the whole genome sequencing?

None of those babies benefited.  3 of them had care withdrawn because of the severity of their clinical condition, the 4th still needed to have heart surgery.

We read in a tiny footnote to the article the following “J.F., S.H., P.S., Z.K., J.C.W., J.B., R.J.G., E.H.M., and K.P.H. are employees of Illumina Inc., which manufactures the HiSeq 2500 instrument.” The initials refer to 9 of the 23 authors of the article. (That’s right an article about 4 babies with 23 authors!) So Illumina Inc will benefit if they sell their machines.

Children’s Mercy Hospital will benefit if they get more referrals. (in the US medical “system” the more sick babies you get referred the more money you make, even at a wonderful non-profit organisation like the Mercy Hospital a lot is driven by money, because they have to stay open, so they have to keep working on their “market share”).

The authors benefit from having a publication in a high profile journal

Clinical scientific knowledge benefits from knowing a bit more about genes that can cause epilepsy and visceral heterotaxy.

But the babies? The families? There certainly is at least a risk that the babies and families might suffer, they might have findings that are of unknown significance, that cause only concern. They might have findings that are of known significance, but have nothing to do with the reason for doing the test. They might have findings that have profound implications for the future health or future life expectancy of the baby, and they will very often be untreatable.

I certainly wouldn’t have whole genome sequencing, nor accept it for my children. If there was a good research project, I would let them take my blood (or my mouth swab or whatever) but I don’t want to know the results thanks very much! A press release suggesting that this technique makes a difference in clinical management, that it is cost effective, and that it could be applied to 1 in 3 NICU patients serves no-one… except Illumina Inc.

Posted in Neonatal Research | Tagged , | 1 Comment

Not neonatology: the Ness of Brodgar

Posted on 6 October 2012 by Keith Barrington

I just heard about this from an article in the Guardian. A huge neolithic temple complex on the isle of Orkney has been discovered. A site, entirely man-made, covering more than 6 acres, which predates Stonehenge and the Pyramids. The site is just now being excavated, but  includes more than 12 buildings which have been called “temples” and bones of hundreds of cattle with signs of being butchered, perhaps sacrificed. Huge standing stones have been recognized here for a very long time, but the newer excavations have revealed something spectacular, and entirely unsuspected. The site dates from over 5000 years ago and was abandoned about 4,300 years ago.

Interesting to think that we know so little about the distant past of Europe, and that it was probably warmer back then for so much activity to have taken place in Orkney!

Another item to add to my list of places I want to go, sometime soon.

Posted in Neonatal Research | Leave a comment