Multiple multiple comparisons

Posted on 2 April 2013 by Keith Barrington

No I am not perseverating; this post is about something not solely neonatal, but very important to clinical research. Recently a psychology investigator became intrigued that many studies of functional MRI produced extremely high correlation coefficients, that is, the relation between brain activity in the region of interest and the task the subjects were set (such as choosing something) was very frequently enormously high.

The relevance to neonatology is of course that fMRI is increasingly used to show how different premie brains are to the rest of humanity.

So this investigator examined how the data being produced were being analyzed (Vul E, Harris C, Winkielman P, Pashler H: Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on psychological science 2009, 4(3):274-290). It turns out that many studies look at brain activity in a particular 3-dimensional place ‘a voxel’ (volumetric pixel) and look at how it changes with a particular task compared to baseline. All well and good, except that depending on the resolution of the study here may be as many as 500,000 voxels to look at.

What many studies have done is to look initially at all of the voxels, then to select only those that have a greater than average change in activity. After that they produce a correlation coefficient based solely on those voxels that showed a positive correlation, the data from the others being deleted. The authors of this investigation realized that this could artificially inflate, or even produce, significant correlations when there were not real correlations. They went as far as to artificially produce a completely random data set and then did exactly the same thing to that data set as other investigators do to real fMRI data. Not too surprisingly they showed that selecting significant data, and then analyzing only those data that you have already decided are significant, is extremely biased, and can produce significant results from random data.

This has been followed by a really interesting study (Bennett CM, Baird AA, Miller MB, Wolford GL: Neural correlates of interspecies perspective taking in the post-mortem atlantic salmon: an argument for proper multiple comparisons correction. Journal of Serendipitous and Unexpected Results 2011, 1:1-5). The authors took a dead salmon. They then performed fMRI of the salmon and… well I will let the authors explain ‘The [dead] salmon was shown a series of photographs depicting human individuals in social situations with a specified emotional valence, either socially inclusive or socially exclusive. The salmon was asked to determine which emotion the individual in the photo must have been experiencing.’ They then repeated the fMRI, they analyzed the result using standard methods, which showed that the dead salmon was very sensitive to human social stress. Now I must admit that, although the study was published in a peer reviewed journal (the journal of serendipitous and unexpected results jsur.org) this is the only paper that this journal has ever published, even though their website suggests that they have been preparing their first issue for about 3 years now.

What this means is that the fMRI results as analyzed in a large number of papers is questionable. Even if the data are reliable, the analysis can produce highly significant results if they are analyzed inappropriately. fMRI has been used in follow up studies in several studies of preterm infants, I don’t know if these inappropriate techniques of analysis were implicated, partly because as the authors of the first study note, the description of the methods is often very sketchy, so you wouldn’t necessarily realize what had been done. The other thing that it might mean is that the next time you eat salmon, it may understand how you feel, especially if it is sushi…

Posted in Neonatal Research | 1 Comment

Weekly Updates #27 or whatever… I’ll have to think of a new title for these posts…

Posted on 28 March 2013 by Keith Barrington

Troger B, Muller T, Faust K, Bendiks M, Bohlmann MK, Thonnissen S, Herting E, Gopel W, Hartel C: Intrauterine Growth Restriction and the Innate Immune System in Preterm Infants of </=32 Weeks Gestation. Neonatology 2012, 103(3):199-204. Infants born with IUGR often have low white blood cell counts, and neutrophil counts, in addition, this study seems to show that there are functional immune deficits also, interleukin levels were lower in the IUGR babies when their blood cell cultures were stimulated, than preterm infants who had normal growth.

An issue of Early Human Development with a section about ‘Psychosocial development of adolescent preterm children’, which includes a review by Saroj Saigal on Quality of life of former premature infants during adolescence and beyond. Early Hum Dev 2013, 89(4):209-213. As you would expect it is excellent. Also if you are at all unsure of the concept of Executive Function, there is a very high quality review from Peter Anderson’s group in Melbourne (Burnett AC, Scratch SE, Anderson PJ: Executive function outcome in preterm adolescents. Early Hum Dev 2013, 89(4):215-220.) This is the group that has done more than any other to study this issue in former preterm infants, and they explain the concepts as well as reviewing the data.

Harvey ME, Nongena P, Gonzalez-Cinca N, Edwards AD, et al: Parents’ experiences of information and communication in the neonatal unit about brain imaging and neurological prognosis: a qualitative study. Acta Paediatrica 2013, 102(4):360-365. Preterm babies (and others in the NICU) often get head ultrasounds, and brain MRI as  a routine. You may remember that Annie and I wrote a commentary recently which bemoaned the fact that no-one had bothered to ask parents is they thought this was a good idea. Well it looks like the group of David Edwards in London is doing that. This nice qualitative study explored the experiences of parents around receiving information about brain imaging. Parents found themselves to be passive recipients of information. I don’t see in this article any questioning of the usefulness of brain imaging for parents, but that may be coming.

Posted in Neonatal Research | Leave a comment

To blindly go where no man….

Posted on 28 March 2013 by Keith Barrington

Do we really need to blind research projects? And if so which parts of them? There is good empirical data that randomization must be masked, it is important that at the moment a patient is enrolled in a study the investigators don’t know which group they are going to be in. That way we can avoid many selection biases.

There is much less good empirical data about the effects of masking the intervention, although it is very likely that when it is possible (and it is not always feasible) masking is a good way to reduce many types of bias. Unfortunately there are sometimes reports of trials as being ‘double-blind’ where it is not always clear what that means.

One thing that can almost always be blinded, even if the intervention can not be blinded, is the evaluation of the outcomes. Measurement of development on standardized scales, for example, can often be performed by blinded individuals, even when the intervention (for example, giving a blood transfusion) can not reasonably be blinded.

A new article in the CMAJ (Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, Ravaud P, Brorson S: Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors. CMAJ 2013, 185(4):E201-211.) compared the outcomes from RCTs where there was both a blinded and an unblinded assessment of the same outcome. There are not many such studies, as you might imagine, but they were able to find 16. They found that, although some studies found no difference, others found substantial effects. and overall there was an exaggerated estimate of the treatment effect by 68%. This is a follow on from another article from the same group that looked at how blinding of dichotomous outcomes are affected (link here) which showed that on average Odds Rations were exaggerated by 36%.

I was once designing a trial of blood transfusion, and I was trying to think of how to blind it, in the end, the idea of having blood hanging by the babies bedside attached to a fake IV pump that didn’t actually work, but still plugged into their IV was abandoned. Quite rightly I think. But we were still able to mask the evaluation of the objective outcome.

A nice introduction to some of the major issues with evaluation of research results is the paper ‘5 ways statistics can fool you—Tips for practicing clinicians‘ by West and Dupras, Vaccine march 2013. Although the discussion is slanted toward studies of vaccines, the ‘tips’ apply everywhere,

(i) consider clinical and statistical significance separately,

(ii) evaluate absolute risks rather than relative risks,

(iii) examine confidence intervals rather than p values,

(iv) use caution when considering isolated significant p values in the setting of multiple testing, and

(v) keep in mind that statistically non-significant results may not exclude clinically important benefits or harms.

Rules for life!

Posted in Neonatal Research | Tagged , | Leave a comment

Breast = best: past your eyes?

Posted on 20 March 2013 by Keith Barrington

The title is from one of those bad punny jokes that you never forget, about taking a bath in milk (pasteurized? No just up to my chin. Boom, boom). For the readers of this blog who are not native English speakers, just ignore this first paragraph and go on to the next.

This is I think a unique study, (Cossey V, Vanhole C, Eerdekens A, Rayyan M, Fieuws S, Schuermans A: Pasteurization of Mother’s Own Milk for Preterm Infants Does Not Reduce the Incidence of Late-Onset Sepsis. Neonatology 2012, 103(3):170-176) the authors from Leuven in Belgium randomized 303 babies (<32 weeks or <1500 g) who were getting maternal breast milk to either get it raw or to have it pasteurized first. This is something which has often been discussed, does pasteurizing milk reduce its anti-infective properties? We have often asked the question because donor milk from milk banks is pasteurized, and we wonder if it will still have the benefits that human milk has before you pasteurize it. We know that pasteurized milk still reduces NEC, as the studies by Alan Lucas from many years ago used pasteurized donor milk in some babies, and they still had the benefits. The data about breast milk and hospital acquired infections aren’t as strong, much less good data in the literature, and the Cochrane review comparing breast milk to formula only includes data from one study, which showed no effect. But we think that human milk feeding probably also reduces some late-onset infections in the preterm baby in an NICU.

We also know that breast milk is not sterile, it is often contaminated with staphylococci, and other organisms which have on occasion been implicated as causing invasive infections. So the authors compared systemic infections between babies that got the pasteurized or raw milk.  The way the title is written is a bit strange, as in the introduction the authors state that they hypothesized that pasteurizing breast milk reduced its benefits as an anti-infective agent, but the title refers to the opposite effect. And, if their hypothesis really was that pasteurization was bad, then there is an ethical issue: why would you test something that is difficult to organize and study, and is not standard practice, if your hypothesis is that it is bad? I think more realistically, that there was real equipoise (which hasn’t been expressed properly), that pasteurizing breast milk may have good effects (reducing the risk of bacterial colonization) and/or bad effects (inactivating immunoglobulins and adversely affecting other important proteins).

The findings of the study were generally negative, that is there was no big difference in any outcome between the groups. What they did show was the there were more infections in the group that received pasteurized milk. A modestly sized increase, compatible with a chance effect, 15% of the babies who received raw milk had at least one infection, compared to 20% of the group who got the ‘past-your-eyes’ stuff. The p value was 0.23. Which is not significant by conventional thresholds, but here again we must question these conventions. Wouldn’t it be more informative for everyone reading this article to state, rather than ‘not significant’ something like the following: ‘there is a 2 in 9 probability that the increase in infections in the pasteurized milk group could have arisen by chance’. Which means in fact, that this ‘non-significant’ result is more likely to be real than not, there is a 7 in 9 chance that the increase in infections is a real difference between the groups.

I am not certainly not saying that this is strong evidence to do anything different, or that we should change practice based on data that have a p value like this, but it is enough evidence for us to think about the issues, and design further, larger, studies to figure out what we should do with breast milk before giving it to our babies.

Of interest the frequency of severe NEC also looks possibly different between the groups, although again, not ‘significant’ (4 cases with pasteurized milk, 0 with raw milk).  Pasteurizing breast milk, as well as killing pathogens, also will inhibit the growth of good bacteria, so we might really be better off to give raw breast milk, as a source of bifidobacteria and lactobacilli. Or maybe we should give pasteurized donor breast milk with extra probiotics, if the mother can’t provide her own milk.

There is another issue with this study which isn’t fully explained, the authors withheld maternal breast milk if the cultures, of the milk, were positive, which sounds like they were doing routine cultures, and then waiting for the results before giving the milk. They don’t describe their protocol for doing this, and we certainly don’t do that. Maternal milk is not routinely cultured in most units in North America. Some babies in the raw milk group received pasteurized milk when the cultures were persistently positive, which is appropriate as part of their research design, but reduces the power of the study.

This study provides no evidence that pasteurizing mother’s milk is beneficial, and suggests that maybe it is better to use raw milk.

Posted in Neonatal Research | Tagged , , , , | 3 Comments

Extremely Preterm Infants, long term outcomes

Posted on 20 March 2013 by Keith Barrington

A new publication examines in great detail the cognitive outcomes of preterm babies less than 29 weeks gestation. The group from Melbourne, the Victorian Infant Collaborative, has published another study. (Hutchinson EA, De Luca CR, Doyle LW, et al: School-age Outcomes of Extremely Preterm or Extremely Low Birth Weight Children. Pediatrics 2013) I don’t know where Lex Doyle gets the time to do everything that he does, but this study is another from an incredibly productive group that examines the outcomes at about 8 years of age for a population cohort of babies (n=189, 94% of the babies were followed up) that were less than 28 weeks or less than 1000g birth weight, and were born in 1997.

The results are extremely encouraging; the full scale IQ of the extremely preterm infants is about 0.8 SD below the term controls, actual academic achievement is about 0.5 SD worse than the term controls, and there are 15% of the babies classified as having major impairment, compared to 3% of controls. In their results they compare the numerical results, and the proportion of infants with impairments between those who are the most extremely preterm, 23-25 weeks, compared to 26-27  weeks. There are no differences between the subgroups.

As the authors rightly say, the rate of ‘neurobehavioral impairment… remains too high relative to controls’ it may be too high, but it is remarkable how well these babies are doing. Despite missing 17 to 13 weeks of intrauterine brain growth and development, the large majority are functioning well within the normal range. We still need to focus on why that period of being outside of the uterus affects brain development, though it would be very interesting to see in this cohort, what are the outcomes of those that did not get serious neonatal complications. If you take out the babies who had a late-onset infection, NEC or needed surgery, I guess you might end up with a group of babies nearly indistinguishable from the term controls.

Posted in Neonatal Research | Tagged | 3 Comments

Prepuce wars

Posted on 19 March 2013 by Keith Barrington

A commentary written by a very long list of authors in response to the AAP’s position on circumcision. (Frisch M, Aigrain Y, Barauskas V, et al: Cultural Bias in the AAP’s 2012 Technical Report and Policy Statement on Male Circumcision. Pediatrics 2013). They accuse the task force of the AAP of having a cultural bias, and as a result of having produced a flawed statement. One of their points is that much of the evidence retained is of limited relevance for the practice of routine infant circumcision.

The response of the task force (Circumcision task force: Cultural Bias and Circumcision: The AAP Task Force on Circumcision Responds. Pediatrics 2013, 131(4):801-804.) is to say ‘we’re not culturally biased, its you that are culturally biased’. And we were right.

I think they are all culturally biased, I don’t think its possible not to have biases. We need to recognize them and try to take them into account. I do think that Frisch et al have a point, the prevention of HIV acquisition and AIDS in randomized trials of adults who agreed to be randomized to be circumcised has no relevance at all for routine neonatal circumcision. We have no idea if the same effect would occur with neonatal circumcision, and, of course, the baby doesn’t have a chance to consent. Similarly the data on penile cancer are all observational studies which were all done before HPV vaccination, which is likely to be much more effective and doesn’t require surgery. I don’t think that many adults would consider having a circumcision to reduce the risk of penile cancer (number needed to treat about 200,000), so to use that argument to support routine neonatal circumcision makes little sense.

Posted in Neonatal Research | 2 Comments

Pacifiers (soothers, dummies) are friendly to babies

Posted on 19 March 2013 by Keith Barrington

The baby friendly initiative has been a force for improving breast-feeding rates, with many sensible parts to the program. Unfortunately some of it isn’t evidence based, such as the prohibition of what I will call soothers. I’m originally from England where they are dummies, in the US I  guess a lot of people call them pacifiers,  in much of Canada they are soothers, in Quebec they are ”suces”…. all of which goes to show they have been around a long time, and many cultures have a familiar name for them.

Step 9 of the standards for being called baby friendly states that you must give no soothers to breastfed babies. I don’t know where this ever came from, there is no evidence that soothers have an adverse effect on breast-feeding success. Now there is a study suggesting the opposite, that they are actually beneficial. Kair LR, Kenron D, Etheredge K, Jaffe AC, Phillipi CA: Pacifier Restriction and Exclusive Breastfeeding. Pediatrics 2013. In this before and after study there was a deterioration in exclusive breastfeeding when they instituted a ban on soothers.

There are actually 4 RCTs that have looked at soother use and a systematic review that confirms no beneficial impact on breast-feeding from trying to reduce pacifier use. Of course doing an RCT of this is rather difficult, anyone can easily buy a pacifier or get one as a gift from a helpful relative. The largest trial was Michael Kramer’s PROBIT, done in Belarus. This was a fascinating cluster randomized trial that is still producing results. One of the comparisons was that mothers who received the educational breast-feeding promotion intervention either got extra advice to not use a soother or they got no such advice. The intervention was partially successful, there was quite a bit less pacifier use in the group who got that advice, but that had no effect on the breast-feeding rates. In case you don’t know the trial, it is among the best evidence we have for the benefits of breast-feeding on reduction of intestinal infections, and prevention of atopic eczema. As well as a brand new publication showing no effect on obesity (Martin RM, Patel R, Kramer MS, et al: Effects of promoting longer-term and exclusive breastfeeding on adiposity and insulin-like growth factor-I at age 11.5 years: a randomized trial. JAMA 2013, 309(10):1005-1013)..

This has some importance in neonatology, as we know that soothers decrease pain during minor invasive procedures such as a heel prick, especially when used together with sucrose. So we routinely use soothers with our preterm babies unless the mother doesn’t want us to. That makes it problematic to receive the baby friendly label. Even though we really are friendly to babies! Really, really friendly.

Posted in Neonatal Research | Tagged , , | 1 Comment

Weekly Updates #0

Posted on 18 March 2013 by Keith Barrington

I know this is out of sequence, and my weekly updates aren’t in any way weekly, but I thought I’d direct anyone who is interested in keeping an eye on general medical issues to Richard Lehman’s journal review. He is where I got the idea of doing a similar thing for neonatology, he publishes a blog each week in which he reviews the main papers in JAMA, PNEJM, BMJ and the Lancet. He is often very rude about the articles, the motivation of the publishers, and the editor of the Lancet. This week is a typical and sometimes scathing example. I get lost with the new drugs that are being tried and tested in adult medicine, like the etanercept studied in the RCT in the Lancet that he mentions, but some of them seem to be major advances on what was available to me 30 years ago the last time I treated an adult. I’m glad I’m not a general family doctor, keeping up with all this stuff must be tough.

Posted in Neonatal Research | Leave a comment

Not neonatology: Brian Cox again

Posted on 15 March 2013 by Keith Barrington

There is an amazing new series on BBC. Animated by Brian Cox again, called ‘Wonders of Life’. If you can figure out how to get access (within the UK it is available to licence holders, outside of the UK you have to make their servers think you are in the UK. I am not going to tell you how to do that, as it might make me liable for something, but if the initials VPN mean anything to you, you might be able to work out how to do it.)¸In the first episode he talks about (and shows some amazing footage of) an animal that I didn’t know about, a jellyfish that has an intracellular parasite that photosynthesizes. So the jellyfish migrates every day in a reproducible pattern, that leads to the greatest exposure of these parasites to light. They even rotate as they swim so that the largest possible number of parasitic algae are exposed to the light.

We know that one of the crucial steps in the evolution of  eukaryotes was the invasion of prokaryotic cells by parasitic organisms that became mitochondria or chloroplasts. I do not have the biology education that I wish I had, so I did not realize that there are many examples of intracellular parasites, similar to what must have happened a few billion years ago in the early evolution of complex life.

You might ask why a physicist is presenting a biology program. I will let Brian Cox himself explain; so go see the programs, if you can.

Posted in Neonatal Research | Leave a comment

New link in side bar

Posted on 15 March 2013 by Keith Barrington

I don’t play with this blog’s appearance very much, I don’t have enough time to fiddle about. But I have just added a new link to the side-bar. A link to a very young foundation, the Canadian Premature Babies Foundation. If you follow the link, you may see a familiar image, in colour this time, of a famous little girls hand, with my wedding ring around the wrist.

This new foundation has my full support, including, but not limited to, the use of the image. If you look at the ‘about us’ page you will see why. The purpose and goals of the foundation are very much in line with what I think we need to do. You can also view a few youtube videos (link here) related to the work of the foundation, including one in English and one in French from the president of the foundation Katharina Staub.

Posted in Neonatal Research | 1 Comment