Multiple multiple comparisons

No I am not perseverating; this post is about something not solely neonatal, but very important to clinical research. Recently a psychology investigator became intrigued that many studies of functional MRI produced extremely high correlation coefficients, that is, the relation between brain activity in the region of interest and the task the subjects were set (such as choosing something) was very frequently enormously high.

The relevance to neonatology is of course that fMRI is increasingly used to show how different premie brains are to the rest of humanity.

So this investigator examined how the data being produced were being analyzed (Vul E, Harris C, Winkielman P, Pashler H: Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on psychological science 2009, 4(3):274-290). It turns out that many studies look at brain activity in a particular 3-dimensional place ‘a voxel’ (volumetric pixel) and look at how it changes with a particular task compared to baseline. All well and good, except that depending on the resolution of the study here may be as many as 500,000 voxels to look at.

What many studies have done is to look initially at all of the voxels, then to select only those that have a greater than average change in activity. After that they produce a correlation coefficient based solely on those voxels that showed a positive correlation, the data from the others being deleted. The authors of this investigation realized that this could artificially inflate, or even produce, significant correlations when there were not real correlations. They went as far as to artificially produce a completely random data set and then did exactly the same thing to that data set as other investigators do to real fMRI data. Not too surprisingly they showed that selecting significant data, and then analyzing only those data that you have already decided are significant, is extremely biased, and can produce significant results from random data.

This has been followed by a really interesting study (Bennett CM, Baird AA, Miller MB, Wolford GL: Neural correlates of interspecies perspective taking in the post-mortem atlantic salmon: an argument for proper multiple comparisons correction. Journal of Serendipitous and Unexpected Results 2011, 1:1-5). The authors took a dead salmon. They then performed fMRI of the salmon and… well I will let the authors explain ‘The [dead] salmon was shown a series of photographs depicting human individuals in social situations with a specified emotional valence, either socially inclusive or socially exclusive. The salmon was asked to determine which emotion the individual in the photo must have been experiencing.’ They then repeated the fMRI, they analyzed the result using standard methods, which showed that the dead salmon was very sensitive to human social stress. Now I must admit that, although the study was published in a peer reviewed journal (the journal of serendipitous and unexpected results jsur.org) this is the only paper that this journal has ever published, even though their website suggests that they have been preparing their first issue for about 3 years now.

What this means is that the fMRI results as analyzed in a large number of papers is questionable. Even if the data are reliable, the analysis can produce highly significant results if they are analyzed inappropriately. fMRI has been used in follow up studies in several studies of preterm infants, I don’t know if these inappropriate techniques of analysis were implicated, partly because as the authors of the first study note, the description of the methods is often very sketchy, so you wouldn’t necessarily realize what had been done. The other thing that it might mean is that the next time you eat salmon, it may understand how you feel, especially if it is sushi…

About Keith Barrington

I am a neonatologist and clinical researcher at Sainte Justine University Health Center in Montréal
This entry was posted in Neonatal Research. Bookmark the permalink.

1 Response to Multiple multiple comparisons

  1. I LOVE IT! I love salmon too, but this may be the best use I have ever seen beyond Lox.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.