Public citizen are at it again, they have written another letter to the director of the DHHSS to try and follow up on their previous nonsense. I didn’t write about this when I first heard about it, as 1. it is dumb and 2. I didn’t think anyone will take any notice of more misguided criticism of SUPPORT. But with a very interesting and thoughtful viewpoint article in JAMA just published, about comparative effectiveness research, I thought I would come back to the issues briefly. 

In this new missive to Kathleen Sebelius they introduce more ridiculous arguments, criticizing the SUPPORT trial for study design lapses, inadequate oversight and failure to stop the study early.

In this new letter they state for example that oxygen saturation targets should normally be adjusted according to capillary filling time, hepatic function, intravascular volume and impending NEC (among other indications). As far as I can see they have just made these things up.

Let me make this clear in case anyone from Public Citizen is reading this blog.

THIS IS CRAP.

We do not do this. We do NOT adjust saturation targets according to capillary filling, hepatic function or intravascular volume it doesn’t make any sort of physiologic sense, and if anyone in my NICU did this we would have a very serious discussion. We do NOT adjust saturation targets in the case of non-existent diagnoses either, such as ‘impending NEC’ even if we knew when NEC was impending we have no evidence from any sort of trial (and certainly not from high quality large RCTs such as SUPPORT) that changing saturation limits is appropriate.

So here is my advice for people who want to criticize clinical research in a particular field, if you have no idea what you are talking about, either shut up, or go and ask someone who does know. Otherwise you just make yourself look stupid.

The authors of the letter then make certain points about early termination of the study which show still more clearly that they don’t understand clinical research, or how this sort of study is run.

Most trials, and all large multicenter trials, now have an independent data safety monitoring committee (DSMC). They are set up by the study steering group, but are completely independent of them, they have access to study data as the study is being run. At pre-specified points in the study the are supplied with data from currently enrolled subjects, which is usually supplied as 2 groups of data, 1 from each arm of the study, but without necessarily revealing which arm is which. They then look at the data to see if there are safety concerns, the idea being that if one group is clearly having worse outcomes on an important outcome variable than the other group, then the study can be stopped. This is an important principle to avoid exposing further patients to risks when the answer is already crystal clear.

There are risks to doing this, every time you look at the data there is a chance that you will find a difference just because of random variation, and the more times you look at the data these risks accumulate. You protect against such risks by having a limited number of looks at the data, and by requiring highly significant differences in clinically important outcomes in order to stop a trial.

What the Public Citizen authors claim is that there was no plan to examine retinopathy and death separately, and that if that had been done then the study could have been stopped after 25% enrollment. Which would have spared the remaining infants from the risks of death or severe retinopathy.

Firstly it is highly likely that the DSMC (data safety monitoring committee) did look at components of the primary outcome separately. The DSMC is independent, and is tasked with protecting future enrollees, they are not in any way restricted in what they can examine, and often ask for extra details and clarifications. I can give the example of the CAP trial, where the DSMC noted a difference in PDA ligation rates between the groups, even though that was not at all in their list of outcomes to examine according to the approved protocol.

Secondly the letter writers guess what the retinopathy rates were after 25% of the enrollment, assuming that retinopathy was uniformly distributed amongst patients. In fact that is highly unlikely. If 10% of the patients have an adverse even, it is highly unlikely that every tenth patient will be the one affected, it is equally (un)likely that the first 10 all have the adverse event and the next 90 are fine. That is one reason why much higher levels of statistical significance are required for early stopping of a trial. So according to their calculations there would have been a statistically significant difference in retinopathy between the groups after 25% enrollment. All that statement does is to show their profound ignorance. A p<0.05 level of significance is not used for early stopping of trials; more commonly a 3SD difference is required in an important outcome, equivalent to about p<0.001, for the reasons outlined above.

If that had indeed happened, however, and the study had been stopped as they suggest it should have been, this would, of course, have prevented the investigators from noting the increased mortality in the low saturation group, and the result of the trial would have encouraged everyone to start using lower saturations and causing the death of many preterm babies.

According to the Public Danger (my new pet name for the group) authors, this would not have happened because there was no ‘usual care’ group, and therefore everyone would have ignored the results and carried on as before!!!

Public Danger’s lack of understanding of what constitutes ‘usual care’ in a modern NICU is not just profound, it is persistent. It resembles schizophrenic delusions that persist despite all the evidence to the contrary.

Le me try and explain this again in case any of them are listening, which I doubt. Both arms of the SUPPORT trial were ‘usual care’. It is ‘usual care’ for an NICU to have pre-specified limits of saturations for all the preterm babies in the unit, which are not changed based on clinical factors, but stay unchanged in the face of, for example liver disease or ‘impending NEC’.

Since SUPPORT and the other oxygen trials have been published we, and many other centers, have changed our oxygen saturation limits, we now aim for over 90% for all of our preterm babies (and try to stay below 95%). We may have more retinopathy as a result, but will try to compensate by optimizing nutrition, and investigating other interventions such as perhaps omega3 fatty acids.

It would have been an enormous disaster for preterm babies, in the USA, Canada, and around the world, if the viewpoint of Public Danger had prevailed and we had all set our saturations to less than 90% because the study had terminated early. Thousands of dead babies would not have thanked us for that.