There are many ways that an article in the medical literature may not be worthy of trust, and whose results may be unreliable. I remember one of my own trials, LACUNA, which was a pilot RCT of lactoferrin supplementation to prevent late-onset sepsis (LOS). As a pilot. the primary outcome was feasibility of a larger trial, but the clinical outcomes that I reported included the incidence of culture-positive sepsis. I calculated that as planned, as the proportion of babies who had at least one episode, out of the total number included in each group.

When I looked at the data, I realized that if I presented the results as infections per 100 patient days, there was actually a statistically significant difference between groups! I must admit to being tempted to report that outcome, as if it had been the primary outcome of my trial. I think the trial would have had a greater impact than it actually did, and could potentially have led to widespread adoption of lactoferrin as prophylaxis against late-onset neonatal sepsis. I could have been famous!

I am glad I resisted the temptation, because I know that it is vitally important to follow the trial design, outcomes, and statistical analysis that are planned prior to the trial. Any new analysis, or redefinition of outcomes, which is decided after examining the data, is inherently, extremely unreliable.

If I had reported that outcome, probably few people would have realized that I had changed the definition after performing the trial. The trial was pre-registered, but few people look at the registration documents, or compare the reported outcomes to those which were pre-planned. That includes peer reviewers, and meta-analysers.

It tuns out, of course, that the lactoferrin preparation that we used almost certainly has no real impact on LOS, and if I had reported the other outcome, it would eventually have been proven to be a red herring.

I was stimulated to think about this issue, which reminded me of the LACUNA outcomes, by a post from the Health Nerd https://gidmk.substack.com/p/the-terrifying-problem-of-fraud in women’s health. It is a really interesting blog, in general, and this post is indeed terrifying. He refers to several Cochrane reviews, including 2 that have major neonatal implications. He starts by discussing an influential Cochrane review of vitamin D supplements in pregnancy, that, in previous versions, was very positive, and led to recommendations from various health groups to give such supplements. He notes that the new version of the review has removed 21 articles because of trustworthiness issues. Twenty-one! The review now shows no benefit, and, in fact, potential harms.

A review with major neonatal impacts is the Cochrane review of steroids prior to Caesarian Delivery at term. The previous version included 4 trials, with nearly 4000 babies, and showed some significant benefits. The new version has removed 3 of those trials, for being untrustworthy, and now only shows a possible reduction in NICU admission among the 900 babies included in the 1 remaining trial, but not in RDS, or respiratory support. That single included trial did not report hypoglycaemia, which other trials, in late preterm infants, have shown to be a major potential complication of steroids in that population. Based on this new Cochrane review, I think such use of steroids is highly questionable.

Another Cochrane review from the obstetric group with a neonatal impact is their review of sildenafil and other NO pathway medications for fetal growth restriction. The initial search identified 19 studies, but 11 had to be excluded because of trustworthiness issues. Even one of those included was not prospectively registered, so they performed a sensitivity analysis excluding those data. Only sildenafil had enough data to be informative, and the review shows no benefit.

It is now essential that systematic review authors evaluate the trustworthiness of the trials being included. Almost all journals state that they follow the ICJME guidelines, which preclude the publication of trials which are not pre-registered. On many occasions however, I have seen articles published that are reports of clinical trials, but do not mention registration, or were retrospectively registered. Systematic reviews often include data from those trials without commenting on the issue. There should always be a sensitivity analysis excluding unregistered or retrospectively registered trials in a SR. The SR of erythopoietin for prevention of NEC, that I discuss in the post linked below, went from showing a reduction in NEC with erythropoietin, to a null effect when the retrospectively registered trials were deleted.

Just like my example above, untrustworthy research is not necessarily fraudulent. It may be a minor twist to the definition of the outcome, or deciding post hoc, that a certain observation is an outlier, or changing the analysis from treating a value as a dichotomous to a continuous outcome. Sometimes, of course, there is overt fraud, with entirely imaginary numbers. The pressure on individuals in some countries to produce published research is so enormous, that there is no consideration of the downstream harms that can be caused by such fraudulent results. Or by results which are “tweaked” to show a benefit.

The problem of untrustworthy published research is not new, but I fear that it will continue to grow, with articles now being generated entirely by AI. The least we can do is to demand that all published RCTs are publicly registered, in a database including projected sample size, eligibility criteria, and primary outcomes.

Those of us performing systematic reviews must now perform trustworthiness assessments, using one of the published tools. It substantially increases the workload involved, but will also improve the reliability of the results.