BMJ Reporting Bias Study
2022 study shows levels of reporting bias, which can lead to over-estimation of benefits of treatments, are lower in homeopathy research than in conventional medical research.
A study in BMJ Evidence-Based Medicine1 published in March 2022 highlights a well-known problem affecting all areas of medical research known as ‘reporting bias’, which can distort the overall evidence for how well a medical treatment works, usually over-estimating the benefits.
Media reports based on this study, claiming that the benefits of homeopathy have been ‘substantially over-estimated’, fail to mention the authors’ own statement that reporting bias is “not a phenomenon that is limited to homeopathy” – it occurs in all areas of clinical research.
However, by choosing to focus solely on whether this problem affects the homeopathy evidence base, this new study has provided important new insights, showing that the homeopathy research sector appears to be out-performing conventional medicine in regard to scientific and ethical standards, with lower levels of reporting bias.
Important examples of reporting bias include ‘publication bias’ i.e. not all studies are published, with positive studies being more likely to be published than negative studies. Another bias is changing the ‘primary outcome’ i.e. which symptom/measurement (out of several being monitored during a study) should be considered the main ‘result’ for determining whether the drug being tested is effective or not.
The authors state that, “non-publication of trial results and selective outcome reporting …. is not a phenomenon that is limited to homeopathy”, yet they failed to provide adequate context for their results by making any direct comparisons to other areas of clinical research.
According to previous studies published in the BMJ which looked at reporting bias in all medical fields:
- Half of all registered clinical trials2 in conventional medicine fail to report their results within a 12 month period; whereas 62% of all registered homeopathy trials reach publication.
- Inconsistencies in reporting of primary outcome3 occur in 43% of conventional medical studies; whilst this occurs in only 25% of published homeopathy trials.
The work of Dr Robert Mathie, an HRI-affiliated researcher, was recognised as providing “landmark systematic reviews”4,5 of trials assessing the effects of homeopathic treatment, used as the starting point for the BMJ analysis. The potential impact of unregistered/unpublished results on estimates of treatment effects is well known,6 yet for homeopathy, according to Gartlehner et al, the impact may be minimal, or nothing at all: “the difference in effect sizes between registered and unregistered studies did not reach statistical significance”.
It is therefore surprising to see the authors claim that Dr Mathie’s landmark meta-analyses “might substantially overestimate the true treatment effect of homeopathic remedies and need to be interpreted cautiously”. Fortunately, they qualified their findings with the word “might”, as a thorough examination of their study reveals that this claim is not supported by their data.
Dr Mathie’s reviews,7 which provide the most rigorous summary of clinical trial evidence in homeopathy to date, remain sound: conclusions regarding the size of the beneficial effects of homeopathic treatment were based on only the highest quality studies, which had a low risk of bias. As most of the non-registered trials have high risk of bias, they would not have influenced these reviews’ conclusions.
While attempts have clearly been made to use this new study to undermine the evidence base in homeopathy, claiming ‘poor research practice’,8 such claims are entirely unfounded.
As Dr Alexander Tournier, HRI Executive Director explains, “Reporting bias is a well-recognised issue in all areas of medical research, so it is unsurprising that it occurs in homeopathy research. The most interesting finding from this new study, published in ‘BMJ Evidence Based Medicine’, is that we now know homeopathy is out-performing conventional medicine in this respect, with lower levels of reporting bias.”
HRI is committed to promoting and supporting the highest quality of research: we therefore support all efforts being made to reduce reporting bias in clinical research to more accurately inform healthcare choices. In the meantime, it is reassuring for patients, decision-makers and scientists to now know that, contrary to these authors’ claims, the clinical evidence base in homeopathy does not need more “cautious interpretation” than any other scientific evidence.
