No. We have been unable to find another review, whether conducted by NHMRC or another research group, which imposed a blanket dismissal of the findings of all trials of N<150 due to having insufficient participants.

This is not surprising as statisticians will confirm that the number of participants needed for a particular trial to be sufficiently ‘powered’ (i.e. have sufficient participants to provide meaningful results) depends on a number of factors unique to that trial; there is therefore no such thing as a set ‘minimum sample size’ for every trial, nor any statistical justification that such a ‘minimum’ be set (arbitrarily) at N>150.

Furthermore, NHMRC routinely funds and collaborates on trials with less than 150 participants. If these trials are all deemed to be inherently unreliable because there are not enough people in them, why are they being funded by NHMRC?

NHMRC state explicitly that trials had to have more than 150 participants to have ‘sufficient participants’ in only two places: Appendix C to the Information Paper (p.35) and Appendix C of the Overview Report Appendices (p.275):

“For the purposes of the homeopathy overview, studies were considered to be of sufficient size where N>150 (i.e. those studies categorised as ‘medium’ sized or larger), as the outcomes were generally continuous outcomes.”

NHMRC presented the results of their review in terms of whether they found ‘reliable evidence’ of the effectiveness of homeopathy, yet from their documents it is far from easy to find out what definition of ‘reliable evidence’ they used.

The ‘Australian Report’ comprises three main documents:

• Information Paper – plain language summary for the public
• Overview Report – full scientific evaluation of the evidence
• Overview Report Appendices – additional background information

In the Information Paper, NHMRC indirectly define a ‘reliable’ study or trial as ‘good-quality, well designed and with enough participants for a meaningful result’ (IP, Findings – Overview, p.6, our underline):

‘There was no reliable evidence from research in humans that homeopathy was effective for treating the range of health conditions considered: no good-quality, well-designed studies with enough participants for a meaningful result reported either that homeopathy caused greater health improvements than placebo, or caused health improvements equal to those of another treatment.’

Information Paper (IP)
Despite the Information Paper being the main document used to present NHMRC’s findings to the public, the number of participants needed for a trial to be ‘reliable’ is only defined in the Appendix to this document (p.35). Although variations on the descriptive phrase ‘enough participants to give a meaningful result’ are used 11 times in the Information Paper, the actual required sample size is not stated.

Overview Report
The Overview Report is the main scientific document, intended to provide technical details about how NHMRC conducted their review. Nowhere in this document do NHMRC clearly and explicitly define ‘sufficient participants for a meaningful result’ in a single trial as n>150.

The sample size threshold is only mentioned indirectly e.g. p.170 Evidence Summary for Allergic Rhinitis, second bullet point:

“Though not the largest study identified in this body of evidence, and while it is below the agreed threshold for a sufficiently sized study, one good quality study with 144 participants (Reilly et al, 1986) reported a significant difference in favour of homeopathy over placebo. However, the findings of this study were not confirmed by other good quality, sufficiently sized studies.”

p.284 5.1 Summary of main results:

“In 36 of the 61 conditions, the total number of participants included in the trial(s) was less than 150; therefore, the evidence base for the majority of clinical conditions was considered of insufficient size to enable clear conclusions on the efficacy of homeopathy to be drawn.”

This latter example has been a source of confusion for many readers of the Report. It refers to the fact that NHMRC used the N>150 threshold for defining the sufficient number of people in a group of trials (when calculating Level of Confidence in the body of evidence for a health condition), as well as using it as the minimum threshold for a single trial to be ‘reliable’ (see FAQ 10 for further details).

Overview Report Appendices
NHMRC explicitly and directly define ‘sufficient participants for a meaningful result’ in the separate Overview Report Appendices document: Appendix C – Criteria for development of evidence statements, Guidance for Element 1: Describing the body of evidence, Point 7, 2nd bullet point (p.275 of 282).

We have been unable to find another review, whether conducted by NHMRC or another research group, which imposed a blanket dismissal of the findings of all trials of N<150 as having insufficient participants to be ‘reliable’.

This is not surprising as statisticians will confirm that the number of participants needed for a particular trial to be sufficiently ‘powered’ (i.e. have sufficient participants to provide meaningful results) depends on a number of factors unique to that trial; there is therefore no statistical justification that such a ‘minimum’ be set (arbitrarily) at N>150.

The reviewers assessed 176 trials as being suitable for inclusion in the study. These are all described and evaluated in the Overview Report, so technically speaking they have been included in the study.

However, the results of all but 5 trials were then completely dismissed on the basis that “These studies are of insufficient quality and/or size to warrant further consideration of their findings” (Overview Report, p.38). This means that in terms of the findings and conclusions of the NHMRC report, all but 5 trials were effectively excluded.

This subtlety has obviously caused some confusion, with a misunderstanding that the 171 trials did not appear in the Australian Report at all.

It is important to note that HRI always refers to trials smaller than 150 participants and/or of ‘insufficient quality’ as being ‘dismissed’ rather than ‘excluded’. This choice of terminology is deliberate because ‘excluded’ has a particular meaning in systematic reviews: excluded studies are not included in the data analysis at all, having been considered unsuitable in some way e.g. in the case of the Homeopathy Review, studies were excluded for reasons such as being the wrong type of trial or written in the wrong language.

Trials that failed NHMRC’s ‘reliability’ criteria were ‘included’ in the sense that they were counted as part of the total number of trials during the ‘adapted GRADE analysis’. However their results were not taken into consideration when drafting the evidence statements, even if the trials showed effectiveness of homeopathy (See FAQ #8).

This is a typical example of the degree of obfuscation within this report.

Children with diarrhoea

The evidence on homeopathy for children with diarrhoea is presented in  Chapter 4.5.2 of the Overview Report (p.36-42). This includes:

• Table summarising all systematic reviews (SRs) which include trials on diarrhoea in children
• A brief description of the SRs and the trials within them
• Reviewer comments
• Evidence statement – this is NHMRC’s overall findings, including a ‘level of confidence’ in the evidence (see FAQ#7) and a final conclusion as to whether there is evidence for the effectiveness of homeopathy this condition
• Table providing full details of the evidence provided by the SRs.

HRI analysis of NHMRC’s handling of this evidence 

The 5 trials which met NHMRC’s definition of ‘reliable’ are:

• Lewith et al. (2002) Double blind placebo-controlled RCT on isopathy for dust mite allergy¹
• Jacobs et al. (2006) Double blind placebo-controlled RCT assessing non-individualised complex remedy for diarrhoea in children.²
• Stam et al. (2001) Randomised, double-blind equivalence trial on homeopathic gel and capsicum-based product for acute low back pain.³ Note: Result of trial was positive
• Grecho (1989) Assessing two homeopathic products for resumption of transit after digestive surgery.⁴ Note: Study in French so according to their protocol it should have been excluded
• Steinsbekk et al. (2005) Double blind placebo-controlled RCT assessing self-treatment (choice of 3 remedies) for prevention of URTIs in children.⁵ Note: Prevention trial so according to their protocol it should have been excluded

References

1. Lewith, G. T., Watkins, A. D., et al. (2002). “Use of ultramolecular potencies of allergen to treat asthmatic people allergic to house dust mite: double blind randomised controlled clinical trial.” BMJ 324(7336): 520.
2. Jacobs, J., Guthrie, B. L., et al. (2006). “Homeopathic combination remedy in the treatment of acute childhood diarrhea in Honduras.” Journal of alternative and complementary medicine 12(8): 723-732
3. Stam, C., Bonnet, M. S., et al. (2001). “The efficacy and safety of a homeopathic gel in the treatment of acute low back pain: a multi-centre, randomised, double-blind comparative clinical trial.” British Homeopathic Journal 90(1): 21-28
4. Grecho (1989). “[Evaluation of 2 homeopathic products on the resumption of transit after digestive surgery. A multicenter controlled trial].” Presse medicale 18(2): 59-62
5. Steinsbekk, A., Bentzen, N., et al. (2005). “Self treatment with one of three self selected, ultramolecular homeopathic medicines for the prevention of upper respiratory tract infections in children. A double-blind randomized placebo controlled trial.” British Journal of Clinical Pharmacology 59(4): 447-455

The list of trials which met NHMRC’s definition of ‘reliable’ is not provided in the Australian Report. This had to be established by HRI’s analysis as follows:

• 176 trials were included in the Homeopathy Review
• NHMRC said that for a trial to be reliable they had to have 150 participants. HRI found that of the 176 trials, only 30 trials met this size threshold (according to the sample sizes in the Overview Report)
• NHMRC said that for a trial to be reliable it also had to have a quality score of 5/5 on the Jadad rating scale (or equivalent on alternative scales). HRI found that of these 30 trials, only 5 trials also met this quality threshold
• Thus only 5 trials met NHMRC’s definition of a reliable trial.

HRI’s data analysis table shows the impact of NHMRC’s definition of a ‘reliable’ trial, something which was not possible to see from their report as this data was not provided.

NHMRC state in their Executive Summary that, “…no good-quality, well-designed studies with enough participants for a meaningful result reported either that homeopathy caused greater health improvements than placebo, or caused health improvements equal to those of another treatment” (Information Paper, p.6).

This is not true. The following studies met their definition of good quality, they are well-designed and had enough participants for a meaningful result (the results are statistically significant).

These studies reported that ‘homeopathy caused greater health improvements than placebo’:

• Two trials showed individualised homeopathic treatment to be better than placebo for treating diarrhoea in children (N=92¹ and N=126²)
• Homeopathic over-the-counter remedy Sinfrontal was found to be better than placebo for sinusitis (N=113) (p<0.0001)³
• Homeopathic combination remedy Pollen C30 was found to be better than placebo for treating hay fever (allergic rhinitis) (N=144) (p=0.02)⁴
• Individualised immunotherapy (using homeopathic medicines made from the substance a patient is allergic to e.g. house dust mite or cat hair) worked better than placebo for hay fever (allergic rhinitis) (N=51)(p=0.0001)⁵
• Homeopathic cough syrup was found to be better than placebo for cough caused by upper respiratory tract infection (URTIs) (N=80)⁶

These studies reported that a homeopathic treatment caused ‘health improvements equal to those of another treatment’:

• Homeopathic over-the-counter remedy Luffa compositum S nasal spray for hay fever (allergic rhinitis) (N=146)⁷
• Homeopathic gel Sprioflor SRL was found to be equally effective as the standard treatment Cremor Capsici Composites for lower back pain (N=161).⁸

It is important to note that the positive findings of these trials are un-refuted by any other trials testing the same treatments for these conditions.

References

1. Jacobs, J., Jiménez, L. M., Gloyd, S. S., Gale, J. L. & Crothers, D. (1994) Treatment of acute childhood diarrhea with homeopathic medicine: a randomized clinical trial in Nicaragua, Pediatrics, 93:719-725
2. Jacobs, J., Malthouse, S. & et al. (1997) Childhood diarrhea: Results from Nepal and combined analysis, Proceedings of the 52nd Congress of LMHL, Seattle, WA
3. Zabolotnyi, D. I. et al. (2007) Efficacy of a complex homeopathic medication (Sinfrontal) in patients with acute maxillary sinusitis: a prospective, randomized, double-blind, placebo-controlled, multicenter clinical trial, Explore (New York, N.Y.), 3:98-109
4. Reilly, D. T., Taylor, M. A., McSharry, C. & Aitchison, T. (1986) Is homoeopathy a placebo response? Controlled trial of homoeopathic potency, with pollen in hayfever as model, Lancet (London, England), 2:881-886
5. Taylor, M. A., Reilly, D., Llewellyn-Jones, R. H., McSharry, C. & Aitchison, T. C. (2000) Randomised controlled trial of homoeopathy versus placebo in perennial allergic rhinitis with overview of four trial series, BMJ (Clinical research ed.), 321:471-476
6. Zanasi, A. et al. (2014) Homeopathic medicine for acute cough in upper respiratory tract infections and acute bronchitis: a randomized, double-blind, placebo-controlled trial, Pulmonary Pharmacology & Therapeutics, 27:102-108
7. Weiser, M., Gegenheimer, L. H. & Klein, P. (1999) A randomized equivalence trial comparing the efficacy and safety of Luffa comp.-Heel nasal spray with cromolyn sodium spray in the treatment of seasonal allergic rhinitis, Forschende Komplementärmedizin, 6:142-148
8. Stam, C., Bonnet, M. S. & van Haselen, R. A. (2001) The efficacy and safety of a homeopathic gel in the treatment of acute low back pain: a multi-centre, randomised, double-blind comparative clinical trial, The British Homoeopathic Journal, 90:21-28

NHMRC have said that they conducted the Homeopathy Review using their usual methods, but this is not the case.

Comparison with other Cochrane Overviews

NHMRC state that they carried out an Overview using a method “derived from Chapter 22 of the Cochrane Handbook of Systematic Reviews of Interventions” (Overview Report, p.12) which describes the ‘Cochrane Overview’ method¹. At the time of writing the Ombudsman Submission (Feb-Aug 2016) we identified 16 published Cochrane Overviews, none of which had a minimum trial size or minimum quality threshold.

Comparison with other NHMRC evidence reviews

Other NHMRC-published evidence reviews have no minimum trial size for reliability. For example, in the Information Paper on The Effects of Lead on Human Health, NHMRC reached conclusions on the effectiveness of a range of interventions based on the results of trials ranging from N=52 to 780.²

NHMRC’s Clinical Practice Guideline for the Management of Borderline Personality Disorder (BPD) ³ presents their conclusions and recommendations regarding the treatment of BPD, based on the results of studies of pharmacological treatments ranging in size from N=16 to 263 and psychological treatments ranging in size from N=19 to 137 ⁴.

Comparison with BMJ Clinical Evidence reviews

The highly respected British Medical Journal (BMJ) Clinical Evidence reviews do have a minimum size threshold, stipulating that randomised controlled trials (RCTs) must include ‘at least 20 people’ (N>20), although “… these criteria vary between reviews depending on the subject area: for example, where little or no evidence is available, we may include RCTs of fewer than 20 people.”⁵

HRI’s analysis shows that if NHMRC had used the BMJ threshold of N=20, the results of only 10 trials would have been dismissed, not the results of 146 trials as was the case with the N=150 trial threshold.

This approach by the BMJ emphasises the primary purpose of Overviews and Clinical Evidence reviews, i.e. to summarise all the available evidence for a topic in its entirety, particularly in areas where only a small amount of research has been carried out. This is in stark contrast to the approach used by the NHMRC with a critical ‘N>150 threshold’ and 5/5 quality score needed for positive trials to be presented to the public in the Information Paper as reliable evidence of effectiveness of homeopathy.

It is therefore clear that certain aspects of the unique methodology created by NHMRC for the Homeopathy Review are unprecedented and without scientific justification.

References

1. Becker, L. A. & Oxman, A. D. (2011) Chapter 22: Overviews of reviews. Cochrane Handbook for Systematic Reviews of Interventions [Link]
2. NHMRC (2015) Section 3. Evaluation of evidence related to exposure to lead : p94–98 [link]
3. NHMRC (2012) Clinical Practice Guideline for the Management of Borderline Personality Disorder. [Link]
4. NHMRC (2012) Appendix H: Clinical Practice Guideline for the Management of Borderline Personality Disorder : Clinical Qs 6, 7 and 9 [Link]
5. BMJ Clinical Evidence (2016) Nuts, bolts, and tiny little screws: how Clinical Evidence works. [Link] (Accessed: 18th February 1016)

‘GRADE analysis’ is a standardised way of defining the quality of a body of evidence in terms of how confident one can be that an estimate of clinical effect detected by the group of studies is accurate¹. GRADE is used to assess how confident we can be that a particular treatment is effective for a specific ‘clinical outcome’ e.g. a symptom such as pain or cough. For this reason, GRADE is usually only used if all the studies in the group measured the effect of the same treatment against the same clinical outcome¹.

GRADE analysis is only performed if the data is suitable for this method; for the majority of health conditions identified in the Homeopathy Review there were not enough trials with the same clinical outcome for GRADE to be a suitable assessment tool. However, instead of taking the appropriate decision not to use GRADE, the Homeopathy Working Committee created their own “adapted version of GRADE … in order to make broad statements about the LOC in the body of evidence as a whole” (Information Paper, p.36).

They used this ‘adapted GRADE analysis’ to assign a Level Of Confidence (LOC) to the body of evidence for trials testing homeopathy against placebo, or another treament, for each health condition.

Although GRADE is designed to be adaptable to various clinical questions and scenarios, there are core principles to this method that must be adhered to – most crucially, applying the process to single outcomes and secondly, applying the method to trials testing the same treatment. The HWC did not adhere to these principles – they applied their novel method to groups of trials covering a range of different clinical outcomes and (in some cases) to trials which tested different homeopathic treatments. This is an unprecedented and inappropriate use of the GRADE approach.

The ‘adapted GRADE’ system created by the HWC primarily used ‘quality’ and ‘precision’ (the two criteria for which there was sufficient information available across the 176 primary studies identified) because “the remaining GRADE factors were difficult to apply” (IP, p.37). This decision led to an exaggerated impact of trial quality and sample size (precision) on the LOC assigned to the body of evidence per clinical condition compared with a normal GRADE assessment; consequently, 66 of 71 bodies of evidence were assessed as having a ‘low’ or ‘very low’ LOC based on these two criteria alone, before any other factors were considered.

Transparency is an essential element of good quality research as it enables other researchers to assess the appropriateness of the approach taken and the validity of the results generated. We would argue that this is even more critical when a totally novel approach has been taken such as HWC’s adapted GRADE assessment of bodies of evidence covering different clinical outcomes and conditions.

Reviews usually provide GRADE tables that show exactly how decisions about the quality of evidence were reached ¹ – a level of transparency that is seen in other NHMRC reviews, e.g. the Information Paper on The Effects of Lead on Human Health ².

NHMRC claim that the Homeopathy Review was conducted ethically and transparently, yet they have not provided their ‘adapted GRADE’ calculations anywhere in their 600 pages of documentation. This means that the reader has to rely entirely on the ‘judgment and expertise’ of the HWC (IP, p.36) and to trust that they have judged the quality of the evidence accurately and fairly.

The unprecedented method used by NHMRC to assign a LOC to the bodies of evidence identified for homeopathy is clearly not a standardised or accepted method; it also lacks transparency and did not pass peer review. The LOC assessments cannot therefore be considered robust.

References

1. GRADE Working Group (2004) Grading quality of evidence and strength of recommendations, BMJ, 328:1490
2.  NHMRC (2015) NHMRC Statement and Information Paper: Evidence on the Effects of Lead on Human Health https://www.nhmrc.gov.au/guidelines-publications/eh58 (Appendix15,p283-4)

It is understandable that there is some confusion about use of N>150 in the Australian Report. A sample size of 150 has been used in two different ways.

Firstly, NHMRC have used it as a threshold when describing the size of trials (Information Paper, p.35):

• < 50 : very small
• 50 to 149: small
• 150 to 499: medium
• 500 to 999: large
• ≥1000: very large.

These trial size descriptions were used when NHMRC performed an ‘adapted’ GRADE analysis to assess the ‘level of confidence’ in their results for each condition (see FAQ #8).

Use of N=150 as the cut-off between a small and medium trial is scientifically fine, as there is no standard definition of a very small, small, medium, large or very large trial. It is therefore usual for each review team to decide how to describe trial sizes in their review.

Secondly, NHMRC used the N>150 size threshold as a threshold for deciding whether a trial was reliable or not. This is a completely different matter. It is scientifically unprecedented and unjustified (see FAQ #1).

NHMRC’s results for each condition were entirely determined by which studies they considered ‘reliable’. The results of all trials with less than 150 participants were dismissed as unreliable, even if the studies were high quality, positive and statistically significant (see FAQ #1). So, NHMRC’s findings were completely distorted by use of this unprecedented rule.

Given the unprecedented use of N>150 as a threshold for trial reliability in the Homeopathy Review, one would expect a clear justification to be given for this rule, but it has neither been explained nor justified anywhere in the report.

When explaining their decision to describe trials with less than N=150 as being ‘small’ or ‘very small’, NHMRC refer to an article in the highly respected journal BMJ.¹ NHMRC said: “HWC [the Homeopathy Working Committee] considered the following study in the development of these thresholds: Influence of trial sample size on treatment effect estimates: meta-epidemiological study.” (Overview Report Appendices, p.274). This implies that their decision to dismiss trials smaller than N=150 as ‘unreliable’ is scientifically justified by this paper. It is not.

Firstly, this paper does not say that the results of trials of any particular size should be dismissed as unreliable. Secondly, NHMRC correctly state that most of the homeopathic trials they assessed use ‘continuous outcomes’, yet the BMJ paper states categorically that its findings cannot be applied to trials of this kind.

References:

1. Dechartres, A., Trinquart, L., Boutron, I. & Ravaud, P. (2013) Influence of trial sample size on treatment effect estimates: meta-epidemiological study, BMJ, 346:f2304