Methodology

"It is important the Information Paper explicitly explains that the NHMRC review did not include a systematic assessment of RCT evidence. I do not agree that considering random evidence provided by interest groups offsets this deficiency. High quality RCTs with narrow confidence intervals (Level 1 evidence) should have been searched for and included in this review."
- Undisclosed expert reviewer feedback to NHMRC, May 2014
Adoption of arbitrary criteria never used before or since:

The formal investigation into the National Health & Medical Research Council (NHMRC) conduct of the Homeopathy Review has uncovered multiple methodological flaws and irregularities. 

One of the most serious methodological anomalies associated with the Homeopathy Review was the creation of a unique concept of "reliable evidence", which was underpinned by arbitrary criteria never used before or since by any other research group (including NHMRC). 

Further, the framework was developed as part of a systematic process of post-hoc re-invention of the research protocol, which retrospectively dismissed the results of 171 out of the 176 included studies from any consideration in the findings.
  • It is important to note that although these 176 trials were "included" in the Review, the results of 171 of them were dismissed from the Review's findings, on the basis of these pos-hoc changes

Further, none of the changes to the protocol were disclosed (see 'Procedural' section).  

​The NHMRC/HWC determined that for a trial to be considered "reliable", it had to have both:
​
  1. More than 150 participants, which retrospectively dismissed the results of 146/176 studies from contributing to the Review's findings

  2. Have an unusually high 100% quality rating (5/5 on the Jadad or equivalent using another scale), which retrospectively dismissed the results of 25/30 of the remaining studies from contributing to the Review's findings.

No research guidelines - including those of Australia's Therapeutic Goods Administration (TGA) in assessing registered, high risk prescription medicines - recognise such unusually stringent thresholds.

An in-depth scientific analysis conducted by the Homeopathy Research Institute (HRI) has been used as part of a submission of complaint to the Commonwealth Ombudsman, brought by Complementary Medicines Australia (CMA), the Australian Homoeopathic Association (AHA) and the Australian Traditional Medicine Society (ATMS).

The following FAQ link and video summarise some of the key issues identified through the HRI's scientific investigation:
​
"Thresholds for descriptions of trial sizes were determined ... based on the (generally) continuous outcomes measured in the trials. HWC considered the following study in the development of these thresholds ... www.bmj.com/content/346/ bmj.f2304)."
- NHMRC FAQ document, 11 March 2015
NHMRC regularly funds trials with less than 150 participants: 

NHMRC regularly funds and collaborates in trials with far fewer than 150 participants, which are not regarded as "unreliable". 

No published Cochrane Overview (whose guideline NHMRC referenced), or any other evidence review, have ever used sample size thresholds or 'quality rating' scales as a measure of trial "reliability" (noting that in published Cochrane Overviews, a quality rating of 3/5 on the Jadad scale has been regarded as 'good quality', with such quality ratings not applied as thresholds to exclude consideration of evidence).
www.bmj.com/content/346/ bmj.f2304:

​"However, because our results were based on meta-analyses of trials assessing binary outcomes, they cannot be extrapolated to trials assessing continuous outcomes..."
The highly respected British Medical Journal (BMJ) Clinical Evidence reviews do have a minimum size threshold, stipulating that randomised controlled trials (RCTs) must include ‘at least 20 people’ (N>20), although "... these criteria vary between reviews depending on the subject area: for example, where little or no evidence is available, we may include RCTs of fewer than 20 people." [1]

This approach by the BMJ emphasises the primary purpose of Overviews and Clinical Evidence reviews, i.e. to summarise all the available evidence for a topic in its entirety, particularly in areas where only a small amount of research has been carried out. 

NHMRC erroneously cites a BMJ study in attempting to justify the N=150 threshold:

In attempting to justify the N=150 sample size criterion, NHMRC correctly identified the homeopathic studies it assessed as being ‘continuous outcomes studies’ and cited a BMJ paper (www.bmj.com/content/346/bmj.f2304).

This was reiterated in the NHMRC Information Paper, Frequently Asked Question document and the Optum Overview Report, as well as in responses to public correspondence on the issue, as follows:
However, the BMJ paper, which makes no link between trial size and ‘reliability’, states that its results “cannot be extrapolated to trials assessing continuous outcomes”​​:
NHMRC uses this citation of the BMJ study against the N=150 threshold multiple times across the final report documents released to the public, who would not question that an expert body such as NHMRC would make such a fundamental error and/or intentionally publish misleading information. 
​
How the N=150 sample size rule impacted the findings:

Following are some examples of the way in which the arbitrary N=150 sample size exclusion threshold dismissed as "unreliable" the results of positive research evidence from consideration in the Review's findings [2]:
Full details of the NHMRC's procedural and scientific conduct of the Homeopathy Review are detailed in the Submission of Complaint to the Commonwealth Ombudsman, which identifies multiple other issues.



​
[1] Evidence, B. C. (2016) Nuts, bolts, and tiny little screws: how Clinical Evidence works ​​
[2] Courtesy Homeopathy Research Institute (HRI)