NHMRC Homeopathy Review - Key facts

---

“If the intent is to provide general statements about the effectiveness of homeopathy, then ‘no reliable evidence’ may not adequately reflect the research. For example, when a substantial proportion of small (but good quality) studies show significant differences, […] 'no reliable evidence' does not seem an accurate reflection of the body of evidence."
- Undisclosed expert methodological peer reviewer feedback to NHMRC, 30 Aug 2013

An extensive, detailed investigation by the Australian Homeopathic Association (AHA) and Complementary Medicines Australia (CMA) into NHMRC’s conduct, combined with an in-depth scientific analysis of the report by the Homeopathy Research Institute (HRI), has revealed evidence of serious procedural and scientific misconduct in producing this report, as follows:

The hidden first review, April-August 2012:

• NHMRC did the homeopathy review twice, producing two reports, one in August 2012 and the one released to the public in March 2015 (the second attempt).
  
  The existence of the first report and taxpayer expenditure has never been disclosed to the public – it was only discovered through Freedom of Information (FOI) documents.  

 

• NHMRC claims it rejected the first report because it was 'poor quality', despite it being undertaken by a reputable scientist and author of NHMRC’s own accepted guidelines on how to review health evidence.

 

• Freedom of Information requests have revealed that a member of NHMRC’s expert committee overseeing the review process confirmed the first review to be high quality saying:
  
  “I am impressed by the rigor, thoroughness and systematic approach given to this evaluation of the published reviews of efficacy and side effects of homeopathy [….] Overall, a lot of excellent work has gone into this review and the results are presented in a systematic, unbiased and convincing manner.” 

​
'Rigorous assessment of over 1800 papers'?

• NHMRC said the conclusion of the (second) report published in 2015 were based on a “rigorous assessment of over 1800 papers” - when this was not the case. In fact, the results were based on only 176 studies, with the remainder of the evidence base ignored altogether. NHMRC merely gave the public the impression that it "rigorously assessed over 1800 papers".
  
  
• Thus the Review's conclusions are not representative of the broader evidence base and excludes a wealth of positive research evidence.
  
  ​​- See 'Further resources' for a summary of homeopathy research excluded from the NHMRC Review

 

• Of the 176 included studies, no original papers were retrieved or assessed, which is unprecedented in NHMRC reviews. This limitation meant that NHMRC had to abandon its 'rigorous approach that has been developed by Australian experts in research methods' mid way through the Review. This was because the secondary 'summary' data that NHMRC chose to exclusively rely on (systematic reviews) was limited by missing and/or incompletely reported primary trial data.
  
  NHMRC did not report this fact in the Information Paper that presented the Review findings to the public. Instead, it misleadingly informing the public that it did use its usual 'rigorous approach' (NHMC Homeopathy Review Information Paper, p.10), when it didn't. 

 
Post-hoc changes to the research protocol & use of arbitrary criteria:
​

• Instead, NHMRC used a method that has never been used before or since (including by NHMRC), employing criteria not recognised by any scientific guidelines or standards:
  
  NHMRC decided that for a trial to be ‘reliable’ it had to have at least 150 participants and reach an unusually high (100%) threshold for quality. This is despite the fact that NHMRC itself routinely conducts studies with less than 150 participants, which are not regarded as 'unreliable'.
  
  Australia's Therapeutic Goods Administration (TGA) does not recognise or apply such stringent thresholds, even when assessing the evidence for registered prescription medicines. 
  
  
• These unprecedented and arbitrary rules meant the results of 171 of the trials were completely disregarded from the findings as being ‘unreliable’, leaving only 5 trials NHMRC considered to be ‘reliable’.
  
  Since NHMRC assessed all 5 of these trials as 'negative', this explains how NHMRC could conclude that there was no ‘reliable’ evidence.​
  
  Below is an in depth analysis of the NHMRC Review, conducted as part of the investigation into NHMRC's procedures and methods:

Study wrongly cited:

• NHMRC cited a 2013 BMJ study throughout its final report documentation to authenticate the arbitrary '150 sample size' exclusion threshold, even though the BMJ paper made no association between a trial size and the 'reliability' of its results. 
  
  
• However the NHMRC Working Committee could not have read the BMJ paper, which cautioned that its findings could "not be extrapolated" to the types of studies NHMRC considered in the Review: 'continuous outcomes studies'.
  ​
• NHMRC has been unable to justify this embarrassing error, which also further confirms the arbitrary nature of the 150 trial participant exclusion criterion.​
  ​

No subject/ research experts appointed to the 'expert committee':
​

• In violation of NHMRC’s own mandatory standards, there was not one homeopathy expert appointed to the NHMRC Homeopathy Working Committee, unprecedented in NHMRC review processes.

​
Bias & conflicts of interest:
​

• Professor Peter Brooks, the first Chair of the NHMRC committee that conducted the 2015 review, initially failed to declare that he was a member of the anti-homeopathy lobby group ‘Friends of Science in Medicine’ (FSM). He was allowed to remain on the committee and NHMRC never formally managed the conflict for the duration of the Review.

​

• During the Review, multiple members of FSM sat on NHMRC Council and Health Care Committee - involved in decision-making processes relating to the Review - without any of these conflicts being declared or managed - in breach of statutory conflicts of interest policy.

 ​

• In 2010, the NHMRC CEO, Prof Warwick Anderson, personally instigated NHMRC's focus on homeopathy while publicly declaring it to be an "alleged therapy". The same month, NHMRC Council approved the content of a draft NHMRC Position Statement on homeopathy, declaring it to be "inefficacious" and its practitioners ​"deceptive" - on the sole basis of a political (not scientific) UK report and without assessing any scientific evidence or consulting with subject/ research experts. 

​

• In July 2011, the NHMRC Chairman revealed and confirmed NHMRC's anti-homeopathy bias when he publicly declared:
  
  “Let me assure you I am no supporter of homeopathy. As Chairman of NHMRC I can also assure you that NHMRC does not support homeopathy.”

 

• On 8 April 2014, FSM published a letter congratulating the NHMRC CEO of the Draft Review findings - the day before NHMRC officially released it to the Australian public (9 April). Homeopathy stakeholder groups were excluded from notification. In the letter, FSM's Vice-President, Prof Alastair MacLennan, urged that Australians "not be sold snake oil".

​

• In May 2014, NHMRC contracted a FSM Supporter within Prof MacLennan's research Institute at Adelaide University to assess additional evidence submitted by external parties. NHMRC did not declare or manage the conflict and all the additional evidence submitted (93% of which was positive for homeopathy) was dismissed. 

​

• In April 2015 after release of the final Review, the NHMRC CEO reiterated the FSM Vice-President's message, vilifying the homeopathy sector as "snake oil merchants". 

​​

• In February 2016, the Chair of the Homeopathy Working Committee, Prof Paul Glasziou, also vilified homeopathy as a "therapeutic dead end" on the sole basis of the Review he presided over and co-developed the criteria for.

​

• Such conduct did not uphold the principles of the NHMRC Service Charter, which assures stakeholders, "We treat our stakeholders with dignity and respect" - a policy intended to uphold the legal Australian Public Service Values of 'ethical', 'impartial', 'respectful' and 'transparent' conduct.

​
Expert peer reviewer advice ignored & not disclosed:

• NHMRC’s own expert advisers disagreed with the “overly definitive findings” of the Review's findings, with the Australasian Cochrane Centre advising that the NHMRC’s finding of ‘no reliable evidence’ did not accurately reflect the research:
  
  “If the intent is to provide general statements about the effectiveness of homeopathy, then ‘no reliable evidence’ may not adequately reflect the research. For example, when a substantial proportion of small (but good quality) studies show significant differences, […] 'no reliable evidence' does not seem an accurate reflection of the body of evidence."

 

• Another expert reviewer reiterated this conclusion, advising NHMRC:
  
  “The dismissal of positive systematic reviews compounded with the lack of an independent systematic review of high quality randomised controlled trials leaves me uncertain of the definitive nature of the Report’s conclusions.”

 

• ​NHMRC ignored this important advice and withheld it from the Australian public, despite publishing a dedicated 'Expert reviewer comments' document.

NHMRC Statement on Homeopathy - opinion, not evidence-based:

The NHMRC issued a formal Statement on Homeopathy alongside the Information Paper that states, “People who choose homeopathy may put their health at risk”.

The presence of this statement within a section called “NHMRC’s interpretation of the assessment of the evidence on the effectiveness of homeopathy” misleads the public into believing that it is based on the findings of their Overview, when it is not.

The Homeopathy Review did not include any assessment of the safety of homeopathy, nor any studies comparing the relative benefits/risks of using homeopathy versus conventional medicine; in fact all safety studies submitted to NHMRC during the Review were excluded as ‘out of scope’.

This false impression is strengthened further by NHMRC repeatedly and inaccurately describing the Homeopathy Review to the public as a "Health Technology Assessment" (IP, p.5 and p.38; FAQs document, p4,p9,p11; Administrative Report, p.5-6); HTAs by definition do assess safety, but NHMRC did not conduct an HTA, it conducted an Overview. The Overview excluded from scope any consideration of customary HTA parameters, such as 'safety', 'effectiveness' (studies that assess how homeopathy works in real-world clinical settings), 'cost-effectiveness' and 'quality'.