The First (hidden) NHMRC Homeopathy Review"Overall, a lot of excellent work has gone into this review and the results are presented in a systematic, unbiased and convincing manner.” First Review of the evidence on homeopathy conducted in 2012:
NHMRC did not disclose that between April and August 2012 it completed a First Review of the evidence on homeopathy. That is, it withheld from public knowledge that it reviewed the evidence twice. The First Review was titled ‘A Systematic Review of the Evidence on the Effectiveness of Homeopathy’ and was conducted by a highly respected and experienced reviewer, a principal author of NHMRC's own guidelines on how to review health evidence (that NHMRC referenced against the Review) [1]. The first reviewer’s contract commenced on 4 April 2012, coinciding with the commencement of the term of the Homeopathy Working Committee (HWC). A publically-funded, completed review: NHMRC has refused to release the findings or methodology of the First Review, despite it being a tax-payer funded report. Freedom of Information (FOI) documents reveal that the reviewer delivered on their contract to assess the research evidence on homeopathy, and submitted their final Draft Report to NHMRC at the start of August 2012 [2]. The Office of NHMRC sacked the contractor within days of receiving their final Draft Report, apparently without consulting the Homeopathy Working Committee (the expert committee NHMRC appointed to oversee the Review) [2]. At this time, the NHMRC website announced that the term of the HWC had been extended for another two years, without providing any explanation why. The First Review - methodology: An indication of the approach taken by the first reviewer has been revealed in a document released under FOI, revealing the table of contents of the penultimate version of the reviewer's final Draft Report [2A]. This shows that the First Review identified 48 relevant systematic reviews (SRs), as opposed to 57 SRs in the Optum overview (which was commenced 8 months later, thus included newly published data). The July 2012 draft report table of contents indicates that the reviewer categorised the evidence as follows: 
The First Review - a good quality report:
Good quality research evidence on homeopathy is known to exist in several medical conditions (e.g. upper respiratory tract infections, sinusitis, cough, hay fever, children with diarrhoea, lower back pain) and it appears highly likely that the First Review reported this evidence. FOI documents capturing HWC member Prof Fred Mendelsohn's feedback on the first reviewer's Draft Report reveals that the First Review was of high methodological quality [3]:
This is perhaps not surprising, since the first reviewer co-authored NHMRC's seminal, accepted method for assessing health evidence, which they had used in reviewing evidence on numerous occasions before undertaking the Homeopathy Review. The HWC member's feedback to NHMRC on the draft first reviewer's report feedback began [4]: 
The Reviewer's overall assessment of the (penultimate) July 2012 version of the draft report was [4]:
Such feedback is not consistent with a review so deeply flawed it warranted terminating so shortly thereafter. First Review and conflicts of interest: The initial Chair of the HWC (Prof Peter Brooks) was a Supporter of the anti-homeopathy advocacy group Friends of Science in Medicine (FSM), whose conflict was not acted on by NHMRC until half way through the first reviewer’s term [5]. This was not disclosed by NHMRC for the duration of the Review. NHMRC also failed to disclose that Prof Brooks did not declare his conflict upon initially being appointed to the HWC [6] and he did not acknowledge the conflict when stepping down as Chair [7]. He was retained on the HWC without a formal management plan being instigated for the duration of the Review [8].
Existence of First Review and public expenditure not disclosed: NHMRC did not disclose the existence of this review, its findings, methodology, why it was terminated or public expenditure on the process. The first review’s existence is not mentioned anywhere in the NHMRC Administrative Report, NHMRC Annual Reports, NHMRC website or any other publically available NHMRC documentation. The first reviewer - an Australian expert in reviewing health evidence: The NHMRC Information Paper presenting the Homeopathy Review’s findings to the public assured (p.9):
The first reviewer was one of the ‘Australian experts in research methods’ that co-developed this 'rigorous approach', having authored a number of seminal NHMRC guidelines. These include those commissioned by NHMRC to guide authors when developing clinical evidence based guidelines, using a standardised methodology that has since become NHMRC’s accepted protocol for reviewing evidence and developing guidelines to ensure quality standards are maintained (the NHMRC ‘dimensions of evidence’ assessment framework). It includes guidelines that NHMRC directly referenced against the Homeopathy Review [9]. They had co-authored numerous publications or were an NHMRC advisor covering specific conditions applying the NHMRC guidelines they had written. It is highly unusual to abruptly terminate the contract of such a highly respected and credentialed reviewer under such circumstances - particularly when the contractor had delivered on their contract and submitted a completed report for consideration. NHMRC intended to complete draft Information Paper in-house: Freedom of Information returns reveal that after termination of the first review process, the Office of NHMRC (ONHMRC) intended to “complete the work in-house to develop the Information Paper” [10]. The email to the HWC Chair stated that ONHMRC would email HWC members a copy of:
It is unclear on what basis ONHMRC would complete an Information Paper based on an evidence review that had been terminated. This process was then apparently abandoned in favour of starting again under a new contractor (OptumInsight). In response to a FOI request for the above-mentioned draft information paper, NHMRC responded that it "did not exist". NHMRC ‘Researcher Responsibilities and Considerations’ guidelines: The first review results were kept from the general public, the research was not available for critical review, the potential for innovation and progress that could benefit Australians’ access to potentially promising treatments was denied and valuable public resources were wasted. NHMRC stipulates under its ‘Researcher Responsibilities and Considerations’ (9.1) guidelines:
NHMRC’s webpage summarising the Code referred to stipulates:
Section 4.4 of the Code (‘Responsibilities of Researchers - Disseminate all research findings’) stipulates:
[1] Feedback from HWC member to ONHMRC re. first reviewer's draft report on the assessment of the evidence on homeopathy, 15 July 2012. NHMRC FOI 2014/15 021-09 [2] International Centre for Allied Health Evidence, UniSA, homeopathy review search strategy, April 2012. NHMRC FOI 2014/15 021-05 [2A] Email from HWC member to ONHMRC re. UnSA draft report on homeopathy, 15 July 2012. NHMRC FOI 2014/15 021-09 [3] Feedback to NHMRC from HWC member Prof Fred Mendelsohn on the July 2012 version of the first reviewer's Draft Report, 15 July 2012. NHMRC FOI 2014/15 021-08 [4] Feedback to NHMRC from HWC member Prof Fred Mendelsohn on the July 2012 version of the first reviewer's Draft Report, 12 July 2012. NHMRC FOI 2014/15 021-08. NHMRC FOI 2016/17 016-13 [5] Email correspondence between NHMRC CEO & Office of NHMRC re. Prof Brooks' conflict on the HWC, 10 May 2012. NHMRC FOI 2014/15 021-02 [6] Prof Brooks Declaration of Interest (DOI) form, 16 March 2012. NHMRC FOI 2015/16 002-18 [7] Email correspondence from Prof Brooks to NHMRC CEO. NHMRC 2014/15 021-03 [8] NHMRC Administrative Report, p.5 [9] NHMRC Additional levels of evidence and grades for recommendations for developers of guidelines 2009: NHMRC. Page 9. [10] Email from ONHMRC to HWC Chair Prof Paul Glasziou re termination of first reviewer's contract, 13 August 2012. NHMRC FOI 2014/15 021-10 |
"Overall, the consistent use and reporting of the same criteria for each of the evaluations is a strength and reveals the careful systematic approach that has been brought to these evaluations.” “Researchers have a responsibility to their colleagues and the wider community to disseminate a full account of their research as broadly as possible: The account should be complete, and, where applicable, include negative findings and results contrary to their hypotheses.” "I believe that the assessment of secondary literature has been performed very well with careful systematic analysis and the results are supported factually with strong supporting material." 
|