Procedural irregularities
 
​ - Post-hoc changes to the research protocol
​ - Exclusion of subject/ research experts
 - Inauthentic public consultation process

---

"Guideline development, the provision of advice and peer review all require persons with expertise relevant to the topic under review."
- NHMRC Guideline development and conflicts of interest policy

"I am concerned that no homeopathic expert was appointed to the NHMRC Review Panel. I cannot imagine this being agreed in oncology, orthopaedics or other disciplines."
​- Undisclosed expert peer reviewer feedback to NHMRC, May 2014

Background​

The public, health professionals and decision makers rely on the advice of respected institutions such as the NHMRC, which evaluate the evidence on a particular topic, to determine which treatments are effective and safe. Since these are matters relating to public health and safety, the integrity of processes undertaken by such institutions must be above reproach. 

Research fraud and administrative misconduct undermines the public’s trust in science and results in loss of trust in public institutions. It endangers the health of millions, who are denied access to accurate information on clinical interventions relevant to their health and wellbeing.

Post-hoc changes to the Homeopathy Review research protocol

The original, unpublished research protocol:

After NHMRC terminated a first review of the evidence on homeopathy conducted under a first contractor between April and August 2012 (see 'The First Homeopathy Review'), it started again under a second contractor, OptumInsight ('Optum'). 

Freedom of Information (FOI) documents reveal that the original research protocol for the Optum review was 'agreed and finalised' between the Office of NHMRC (ONHMRC), Homeopathy Working Committee (HWC) and Optum in late December 2012 [1].

The original research protocol was never published, but obtained through FOI. It specified the following evidence statement framework to be applied to the Optum assessment [1]:

It is apparent that the original evidence framework did not specify any minimum thresholds for trial 'sample size' or 'quality'; no 'adapted GRADE' tool or 'null hypothesis' approach. FOI documents have revealed that these criteria, which underpinned the Review's concept of "reliable evidence" and the published findings, were all developed post hoc - during the months following Optum's completion of the evidence assessment in March 2013. This is further detailed below:

Research protocol modified midstream:

Documents released under FOI reveal that after finalisation of the research protocol in December 2012, Optum completed the evidence assessment between 3 January 2013 and mid March 2013. The minutes of the 18 March 2013 meeting of the HWC show that a decision was made to deviate from the agreed research protocol in order to alter the results of the Review and reach more definitive conclusions [2]:

• “Optum noted that the approach taken when developing draft evidence statements did not align with the approach proposed in the research protocol as the evidence statement for all conditions would have stated that the effect was uncertain.”

Concluding that an effect is ‘uncertain’ or ‘inconclusive’ (a conclusion permitted by the original evidence statement framework) leaves open the possibility that the reported effect may in fact be positive, with future research shedding light on the issue. Such a conclusion accords with usual scientific reporting protocols. Apparently, the HWC/ ONHMRC were resistant to allowing any ‘uncertainty’ in the reported findings, thus leaving open the possibility that homeopathy could have positive effects. 

​Post hoc manipulation to the research protocol removes an important and customary safeguards against bias, since such changes expose the process to the possibility of bias (i.e. facilitating manipulation of the findings, whether conscious or unconscious). 

After Optum completed the evidence assessment and the HWC met in March 2013 to discuss the findings and evidence statement protocol, the ONHMRC established a HWC Sub-Group for the specific purpose of further refining the research protocol and evidence statement framework. FOI documents reveal that the Sub-Group met on several occasions between April and June 2013 (see table below). 

The existence of this process was not disclosed in NHMRC's administrative reporting of the Review, removing another important safeguard against bias. 

During this Sub-Group process, all the elements of the evidence statement framework and its component criteria were retrospectively developed and applied to the data. These post hoc changes to the research protocol were not disclosed, despite the fact they formed the basis of the Review's published findings. 

Timeline of retrospective changes to the research protocol:

For each of the 61 clinical conditions evaluated, NHMRC developed and published an evidence statement framework comprised of three Elements (NHMRC Information Paper, p.35):

1. Body of evidence - a description of the body of evidence, which included the trial exclusion thresholds based on trial 'sample size' and an allocated ‘quality rating’ 
2. Level of confidence (LOC) rating for the body of evidence as a whole - using a novel ‘adapted GRADE’ tool that was specifically created for the Review (never used before or since)
3. Conclusion - a concluding statement that described the effectiveness of homeopathy as a treatment for a particular condition, compared with either placebo or other treatment(s).

Notably, this framework and its component Elements did not exist in the original research protocol. The original research protocol did not include any 'sample size' or 'quality rating' exclusion thresholds, no ‘null hypothesis approach’ or ‘adapted GRADE’ tool - all of which underpinned the Elements of the Review’s framework for "reliable evidence". 

The table below shows the timeline of changes made to the research protocol during the HWC Sub-Group process and the impact it had on the Review’s findings and conclusions. All the following changes were made by the Office of NHMRC/ HWC (the administrator) and not the contractor (Optum) who simply applied them:

Impact of the changes to the research protocol on the findings:

The NHMRC report neither discloses these changes to the research protocol nor their impact on the Review's findings.

Forensic investigation of NHMRC's methods reveals that these retrospective changes directly resulted in the results of 171 out of the 176 included trials (97%) being dismissed from “any further consideration” in the Review's findings [14]. That is, the results of these 171 studies did not contribute to the Review's findings at all.

NHMRC thereby gave the impression that these trials were ‘included’ in the Review, whereas their results were in fact dismissed from any consideration at all due to being "unreliable", i.e. because they had less than 150 trial participants and/or did not meet the unusually high 100% quality rating cut-off.

Both the ‘N=150 trial participants’ and ‘5/5 Jadad (or equivalent) quality rating’ thresholds are entirely arbitrary criteria that are not recognised by any scientific standards (including NHMRC's) and have never been used before of since by any other research group, including NHMRC.

Dismissing the results of 171/ 176 trials effectively reduced the entire Review's findings to only 5 trials the NHMRC deemed to be "reliable" (see 'Australian Report FAQs' - dot points 4 & 5 for further information on these trials).

The importance of research protocols:

Research protocols are an important safeguard used to reduce and prevent reporting bias in scientific studies and are usually published before a study begins. The whole purpose of scientific inquiry is to ask an open-ended question, review the evidence according to an agreed protocol in an objective and unbiased manner and accept the answer.

Before a study begins, a protocol is created which outlines in detail all essential aspects of the project, such as the research question being asked, methods of data retrieval, criteria used to determine which studies will be included or excluded from the review, and how the included data will be analysed to produce the final results. If any changes to the protocol are required, they must all be fully disclosed and justified.

The systematic re-creation of the Homeopathy Review’s research protocol throughout 2013 after Optum had already completed its evidence assessment, without disclosing any of the changes made, removed customary safeguards designed to protect the methods and findings from manipulation.

Exclusion of subject/ research experts:

One of the major criticisms regarding the process NHMRC undertook in developing a Draft Position Statement on Homeopathy in 2010/11 was the lack of consultation with homeopathy subject and/or research experts. Therefore, ensuring that the homeopathy reference group was balanced by the inclusion of suitably qualified homeopathy discipline/ research experts (a cornerstone of ethical scientific investigation) would have corrected the imbalance. This was also pertinent in light of NHMRC having had no previous experience or expertise in homeopathy, also even lacking a definition of homeopathy. 

In mid 2011, the NHMRC CEO, Prof Warwick Anderson, personally selected and appointed members of the reference group under section 39 of the NHMRC Act 1992, none of whom had any homeopathy subject and/or research expertise. The exclusion of topic experts was unprecedented in NHMRC review and guideline development processes, contravening accepted scientific research protocols as well as mandatory guidelines on the composition of NHMRC committees and working groups.

NHMRC expert committees:

NHMRC quality assurance processes require that NHMRC expert committees contain experts relevant to the topic under inquiry. The inclusion of subject and research experts relevant to the topic under review on expert scientific review committees is also a foundational requirement of ethical scientific inquiry.

The NHMRC has itself stipulated in formal correspondence [14] that its expert committees are required to be comprised of members with both:

• A “high level” of expertise relevant to the matter under investigation (subject expertise);
• Technical and scientific (methodological) expertise.

According to the NHMRC, appointments of members to its expert committees for the purpose of developing health advice (which was the purpose of the Review) is informed by the following instruments [15]:

1. Procedures and requirements to meet the 2011 NHMRC standard for clinical practice guidelines (the ‘2011 NHMRC Standard’)
   • Requirement A.3 of the 2011 NHMRC Standard stipulates as a mandatory requirement that working committees must include “relevant disciplines and clinical experts”;​ ​
2. NHMRC Guideline Development and Conflicts of Interest policy (‘NHMRC COI policy’):​

The failure to appoint any homeopathy subject or research experts to the HWC disregarded accepted scientific protocols and breached NHMRC quality assurance standards.

The question arises why NHMRC would disregard its own policies in this instance?

HWC meeting minutes obtained under FOI confirm that the NHMRC appeared to regard the inclusion of homeopathy experts as representing “an unmanageable conflict of interest” [16]. This represents a double standard NHMRC does not apply when appointing members to other expert committees. NHMRC guidelines acknowledge that expert committees, by their very nature, “all require persons with expertise relevant to the topic under review”.

Inauthentic public consultation process:

NHMRC reported that the ‘risk’ of missing single trials not captured in the Overview Report was ‘offset’ by inviting submissions from homeopathy interest groups (in 2012) and via a formal public consultation (in 2014). NHMRC inaccurately states that this externally submitted evidence was, “...assessed using a similar method to that applied in the overview” [17] but “did not alter the overall findings of the assessment of the evidence” [18].

In fact, external submissions were assessed entirely differently and separately from the rest of the evidence base, in a way that meant it was never possible for any externally submitted evidence to contribute to or alter the results of the Review.

Thus, of 49 submitted randomised controlled trials that NHMRC considered suitable (around a quarter of the eligible evidence), the number that entered the Overview Report was 0. 40 of these additional trials were submitted and accepted during the pubic consultation phase of the Review in 2014, 37 of which reported statistically positive/ clinically relevant findings for homeopathy. 

This makes a sham of NHMRC’s apparent attempt at external co-operation and transparency.

This approach was also strongly critiqued in expert peer reviewer feedback, which NHMRC withheld from disclosure despite publishing an 'Expert reviewer comments' document. FOI documents reveal undisclosed expert reviewer feedback such as [19]:
​

• "I do not agree that considering random evidence provided by interest groups offsets this deficiency. High quality RCTs with narrow confidence intervals (Level 1 evidence) should have been searched for and included in this review."

However, how these extra trials were assessed is actually a moot point, since 27 of the 49 ‘suitable’ trials submitted by external parties were entirely dismissed as being ‘self-selected samples’; the remaining 22 were ‘downgraded’ for the same reason. The end result was that none of them contributed to the findings.

That is, NHMRC invited external parties to submit evidence it 'may have missed', then dismissed this evidence on the basis that external parties submitted it. This lacked procedural/ scientific fairness and transparency. This could easily have been avoided by NHMRC carrying out a systematic literature search for all other trials on the same topic as those submitted, which the NHMRC chose not to do.

These facts are not mentioned at all in the Information Paper, which only emphasises that the NHMRC ‘considered submissions of evidence from external parties’ and that this additional evidence 'was assessed in a similar fashion' to the 176 trials considered by Optum. 

---

1.2.2 - Guidelines, advice and peer review would be of lesser quality without relevant experts.
- NHMRC Guideline Development and conflicts of interest policy

[1] Original research protocol for the Optum Homeopathy Review. NHMRC FOI 2014/15 004-58
[2] HWC face-to-face meeting minutes (final endorsed). 18.03.2013. NHMRC FOI 2015/16 007-03
[3] Ibid.
[4] HWC and HWC Sub Group meeting minutes, 2013-2014. NHMRC FOI 2015/16 008. 
[5] 2013.04.30 - Email from ONHMRC to HWC members re. approach to homeopathy evidence statements. 29.04.2013. NHMRC FOI 2014/15 021-11
[6] HWC Sub Group teleconference meeting minutes. 29.04.2013 - NHMRC FOI 2015/16 008-03 & 06.05.2013 - NHMRC FOI 2015/16 008-03
[7] HWC Sub Group teleconference meeting minutes. 06.05.2013. NHMRC FOI 2015/16 008-03
[8] HWC Sub Group teleconference meeting minutes. 24.05.2013. NHMRC FOI 2015/16 008-07
[9] First round Australasian Cochrane Centre (ACC) methodological peer review. NHMRC FOI 2015-16 008-13
[10] HWC face-to-face meeting minutes. 11-12.07.2013. NHMRC FOI 2015-16 008-06
[11] Homeopathy Research Institute (HRI). Independent expert analysis of NHMRC’s methods, unpublished
[12] Australasian Cochrane Centre (ACC) follow-up methodological peer review. 30.08.2013. NHMRC FOI 2015-16 007-05
[13] Ibid.
[14] Homeopathy Research Institute (HRI). Independent expert analysis of NHMRC’s methods, unpublished
[15] 2015.12.22 - NHMRC response to 30.09.2015 Complementary Medicines Australia (CMA)/ Australian Homeopathic Association (AHA)
[16] Minutes of Face to Face meeting of the Homeopathy Working Committee, 30.07.2014. NHMRC – Freedom of Information document 2015/16 008-11
[17] NHMRC Information Paper, p.8. CAM02A. NHMRC, 11.03.2015
[18] NHMRC Information Paper, p.25. CAM02A. NHMRC, 11.03.2015
[19] 2014.05.10 - Expert reviewer feedback to NHMRC during Review public consultation phase (2014) - NHMRC FOI 2014/15 004-62

"Research fraud and administrative misconduct undermines the public’s trust in science and results in loss of trust in public institutions.
It endangers the health of millions, who are denied access to accurate information on clinical interventions relevant to their health and wellbeing."

"... it is important that the methods to be used should be established and documented in advance. Publication of a protocol for a review prior to knowledge of the available studies reduces the impact of review authors’ biases, promotes transparency of methods and processes, reduces the potential for duplication, and allows peer review of the planned methods (Light 1984)."
- Cochrane Handbook for systematic reviews of interventions (cited against the Review)

"It is important, however, that changes in the protocol should not be made on the basis of how they affect the outcome of the research study. Post hoc decisions made when the impact on the results of the research is known, such as excluding selected studies from a systematic review, are highly susceptible to bias and should be avoided."
- Cochrane Handbook for systematic reviews of interventions (cited against the Review)

​"I am concerned that no homeopathic expert was appointed to the NHMRC Review Panel. I cannot imagine this being agreed in oncology, orthopaedics or other disciplines."
- Expert reviewer feedback to NHMRC, May 2014