NHMRC Service Charter:
The ‘NHMRC Service Charter’ stipulates that “NHMRC operates within a corporate governance framework that promotes integrity, compliance, performance and efficiency” and “individually and collectively upholds the Australian Public Service (APS) Values”.
This is a legal framework under the Public Service Act 1999 that incorporates the core principles: ‘impartial; committed to service (we work collaboratively …); accountable; respectful; ethical (we are trustworthy and act with integrity in all we do)’.
Homeopathy subject & research experts excluded from the expert committee:
A major procedural flaw associated with the Review was the unprecedented exclusion of homeopathy subject, clinical or research experts from the Homeopathy Working Committee (HWC). The NHMRC CEO, Prof Warwick Anderson, personally selected and appointed members to the HWC under s.39 of the NHMRC Act 1992 (see 'Bias' section regarding the CEO's anti-homeopathy public comments). He initially appointed four members to a 'homeopathy reference group' in mid 2011, later expanded to become the HWC from 2 April 2012.
The exclusion of consultation or collaboration with homeopathy experts was a key criticism of the previous development of a Draft NHMRC Statement on homeopathy, abandoned due to perceptions of bias and lack of rigour (see 'Bias'). The exclusion exposed the Review to bias and prevented the input of appropriate subject expertise.
Not only was the exclusion a breach of widely accepted scientific research protocols, it was also a direct breach of mandatory NHMRC standards informing the composition of its expert committees (as stipulated by NHMRC itself in correspondence to stakeholders on 22.12.2015):
Procedures and requirements to meet the 2011 NHMRC standard for clinical practice guidelines (the ‘2011 NHMRC Standard’):
NHMRC Guideline Development and Conflicts of Interest policy:
Freedom of Information returns have revealed that the 2011 NHMRC Standard was stipulated as a guidance document against the Homeopathy Review - in the contracts of both reviewers contracted to undertake the Review, as well as in the minutes of the inaugural meeting of the HWC on 12 April 2012 . The Statement of Requirement in the first reviewer's contract specified :
It is unclear why the contractor was required to comply with the requirements of the Standard, whereas NHMRC decided it could choose which of the mandatory requirements of the Standard it would (and would not) observe.
Exclusion of subject experts criticised by expert peer reviewer & HWC member(s):
In 2014, one of NHMRC’s independent expert peer reviewers criticised the omission of subject experts on the HWC (the second sentence being withheld from public disclosure) :
The NHMRC Administrative Report (p.5) records the dissenting view of a HWC member over the exclusion of homeopathy experts on the HWC, which "was not supported by either NHMRC or the remaining committee membership".
The Administrative Report did not record further dissenting views within the HWC, regarding the failure to appoint any homeopathy experts amongst the three expert reviewers appointed in 2014 during the public consultation phase of the Review  - in breach of NHMRC guidelines relating to the appointment of expert reviewers (see below).
Homeopathy stakeholder sector not informed of key decisions:
The NHMRC Service Charter webpage assures NHMRC stakeholders (which includes the homeopathy sector):
On 9 April 2014, NHMRC released its Draft Information Paper on its findings on the evidence on homeopathy for public consultation and expert review feedback (required steps in NHMRC’s ‘quality assurance process’).
Homeopathy stakeholders, including the AHA (which had corresponded with NHMRC and provided submissions), were excluded from any advance notification during the embargo period - despite the release being a major decision that 'affected them'.
The release was also timed to the eve of the commencement of 'Homeopathy Awareness Week', an annual international professional event commencing on 10 April each year. Whether or not the timing was deliberate is open to interpretation, however it did not model 'dignified' or 'respectful' conduct towards stakeholders.
Anti-homeopathy lobby groups are informed of key decisions:
On 8 April 2014, the day before NHMRC’s official release of the Draft Information Paper, the anti-homeopathy lobby group Friends of Science in Medicine (FSM) published an open letter to the NHMRC CEO ‘congratulating’ him on NHMRC’s findings. The letter was co-signed by seven of FSM’s founding members and confirmed that FSM not only received advanced copies of the NHMRC media release, but also the Optum Overview Report and NHMRC Draft Information Paper - a privilege that homeopathy stakeholders who contributed to the process and liaised with NHMRC were not afforded.
In the open letter, FSMS’s co-founder and Vice President, Prof Alastair MacLennan, urged the CEO:
The importance of research protocols:
Research protocols are an important safeguard used to reduce/prevent reporting bias in scientific studies. Before a study begins, a protocol is created which outlines in detail all essential aspects of the project, such as the research question being asked, methods of data retrieval, criteria used to determine which studies will be included or excluded from the review, and how the included data will be analysed to produce the final results.
Making significant ‘post-hoc’ changes is a recognised source of bias as the reviewers may (consciously or subconsciously) alter the method to achieve a desired result. Hence, ethics committees are often involved in approving changes made. In ethical scientific inquiry, all changes to the research protocol must be fully disclosed and justified.
Much of the following information has been gathered from Freedom of Information (FOI) documents released between 2015 and 2016: not reported by NHMRC in 944 pages of report documentation.
Review's research protocol undergoes extensive post hoc modification:
One the most concerning procedural irregularities of the NHMRC Homeopathy Review was the systematic reconstruction of the research protocol, after it had been agreed and finalised and after the contractor (Optum) had completed its evidence assessment. FOI documents and NHMRC files also reveal that NHMRC altered the purpose of the review and the evidence statements along the way.
These post-hoc changes to 1) the research protocol and 2) the intended purpose of the evidence statements for the Homeopathy Review are summarised in the Table below (click to expand/ download):
NHMRC in fact admitted that the research protocol was altered post-hoc - at the back (on p.270) of the Optum 'Overview Report Appendices', a technical subsidiary document unlikely to be read:
However, the key changes to the protocol developing the criteria used in the analysis, which formed the basis of the published findings were not disclosed at all - only revealed through Freedom of Information documents, as detailed in the narrative summary below:
Research protocol for Optum Overview finalised in December 2012:
Following NHMRC’s rejection of the first review in August 2012 (see 'The First Review'), in October 2012 NHMRC contracted a second reviewer, OptumInsight (Optum) to assess the evidence (NHMRC Administrative Report, p.6), without any reference made to the existence of the first review process.
On 21 December 2012, the second research protocol was agreed and finalised between the Office of NHMRC (ONHMRC), the Homeopathy Working Committee (HWC) and Optum, before Optum began their review in January 2013. ONHMRC emailed it to HWC members on 21.12.2012 :
Original research protocol not disclosed; allowed for "uncertainty" in the results:
The originally agreed research protocol was not disclosed.
The 'standardised' evidence statement framework/ criteria for the original 'agreed and finalised' research protocol can be viewed in the margin at right (click to enlarge) . It centred on the criterion of whether a study was 'adequately powered' - which was dropped midway through the Review (see below).
Most notably, the original protocol allowed for reporting uncertainty in the results. Thus, where findings were inconclusive (common in all research), it allowed for the possibility that there "may" be an effect, or where evidence was poor and/or 'underpowered', that the effect was "uncertain" - consistent with how research findings are usually reported. This bears no resemblance to the definitively negative 'no reliable evidence' statements developed later in the Review and published.
The other notable factor is that the original protocol did not include any of the criteria that eventually underpinned the Review's published findings, which FOI returns reveal were not introduced until much later, as summarised below:
Optum completes assessment by March 2013, research protocol changed:
Optum began its evidence assessment on 3 January 2013 and completed it by March 2013.
The minutes of the 18 March 2013 meeting of the HWC (obtained under FOI) show that when developing draft evidence statements, the original protocol was not followed because it would have found that the evidence for homeopathy in all conditions was “uncertain” (i.e. not negative) :
This shows that a decision was made post-hoc to deviate from the agreed original protocol in order to alter the results of the Review and reach more definitive conclusions - a clear example of bias.
ONHMRC creates a special HWC Sub-Group, which further modifies the research protocol (April-June 2013):
Between April and June 2013, ONHMRC established a HWC Sub-Group process , whose specific task was to develop, entirely post-hoc, the criteria that comprised the published evidence statement framework. NHMRC did not report the existence of this Sub-Group process or its purpose.
The Sub-Group was coordinated by senior ONHMRC staff, in conjunction with the HWC Chair, Prof Paul Glasziou and other HWC members..
None of the criteria developed and applied during the HWC Sub-Group process were included in the original research protocol, yet they went on to underpin the framework used for the Review's published findings (Information Paper) and NHMRC's overall conclusion (NHMRC Statement on Homeopathy).
'Null hypothesis approach' - introduced late April 2013:
The ‘null hypothesis approach’, where homeopathy was ‘assumed to be ineffective unless 'reliable' evidence showed otherwise’ - was introduced midstream by the HWC Chair, Prof Glasziou, in late April 2013, approved by the HWC in early May 2013 . It was not part of the original research protocol.
At this stage, ONHMRC/HWC had not yet developed any of the criteria that underpinned its unique concept/ definition of "reliable evidence".
Decision to add 'conclusive statements', instead of 'informing the community of the evidence', introduced early May 2013:
The decision to add 'conclusive statements', instead of ‘informing the community of the evidence’ (as per the originally agreed protocol), was adopted midstream on 6 May 2013 .
This became 'Element 3' of the revised research protocol's evidence statement framework.
'Adapted GRADE' criterion created & applied during May 2013:
The novel, ‘adapted GRADE’ criterion for assessing the level of confidence in the results of the evidence (noting GRADE analysis is designed to be applied only to original studies, which were not retrieved or assessed in the Review) was formulated by the ONHMRC during May 2013 and then applied to the analysis .
Optum noted that it was unfamiliar with the tool, which was therefore applied/ guided by ONHMRC .
This became 'Element 2' of the revised research protocol's evidence statement framework.
'Adapted GRADE' does not pass expert peer review - unreported:
NHMRC withheld from public disclosure that its ‘adapted GRADE’ criterion did not pass either round of expert peer review (Australasian Cochrane Centre, ACC) [10A]: advice that was both ignored and not reported.
NHMRC also did not release any of its GRADE calculations or tables, in contrast to its other review processes. This was especially odd, given it was a novel modification of an accepted tool.
150 trial participant threshold adopted mid July 2013:
Between June and July 2013, the HWC Sub-Group introduced for the first time the concept of dismissing trials below a minimum sample size threshold from their results being considered in the Review's findings.
This was initially set at 200 participants for a trial to be 'adequately powered' on 24 May 2013 , later changed to 150 trial participants for a trial to be 'reliable' in mid July 2013 .
At some (unspecified) stage, the new N=150 criterion was linked to a British Medical Journal (BMJ) study published on 24 April 2013 (see below).
It meant that any trial that did not have a minimum of 150 participants was automatically deemed to be 'of insufficient size to warrant [any] further consideration of their findings' (IP, p.36) - and was therefore 'unreliable'.
The N=150 exclusion threshold became part of 'Element 1' of the revised research protocol's evidence statement framework, alongside another criterion that ONHMRC/ HWC had yet to develop (see below). This criterion was entirely arbitrary and unprecedented. This was also now seven months after the research protocol had been finalised (December 2012) and four months after Optum had completed its assessment (March 2012).
NHMRC regularly funds and collaborates on trials with less than 150 participants, which are not deemed 'unreliable'. The criterion does not conform to any accepted research method and has never been used before or since by NHMRC or any other research group anywhere in the world.
Impact of 150 trial participant threshold not reported:
NHMRC did not disclose/ quantify the major impact the retrospectively applied N=150 sample size criterion had on the analysis anywhere in its reporting of the Review.
Independent analysis of NHMRC's methods and procedures reveals that it dismissed 146 out of the 176 trials (83%) from their results being considered in the Review's findings .
In the right margin is an example of how this 'rule' enabled NHMRC to dismiss the findings of positive, good quality trials in several medical conditions, to conclude 'no reliable evidence'  (click to enlarge). For further examples, go to the 'Methodological flaws' section.
NHMRC erroneously cites a BMJ study to justify the N=150 criterion:
In attempting to justify the N=150 sample size criterion, NHMRC correctly identified the homeopathic studies it assessed as being ‘continuous outcomes studies’ and cited a BMJ paper (www.bmj.com/content/346/bmj.f2304) .
However, the BMJ paper, which makes no link between trial size and ‘reliability’, states that its results:
That is, not only does the BMJ paper make no link between sample size and trial reliability, it specifically states its findings cannot be extrapolated to the types of trials that NHMRC identified it assessed (‘continuous outcomes studies’).
NHMRC uses this citation of the BMJ study against the N=150 threshold multiple times across the final report documents released to the public (Information Paper, FAQ document, Optum Overview Report and in correspondence), who would not question that an expert body such as NHMRC would make such a fundamental error and/or intentionally publish misleading information.
On its own, this is an extraordinary error invalidates the integrity of the entire Review process. NHMRC has been unable to credibly justify this anomaly in correspondence .
Unusually high (100%) trial 'quality rating' threshold adopted August 2013:
In late July/ August 2013, NHMRC added yet another trial exclusion threshold to the N=150 threshold, as a co-element of its 'reliable evidence' framework. This was that any trial that did not meet an unusually high 100% quality rating (i.e. scoring 5/5 on the Jadad or equivalent using another rating scale) was assumed to be 'not good quality' and therefore "unreliable" .
Like the N=150 sample size threshold, this was an arbitrary criterion that did not conform to any recognised research standards; never used by NHMRC or any other research group before or since.
Impact of 'quality rating' threshold not reported:
NHMRC did not report that the 100% 'quality rating' threshold dismissed a further 25 out of the 30 remaining studies from their results being considered in the Review’s findings .
Thus, the entire Review came down to just 5 trials that ONHMRC/ HWC deemed to be 'reliable'.
Having reduced the evidence base to only 5 ‘reliable’ trials, the HWC unsurprisingly concluded that there is ‘no reliable evidence’ that homeopathy is effective.
'Quality rating' threshold did not pass expert peer review - undisclosed:
NHMRC did not disclose that this criterion did not pass expert peer review:
"Quality rating scales have seldom been tested for inter-rater reliability, there is a lack of agreement between scales as to what is being measured, and scales differ in many respects including items for inclusion and number of items." 
Remaining positive 'reliable trial' excluded from Review's reported findings:
Of the 5 remaining ‘reliable’ studies, 4 were interpreted as negative and 1 was positive, meeting NHMRC's own unique definition of 'reliable evidence'. This positive trial had 161 participants and was rated as 'good quality' (5/5 Jadad) .
Table 1 of the NHMRC Information Paper (pp.18-20) presenting the findings of the Overview to the public substituted this positive study with a negative study in the same condition (lower back pain)  that was not considered as part of the Overview.
Whether or not this was deliberate is open to interpretation, however it is the only one of the 176 trials omitted from Table 1 of the Information Paper.
Impact of post-hoc changes to the protocol on the Review's findings not reported or quantified:
Collectively, these post hoc changes to the research protocol directly resulted in the results of 171 out of the 176 (97%) of the included studies from their results being considered in the Review's findings (at all).
NHMRC did not report or quantify this significant impact anywhere in 944 pages of report documentation - despite claiming it conducted a 'rigorous', 'transparent' and 'ethical' review process.
NHMRC thereby presented the misleading and inaccurate impression that it had included the 176 Overview studies for review, whereas in fact the results of 171 of these studies were excluded from any consideration in the findings at all.
NHMRC's methodological expert peer reviewer not "independent":
NHMRC misinformed the public that its expert reviewer, the ACC, was “independent” (NHMRC Administrative Report, p.9; Information Paper, p.15), without disclosing that it is 100% funded by NHMRC - an undisclosed perceived conflict of interest exposing the Review to potential bias.
Sham public consultation & conflicts of interest:
In 2014, during the public consultation phase of the Review, NHMRC accepted for assessment an additional 40 studies, 37 of which (93%) reported statistically positive results for homeopathy.
All of these studies were dismissed on grounds of having a 'high risk of bias' and/or being ‘unreliable’ as a result of the N=150 sample size threshold rule.
The NHMRC Information Paper (p.6, 14) stated that this evidence "was assessed using a similar method to that applied in the overview" - but this was not the case.
This evidence was in fact assessed entirely separately and differently to the Overview studies, so that the number of these studies that entered the Overview was 0: making a sham of NHMRC’s apparent attempt at external cooperation and transparency.
This was compounded by NHMRC appointing a research group, the Australian Research Centre for the Health of Women and Babies (ARCH, Robinson Research Institute, Adelaide University), to assess the additional evidence submitted during public consultation that:
Despite claiming it observed conflicts of interest policy, NHMRC did not disclose (and therefore did not manage) the major conflict associated with this group. This was despite having appointed a FSM Supporter as the initial Chair of the HWC, without disclosing the fact or formally managing the conflict for the duration of the Review (see 'Conflicts of interest' section).
Studies submitted by homeopathy interest groups and the public were dismissed as biased ‘self-selected samples’
Details of how NHMRC evaluated externally submitted evidence differently from the original 176 trials has been discussed above, but it should be noted that for most of these studies this is actually a moot point, as their results could never have entered the Homeopathy Review, however they had been analysed.
This is because 27 of the 49 ‘suitable’ trials submitted by external parties were entirely dismissed due to being ‘self-selected samples’ (3/9 studies submitted by the homeopathy sector in July 2011 and 24/40 studies submitted during public consultation in June 2014); the remaining 22 were ‘downgraded’ for the same reason.
As stated in the ‘Review of submitted literature’ by Optum (p.10) and ‘Review of literature from public submissions’ by the ARCH (p.13) in reference to these 27 trials they say:
In Optum’s review of 9 trials from the homeopathy sector, at the end of their assessment of every single trial they state:
The ARCH assessed the submitted trials differently from Optum, as they differentiated between trials conducted on health conditions already included in the Overview Report (not dismissed due to self-selection bias because Optum had systematically retrieved evidence on that condition) and trials on health conditions that were not already included in the Overview Report (the 24/40 trials dismissed).
The phrase ‘self-selected sample’ refers (correctly) to the fact that submitted trials are subject to ‘self-selection bias’ i.e. having been chosen by pro-homeopathy organisations or individuals, the studies are more likely to be positive and may not accurately reflect the wider body of evidence available on the topic. This is supported by ARCH’s assessment that 37 of 40 of the submitted studies reported benefits from homeopathy37p.13.
However, this bias could easily have been removed by carrying out a systematic literature search for all other trials on the same topic as those submitted, but NHMRC chose not to do this.
For example, one of the 9 trials submitted by the homeopathy sector that was ‘downgraded’ due to self-selection bias was a pilot study (N=81) in which the original authors state that they found individualised homeopathic treatment to be as effective as conventional care for the treatment of otitis media in children. For this study not to be at risk of bias from being self-selected, NHMRC would only have needed to carry out a literature search for studies on homeopathy for the treatment of otitis media in children, published more recently than those captured in the systematic reviews they had already assessed. This would have taken a matter of hours at most.
NHMRC refers to this as a ‘limitation’ of the Overview Report:
This means that due to NHMRC’s own chosen methods, it was not possible for any of the 49 submitted studies to affect the Review results; even if NHMRC assessed a submitted trial as being both ‘reliable’ and ‘positive’ (i.e. showing that homeopathy is effective), because it was assessed as a ‘self-selected sample’ and thus having a high ‘risk of bias’, it would not be “considered further”.
In terms of public perception of NHMRC’s process, it is interesting to note that NHMRC mention that they considered submissions of evidence by homeopathy interest groups six times in the 40 page Information Paper; by contrast, the fact that all their submitted evidence was excluded from the Overview due to being categorised as self-selected samples, is not mentioned at all in the Information Paper.
It was NHMRC’s own choice not to systematically search for and identify all individual trials on homeopathy which met their criteria, but were not present in SRs, relying instead on external submissions. It is therefore astonishing to find that they then use this lack of systematic search as a reason to dismiss all trials submitted by homeopathy interest groups and the majority of studies submitted via public consultation, making a sham of this apparent attempt at external co-operation and transparency.
NHMRC appoints three expert reviewers without any homeopathy subject/ research expertise:
None of the three expert reviewers NHMRC appointed in 2014 to provide expert peer review “to ensure that the evidence had been accurately interpreted” (IP, p.15) had any subject or research expertise in homeopathy.
FOI returns reveal that a HWC member protested the lack of homeopathy discipline/ research expertise amongst the reviewers selected. At the meeting, ONHMRC noted that if a homeopathy research expert was appointed as one of the reviewers, their comments ‘may have to be published’ - voicing NHMRC’s concern that a subject expert would identify legitimate problems (a clear expression of bias).
The exclusion contravened NHMRC's own guidelines on the appointment of expert reviewers (impacting the credibility of the process). The NHMRC ‘Nomination of Independent Expert Clinical Reviewers for Externally Developed Clinical Practice Guidelines’ clearly stipulates:
This was consistent with NHMRC having ignored the requirements of its own 'quality assurance process', which mandates that members with subject expertise must be included in NHMRC expert committees (as above).
'Health Technology Assessment' label created post hoc, in mid 2014:
FOI returns reveal that post-public consultation phase, the HWC decided to retrospectively describe the Review as a ‘Health Technology Assessment’, for the specific purpose of attempting to counter criticism received from public consultation feedback regarding the exclusion of homeopathy subject and research experts from the Review process.
NHMRC had not referred to the Review as a ‘HTA,’ or even “like” a HTA at any point prior to June 2014: the argument was fabricated, inaccurate and highly misleading to the public and other stakeholders.
In the first instance, NHMRC was explicitly clear that it conducted an ‘Overview’ under the Review’s Terms of Reference, not a ‘HTA’, using a method “derived from Chapter 22 of the Cochrane Handbook of Systematic Reviews of Interventions” (Overview Report, p.12; IP, p.11). These are entirely different research processes conducted for different purposes.
HTA is a form of policy research that examines short- and long-term consequences of the application of a health-care technology. Properties assessed include evidence of ‘safety’, ‘efficacy’, ‘patient-reported outcomes’, ‘real-world effectiveness’, ‘cost’, and ‘cost-effectiveness’ as well as social, legal, ethical, and political impacts. The Department of Health’s definition of HTA is consistent with this.
NHMRC excluded all these customary HTA parameters from scope of the Review, except ‘efficacy’. Thus, NHMRC’s ‘Overview’ bears no resemblance to a HTA.
The final report also attempted to describe the HWC as a ‘HTA committee’ (NHMRC Administrative Report, p.5), to align it with ‘other HTA committees’ that NHMRC misleadingly and inaccurately claimed ‘also do not contain any subject experts’ - a claim that is patently incorrect:
The HWC was not a HTA committee and the ‘other HTA committees’ NHMRC provided as examples in fact contain a wide array of subject and clinical experts in their respective fields - as required by their terms of reference (e.g. the Therapeutic Goods Administration’s Advisory Committee on Complementary Medicines).
In light of the background information presented on this page, one can begin to judge the accuracy and integrity behind statements NHMRC presented as authoritative and true in its reporting of the Review to the public, for example (IP, p.38):
Whereas the Review was not a HTA and the null hypothesis was retrospectively introduced to 'assume homeopathy is ineffective unless 'reliable' evidence shows otherwise'; with NHMRC then embarking on a process of developing the criteria that defined its unique concept of 'reliable' evidence, which was then retrospectively applied to the data resulting in 171/176 studies being dismissed from consideration in the findings.
 Minutes of the first meeting of the Homeopathy Working Committee (HWC), 12 April 2012 (unredacted). NHMRC FOI 2015/16 012-04
 NHMRC Official Order (Statement of Requirement for First Contractor), 4 April 2012. NHMRC FOI 2014/15 021-04
 Expert reviewer feedback to NHMRC. NHMRC FOI 2014/15 004-Section 62
 Homeopathy Working Committee meeting minutes, 30 July 2014. NHMRC FOI 2015/16 008-11
 Email from ONHMRC to HWC members and Optum: finalised research protocol for the Homeopathy Review. NHMRC FOI 2014/15 004-Sections 58-60
 Minutes of HWC meeting, 18 March 2013. NHMRC FOI 2015/16 007-03
 NHMRC Homeopathy Working Committee meeting minutes, 2013-2014. NHMRC FOI 2015/16 008-Docs 1-13
 Email from ONHMRC to HWC members, 30 Apr 2013 re. revised approach to homeopathy evidence statements. NHMRC FOI 2014/15 021-11
 HWC Sub-Group Teleconference meeting, 6 May 2013. NHMRC FOI 2015/16 008-03
 HWC Sub-Group Teleconference meetings, 29 Apr & 6 May 2013. NHMRC FOI 2015/16 008-02 & 03
[10A] NHMRC FOI 2015/16 008-13; NHMRC FOI 2015/16 007-05
 HWC Sub-Group Teleconference meeting, 29 Apr 2013. NHMRC FOI 2015/16 008-02
 HWC Sub-Group Teleconference meeting, 24 May 2013. NHMRC FOI 2015/16 008-05
 HWC Face-to-face meeting, 11-12 Jul 2013. NHMRC FOI 2015/16 008-06
 Independent analysis of NHMRC's methods, in Complaint to the Commonwealth Ombudsman.
 Deschartres et al (2013). Influence of trial sample size on treatment effect estimates, 24 April 2013. bmj.f2304.fullwww.bmj.com/content/346/bmj.f2304
 NHMRC response to Complementary Medicines Australia re. 30.09.15 preliminary stakeholder submission of complaint to NHMRC, 22 Dec 2015.
 NHMRC Information Paper, p.34 & 37; Australasian Cochrane Centre second round methodological review, 30 Aug 2013. NHMRC 2015/16 008-13.
 Australasian Cochrane Centre second round methodological review, 30 Aug 2013. NHMRC 2015/16 007-05
 Expert peer reviewer feedback to NHMRC. NHMRC FOI 2014/15 004-Section 62
 Stam, C., Bonnet, M. S. & van Haselen, R. A. (2001). ‘The efficacy and safety of a homeopathic gel in the treatment of acute low back pain: a multi-centre, randomised, double-blind comparative clinical trial’, The British Homoeopathic Journal, 90:21-28
 Pach, D. et al. (2011). ‘Efficacy of injections with Disci/Rhus toxicodendron compositum for chronic low back pain--a randomized placebo-controlled trial’, PloS One, 6:e26166