Methodological flawsPublic misled that there are 'no good quality studies':
The 11 March 2015 NHMRC media release misinformed the public that [1]:
The NHMRC Information Paper refers to the homeopathic studies considered being 'poor quality' 40 times in 40 pages - strongly (over) emphasising this message; unusual in science reporting. 44 of 176 trials evaluated by NHMRC were categorised as ‘quality unreported’, meaning that it is unknow whether quality has been correctly assessed (as being poor) for a quarter of the evidence in the Homeopathy Review [2]. Instead of checking any of the original studies to fill in this large gap in the data (a relatively simple task), the HWC instead created an arbitrary ‘rule’ to "assume" these studies were 'poor quality’ [3]. NHMRC’s published finding of 'no good quality evidence' therefore represents mere opinion; it is not evidence-based and the public have been misled. Making such large-scale 'assumptions' about data then making definitive conclusions on the basis of what amounts to guesswork, does not equate to 'rigorous' scientific analysis as claimed. This is a bizarre practice employed by an organisation that describes itself as (response to stakeholders, 22 Dec 2015):
Use of an inappropriate method: To assess the evidence, NHMRC decided to conduct an 'Overview', referenced against Chapter 22 of the Cochrane Handbook [4]. The Cochrane 'Overview' method is intended to summarise the findings of systematic reviews (SRs), which themselves summarise the findings of original studies. Thus, the Overview approach is a 'summary of summaries'. For this reason, the Overview approach is designed to evaluate high quality Cochrane Intervention reviews, or SRs of equivalent high quality. NHMRC/ HWC identified 57 SRs for the Overview, summarising the findings of 176 studies. NHMRC did not report that it only rated 7 out of the 57 (12%) of the SRs it identified as being sufficiently robust to be suited to the Cochrane ‘Overview’ method referenced. An inappropriate method was therefore used for the Overview. Freedom of Information (FOI) returns reveal that the first (terminated) review it conducted between April & August 2012 (see 'The First Review') did screen the data to excluded poor quality SRs, as per the intent of the Cochrane Overview method (emphasis added) [5]:
Irrespective, for the second (Optum) review NHMRC proceeded to make definitive conclusions on the basis of secondary sources it predominantly rated as not fit for purpose. Because the SRs only summarised original studies, they were significantly limited by incomplete and/or missing data. While NHMRC acknowledged this 'limitation' (IP, p. 25), it did not quantify or report the major impact it had on the analysis (as detailed in this and the 'Procedural irregularities' section). NHMRC did not apply the same caution when drafting definitive conclusions on the basis of these predominantly ‘substandard’, inconclusive sources. It is unclear how this constituted a 'rigorous' approach using 'internationally accepted methods for assessing health evidence' claimed. Limitations of approach - incomplete & missing data: Arguably the most serious ‘limitation’ NHMRC encountered in trying to apply the Overview method to SRs of insufficient quality was that essential data about the original studies described in the 57 SRs was either inconsistent across different SRs or entirely missing. This is because it is not the purpose of SRs to include the type of information required to assess primary studies (noted in undisclosed expert peer review feedback to NHMRC on 9 July 2013). NHMRC acknowledged this as a 'limitation' of the Review methodology (IP, p.25) - but did not quantify in its report the major impact this ‘limitation’ had on the analysis, lacking transparency. For example, the SRs made no mention of the quality of 44 out of the 176 included studies (as above), which were then simply "assumed" to be not god quality.. By not publishing such quantitative analysis, NHMRC concealed the significant impact from public and expert scrutiny, which if disclosed would have raised serious doubts about the Review's credibility. Expert peer reviewer criticism of NHMRC's approach withheld: The 'Overview' approach was an unprecedented method in NHMRC evidence reviews, representing a 'short-cut' method - as it meant NHMRC did not need to retrieve or assess any original studies. The NHMRC Information Paper (p.25) informed: “The overview was based on finding systematic reviews of homeopathy, rather than searching for all individual published studies of homeopathy." This was criticised by two out of three expert reviewers in 2014. One reviewer strongly criticised the approach as a breach of established NHMRC best-practice - feedback that was not disclosed by NHMRC, despite publishing a dedicated 'Expert review comments' document (revealed through FOI returns) [6]: 
'Overview' method intended to summarise and report the findings of SRs - yet were dismissed:
NHMRC ignored perhaps the most useful information provided by the SRs: their conclusions. Yet the main purpose of an 'Overview' is to summarise the findings of high quality SRs, to provide researchers with an 'overview' of research findings in a particular area. The NHMRC Information Paper (p.25) even informed:
Yet incongruously, the HWC dismissed the findings of all 57 SRs, despite 'the work having already been done' by SR authors who did assess the original studies and the purpose of an 'Overview' being to report their findings. Information Paper (p.11):
Thus NHMRC ignored the work done by the SR authors, yet re-assessed the original studies without retrieving or assessing any of them, instead relying on incomplete/ missing information about these studies as provided in the SRs. This is a bizarre way of assessing evidence, which does not align with any accepted research method, not even the Cochrane guideline referenced (Chapter 22 of the Cochrane Handbook stipulating:
A number of the 57 SRs identified reported positive findings of the evidence on homeopathy in their assessment of the original studies, as one of NHMRC 2014 expert reviewers informed NHMRC in (undisclosed) feedback critiquing the approach (NHMRC FOI 2014/15 004-Section 62):
Accepted NHMRC research methods abandoned midstream: The NHMRC Information Paper announcing the findings of the Overview assured the public that in the Review NHMRC employed:
But this was not the case. The Information Paper did not inform the community that midway through the Overview, NHMRC abandoned its established, accepted method of reviewing health evidence as intended in the original research protocol, once it became apparent that the secondary data it exclusively relied upon (SRs) was not sufficiently robust, due to ‘limitations’ associated with missing and incomplete data. This fact was not mentioned anywhere in the Information Paper - only on p.20 of the Optum Overview Report - thus remaining unnoticed by the media. The formal investigation into the NHRMC Homeopathy Review has also uncovered that almost none of the criteria NHMR applied to assessing the evidence were 'standardised, accepted methods' for reviewing evidence (see below and 'Procedural irregularities' section). Adoption of arbitrary criteria never used before: One of the most serious methodological anomalies associated with the Homeopathy Review was the creation of a unique concept of 'reliable evidence', which was underpinned by arbitrary criteria never used before or since by any other research group (including NHMRC). For a trial to be considered 'reliable', it had to have BOTH:
NHMRC regularly funds and collaborates in trials with far fewer than 150 participants and no published Cochrane Overview (whose guideline NHMRC referenced) has used 'quality rating' scales as a threshold for trial 'reliability'; also accepting a quality rating of 3/5 on the Jadad scale as 'good quality'. 
How the N=150 sample size exclusion threshold impacted the findings:
The following illustrates how the N=150 sample size exclusion threshold was used to dismiss the findings of positive, good quality trials in several medical conditions, to conclude 'no reliable evidence': 
Figure 1: The full evidence base for individualised homeopathic treatment of children with diarrhoea
Figure 2: Evidence base for isopathic treatment of allergic rhinitis (all placebo-controlled RCTs)
*Corrected sample size differs from NHMRC data
Figure 3: Evidence base for non-isopathic homeopathic treatments for allergic rhinitis.
*Corrected sample size differs from NHMRC data
Figure 4: Evidence base for non-isopathic homeopathic treatments for allergic rhinitis.
*Corrected sample size differs from NHMRC data
Bias in reporting of trials:
As shown in the graphics above, NHMRC’s method of assessing the evidence base led to the unjustified dismissal of a number of positive trials. Many of these were also 'good quality'. Example: evidence base for isopathic homeopathic treatments for allergic rhinitis: The way in which NHMRC assessed the evidence for isopathic treatments for allergic rhinitis (hay fever) (Figure 2) illustrates how NHMRC used the N=150 sample size threshold, alongside other unusual criteria, to dismiss good quality positive trials from being considered 'reliable' and hence being excluded from the findings:
NHMRC’s conclusion that there is ‘no reliable evidence’ for the effectiveness of homeopathy for people with allergic rhinitis is inaccurate, denying the existence of good quality studies that suggest several potentially effective treatments for hay fever sufferers i.e. Pollen C30, individualised immunotherapy, Galphimia glauca and Luffa compositum nasal spray. Therefore it is patent that NHMRC did not provide an accurate or fair assessment of the evidence on homeopathy for allergic rhinitis and its conclusion is highly misleading. The same applies to several other medical conditions assessed (such as those indicated in Figures 1-4 above). Similarly misleading and inaccurate analyses of these and other conditions made by NHMRC/ HWC is detailed in the complaint to the Ombudsman. Exclusion criteria dismiss the results of 97% of the data from any consideration in the findings: Neither the 'sample size' nor the 'quality rating' exclusion criteria align with any known 'accepted research methods' (either individually or in combination) - as NHMRC assured the public it used. Most damning, together these criteria dismissed the results of 171 out of the 176 (97%) Overview studies from being considered as part of the Review's findings (at all) - a fact not reported by NHMRC in over 944 pages of report documentation. The graphics at right show the impact that the N=150 exclusion threshold had on the evidence base in several medical conditions (see also 'Procedural' section). Worse still, FOI documents reveal these criteria were developed and applied entirely post-hoc: Research protocol retrospectively modified: Another serious methodological/procedural issue identified during the investigation into NHMRC's conduct in the Homeopathy Review, was extensive changes made to the research protocol by the Office of NHMRC (ONHMRC)/ Homeopathy Working Committee (HWC) throughout the Optum review - well after the protocol had been "finlaised" in December 2012 and Optum commenced their review on 3 January 2013. The Optum Overview Report (p.12) notes the protocol was further “developed through active consultation between the members of the HWC and the evidence review team” - with none of the specific changes to the protocol disclosed. The final version of the protocol was only officially confirmed in July 2013 and the method of assessing evidence continued to be modified up until August 2013 – well after Optum’s evidence assessment had already been completed. Thus the changes were made retrospectively: after the original protocol had been finalised and after the contractor (Optum) had already completed its evidence assessment - removing probably the most important safeguard reducing/preventing reporting bias in scientific studies. Making post-hoc changes to the research protocol is a serious ethical issue in scientific inquiry, as it exposes such processes to risk of bias (the intentional or unintentional tailoring of the outcome according to pre-determined expectations). More details of these post hoc changes to the protocol are provided in the 'Procedural irregularities' section - summarised below (revealed through Freedom of Information returns referenced in the 'Procedural' section):
Notably, all of these changes:
Changes to the research protocol not reported: A pre-requisite of ethical scientific inquiry is that all changes to the research protocol must be fully disclosed and justified. None of the key changes to the research protocol outlined above were reported or justified. Agreeing a protocol before a review process starts is a recognised safeguard against scientific bias, so making such significant post-hoc changes fundamentally undermines the credibility of the Review results. Extensive study exclusion criteria applied to the Homeopathy Review: The 11 March 2015 NHMRC media release announcing the findings of the Homeopathy review misinformed the public that the Review's findings and conclusions were:
While only focussing on assessment of efficacy data (i.e. randomised controlled trials) is not unusual or unreasonable when reviewing health evidence, the following aspects were unusual:
NHMRC misled the public that its Review was 'thorough' and 'comprehensive' [1], when it wasn't. Poor quality reporting - Lack of consistencies & errors: Some minor inaccuracies due to human error are to be expected in a large report, but the pervasive lack of clarity, combined with the quantity of errors and inconsistencies throughout the Homeopathy Review call into question its fundamental credibility. NHMRC’s failure to provide their quality assessments for each trial, nor to provide any details of their ‘level of confidence’ calculations using their ‘adapted GRADE’ method, have both been mentioned earlier; these reporting flaws need to be reiterated here as they are unusual and serious omissions. The inappropriate use of the Cochrane Overview method for the specific data set involved in the Review has been outlined above, but the extent of the errors created by this flaw becomes apparent when NHMRC’s summary of the evidence base for a specific clinical condition is examined closely: Example: Sinusitis chapter in Overview Report: For example, in the Sinusitis chapter (Optum Overview Report, p.203), which only describes three studies, we find that:
As the above errors all relate to a section of the Overview Report that is only one page in length, analysing the full extent of this problem across a Review comprising over 600 pages is beyond the scope of the complaint to the Ombudsman, however multiple examples identified throughout the Review show that this problem is not confined to the Sinusitis chapter, raising serious questions about the accuracy of NHMRC’s findings in general. If the HWC had included a topic expert, as is usual in such Reviews, their specialist knowledge of the homeopathy research field would have significantly reduced such errors (see 'Procedural irregularities'). For an organisation that describes itself as “an established leader in the development of high quality, evidence-based health advice” [19], the lack of clarity, multiple inconsistencies and simple errors found in this report raise the question of whether such poor quality work was intentional or unintentional. Certainly, the result is the production of a document with such obfuscation of the data that we contend it would not have passed peer review for external publication: the Homeopathy Review should therefore be set aside in its entirety, as academically it is not fit for purpose. Critical methodological flaw – comparing incompatible intervention types: The inaccuracy of the NHMRCs conclusions are also reflected in another fundamental flaw at the heart of the report – i.e. considering the results of all trials for one condition together as a whole, even though the individual trials were assessing very different types of homeopathic treatment. To illustrate this flaw, the NHMRC reviewers asked, “Is homeopathy effective for condition A?’, working from the premise that a positive trial showing that one homeopathic treatment is effective is somehow negated by a negative trial which shows that a completely different homeopathic treatment for that same condition is ineffective. This is a bizarre and unprecedented way of assessing scientific evidence (referred to as ‘heterogeneity bias’, as per Cochrane guidelines that NHMRC referenced). In conventional research the question asked would be, “Is treatment X effective for condition A?”, not “Is conventional medicine effective for condition A?” based on combining the results of all drug trials together. Some treatments work, some don’t. The whole point of medical research is to establish which treatments are useful and which are of no value. This is no different in homeopathy. Ignoring such a basic premise of scientific assessment is not a hallmark of rigour, but a limitation of the short-cut method adopted by NHMRC (due to no original studies being retrieved or assessed during the Review, with the secondary sources relied upon being limited by incomplete and missing information) [20]. [1] 'NHMRC releases statement and advice on homeopathy'. NHMRC media release, 11 Mar 2015 [2] Investigation into NHMRC’s methods & procedures, detailed in Ombudsman Complaint [3] NHMRC Information Paper, p.35 [4] Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011 [5] Internal ONHMRC correspondence. NHMRC FOI 2015/16 012-09 [6] Expert peer reviewer feedback to NHMRC, May 2014. NHMRC FOI 2014/15 004-Section 62 [7] Reilly, D. T., Taylor, M. A., McSharry, C. & Aitchison, T. (1986) Is homoeopathy a placebo response? Controlled trial of homoeopathic potency, with pollen in hayfever as model, Lancet (London, England), 2:881-886 [8] Taylor, M. A., Reilly, D., Llewellyn-Jones, R. H., McSharry, C. & Aitchison, T. C. (2000) Randomised controlled trial of homoeopathy versus placebo in perennial allergic rhinitis with overview of four trial series, BMJ (Clinical research ed.), 321:471-476 [9] Linde, K. et al. (1997) Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trials, Lancet, 350:834-843 [10] Ernst, E. (2011) Homeopathic Galphimia glauca for hay fever: a systematic review of randomised clinical trials and a critique of a published meta-analysis, Focus on Alternative and Complementary Therapies, 16:200-203 [11] Weiser, M., Gegenheimer, L. H. & Klein, P. (1999) A randomized equivalence trial comparing the efficacy and safety of Luffa comp.-Heel nasal spray with cromolyn sodium spray in the treatment of seasonal allergic rhinitis, Forschende Komplementärmedizin, 6:142-148 [12] Bellavite, P., Marzotto, M., Chirumbolo, S. & Conforti, A. (2011) Advances in homeopathy and immunology: a review of clinical research, Frontiers in Bioscience (Scholar Edition), 3:1363-1389 [13] Zabolotnyi, D. I. et al. (2007) Efficacy of a complex homeopathic medication (Sinfrontal) in patients with acute maxillary sinusitis: a prospective, randomized, double-blind, placebo-controlled, multicenter clinical trial, Explore (New York, N.Y.), 3:98-109 [14] Weiser, M. & Clasen, B. (1994) Controlled double-blind study of a homoeopathic sinusitis medication, Biological Therapy, 13:4-11 [15] Wiesenauer, M., Gaus, W., Bohnacker, U. & Häussler, S. (1989) [Efficiency of homeopathic preparation combinations in sinusitis. Results of a randomized double blind study with general practitioners], Arzneimittel-Forschung, 39:620-625 [16] Friese, K.-H. & Zabalotnyi, D. I. (2007) [Homeopathy in acute rhinosinusitis: a double-blind, placebo controlled study shows the efficiency and tolerability of a homeopathic combination remedy], HNO, 55:271-277 [17] Ammerschläger, H., Klein, P., Weiser, M. & Oberbaum, M. (2005) [Treatment of inflammatory diseases of the upper respiratory tract -- comparison of a homeopathic complex remedy with xylometazoline], Forschende Komplementärmedizin Und Klassische Naturheilkunde = Research in Complementary and Natural Classical Medicine, 12:24-31 [18] Weiser, M. & Clasen, B. (1994) Randomized, placebo-controlled, double-blind study of the clinical efficacy of the homeopathic Euphorbium compositum-S nasal spray in cases of chronic sinusitis, Forschende Komplementärmedizin, 1:251-259 [19] NHMRC response to stakeholder correspondence, 22 December 2015 [20] Homeopathy Research Institute (HRI) response to NHMRC Homeopathy Review, 12.3.2015 |
“…it is important that the methods to be used should be established and documented in advance. |