Undisclosed expert peer reviewer feedback:
“The dismissal of positive systematic reviews compounded with the lack of an independent systematic review of high quality randomised controlled trials leaves me uncertain of the definitive nature of the Report’s conclusions." "For example, when a substantial proportion of small (but good quality) studies show significant differences, […] 'no reliable evidence' does not seem an accurate reflection of the body of evidence." |
|
NHMRC 'quality assurance process' - methodological and expert peer review:
When conducting evidence reviews and/or clinical guideline development processes, the NHMRC is required to seek feedback from external peer reviewers as part of its 'quality assurance process', the requirements of which are specified in the NHMRC Procedures and requirements for meeting the 2011 NHMRC standard for clinical practice guidelines (the '2011 NHMRC Standard') referenced against the Review.
NHMRC sought expert reviewer feedback at the following two stages of the Review process:
These expert peer reviewers provided NHMRC with critical feedback that was either not reported (i.e. the ACC advice) or extensively redacted for the purpose of public disclosure. This was despite the NHMRC's 'transparency' assurance, that it had engaged in external peer review to 'ensure a rigorous approach had been taken' (as reported in the NHMRC Administrative Report and Information Paper).
Expert reviewers' feedback and specific comments have been obtained through Freedom of Information (FOI) documents, revealing highly relevant feedback questioning the Review's methods and findings:
Homeopathy Review's findings does not pass external peer review - not reported:
FOI documents have revealed that on 30 August 2013, the ACC advised the NHMRC/ HWC that the definitive nature of the Review's conclusions 'did not accurately reflect the research' as follows [4]:
When conducting evidence reviews and/or clinical guideline development processes, the NHMRC is required to seek feedback from external peer reviewers as part of its 'quality assurance process', the requirements of which are specified in the NHMRC Procedures and requirements for meeting the 2011 NHMRC standard for clinical practice guidelines (the '2011 NHMRC Standard') referenced against the Review.
NHMRC sought expert reviewer feedback at the following two stages of the Review process:
- 2013: NHMRC contracted the Australasian Cochrane Centre (ACC) in July and August 2013 to conduct "an independent Methodological Review of the Overview Report and its supporting documents to confirm that an appropriate and rigorous approach had been taken" [1].
- 2014: NHMRC sought advice from three external expert reviewers "to ensure that the evidence had been appropriately interpreted and synthesised" [2]. None of these expert reviewers had any subject or research expertise in homeopathy, in breach of NHMRC guidelines on the appointment of expert reviewers [3].
These expert peer reviewers provided NHMRC with critical feedback that was either not reported (i.e. the ACC advice) or extensively redacted for the purpose of public disclosure. This was despite the NHMRC's 'transparency' assurance, that it had engaged in external peer review to 'ensure a rigorous approach had been taken' (as reported in the NHMRC Administrative Report and Information Paper).
Expert reviewers' feedback and specific comments have been obtained through Freedom of Information (FOI) documents, revealing highly relevant feedback questioning the Review's methods and findings:
Homeopathy Review's findings does not pass external peer review - not reported:
FOI documents have revealed that on 30 August 2013, the ACC advised the NHMRC/ HWC that the definitive nature of the Review's conclusions 'did not accurately reflect the research' as follows [4]:
It is important to note that the purpose of the Review was to 'inform the public of the research on homeopathy', and not to develop clinical guideline recommendations (as confirmed by the NHMRC, e.g. Administrative Report, pp.5-6).
Also noteworthy is that the ACC acknowledged that "a substantial proportion of small (but good quality) studies show significant differences", in direct contrast to the NHMRC's overly definitive conclusion of "no reliable evidence".
FOI documents also reveal that in May 2014, one of NHMRC's expert peer reviewers also advised NHMRC/ HWC [5]:
This feedback was not disclosed in the NHMRC 'Expert reviewer comments' document published on 11 March 2015 alongside the NHMRC Information Paper and Statement on Homeopathy. In fact, 80% of the reviewer's expert feedback, which was highly critical of the Review's methods and findings, was withheld. It is unclear how this represents 'transparency'.
Instead, the NHMRC provided the following synopsis of its expert consultation process (Information Paper, p.15):
Also noteworthy is that the ACC acknowledged that "a substantial proportion of small (but good quality) studies show significant differences", in direct contrast to the NHMRC's overly definitive conclusion of "no reliable evidence".
FOI documents also reveal that in May 2014, one of NHMRC's expert peer reviewers also advised NHMRC/ HWC [5]:
- "The dismissal of positive SRs compounded with the lack of an independent systematic review of high quality RCTs leaves me uncertain of the definitive nature of the Report’s conclusions. It would have been worthwhile looking closely at RCTs of individualised homeopathy, as this is an essential premise of practice. [...]
if I were to dispassionately consider the evidence of efficacy, I am still left with niggling doubts that there are unanswered questions around the evidence.
Contributing to those doubts is the absence of non-clinical in vivo and in vitro studies that examine the effects of homeopathy, where placebo effects are not relevant."
This feedback was not disclosed in the NHMRC 'Expert reviewer comments' document published on 11 March 2015 alongside the NHMRC Information Paper and Statement on Homeopathy. In fact, 80% of the reviewer's expert feedback, which was highly critical of the Review's methods and findings, was withheld. It is unclear how this represents 'transparency'.
Instead, the NHMRC provided the following synopsis of its expert consultation process (Information Paper, p.15):
- 2013 ACC advice: "All the reviewer’s comments and suggestions were considered in consultation with the Homeopathy Working Committee and the report amended accordingly."
- 2014 Expert reviewer feedback: "All comments received were considered by the Homeopathy Working Committee and the draft Information Paper was amended accordingly."
Following are other specific examples of undisclosed expert reviewer feedback (from FOI documents):
Australasian Cochrane Centre methodological review feedback - 9 July 2013:
On 9 July 2013, the ACC provided first-round feedback on NHMRC's approach. The ACC advice was not published, but obtained through FOI [6]. At this stage of the Review, the "reliable evidence" framework that ultimately informed the findings (e.g. '150 trial sample size' and '100% quality rating' trial exclusion thresholds) had not yet been formulated - thus the ACC could not critically evaluate it.
Cochrane noted that the phrasing of the evidence statements did not reflect the uncertainty in the research results and were overly negative (advice reiterated in the second round of ACC feedback on 30 Aug 2013, below).
Australasian Cochrane Centre methodological review feedback - 9 July 2013:
On 9 July 2013, the ACC provided first-round feedback on NHMRC's approach. The ACC advice was not published, but obtained through FOI [6]. At this stage of the Review, the "reliable evidence" framework that ultimately informed the findings (e.g. '150 trial sample size' and '100% quality rating' trial exclusion thresholds) had not yet been formulated - thus the ACC could not critically evaluate it.
Cochrane noted that the phrasing of the evidence statements did not reflect the uncertainty in the research results and were overly negative (advice reiterated in the second round of ACC feedback on 30 Aug 2013, below).
The ACC also noted that there appeared to be changes to the originally agreed research protocol affecting the way the evidence was assessed:
NHMRC did not heed the ACC's advice to "clearly describe changes to the protocol". The 'Changes to the Protocol' section Optum included in the Overview Report (p.20) did not disclose any of the key changes that formed the basis of the Review's evidence statement criteria and published findings (see 'Post-hoc protocol changes' sections); the Information Paper also failed to disclose them.
In the Homeopathy Review, NHMRC informed the public that:
This refers to NHMRC's 'dimensions of evidence assessment' framework (noted by the ACC above), developed by the 'Australian Experts' referred to - one of whom was NHMRC's terminated first reviewer - which was also specified in the original protocol. Midway through the Review, the NHMRC realised that the secondary sources (systematic reviews, SRs) it solely relied upon did not contain required primary trial data to be able to apply NHMRC's 'rigorous approach', since no original studies were retrieved or assessed. The Information Paper did not report this.
The ACC corrected NHMRC on its 'unreasonable' criticism of the lack of required information provided by the SRs, advising:
In the Homeopathy Review, NHMRC informed the public that:
- "When evaluating health evidence and drafting health advice, NHMRC uses a rigorous approach that has been developed by Australian experts in research methods." (IP, p.9)
This refers to NHMRC's 'dimensions of evidence assessment' framework (noted by the ACC above), developed by the 'Australian Experts' referred to - one of whom was NHMRC's terminated first reviewer - which was also specified in the original protocol. Midway through the Review, the NHMRC realised that the secondary sources (systematic reviews, SRs) it solely relied upon did not contain required primary trial data to be able to apply NHMRC's 'rigorous approach', since no original studies were retrieved or assessed. The Information Paper did not report this.
The ACC corrected NHMRC on its 'unreasonable' criticism of the lack of required information provided by the SRs, advising:
The problem rested not with the SRS, but with the NHMRC's choice of an inappropriate method, which was to draw definitive conclusions about original studies by relying on incomplete/ missing information in secondary summary sources (systematic reviews) - which cannot be done.
Further unreported 9 July ACC feedback can be accessed from the NHMRC FOI website - see FOI 2015/16 008 Document 13.
Further unreported 9 July ACC feedback can be accessed from the NHMRC FOI website - see FOI 2015/16 008 Document 13.
Australasian Cochrane Centre methodological review feedback - 30 August 2013:
NHMRC also did not release/ publish the second-round ACC feedback dated 30 August 2013, also obtained through FOI [4]. The ACC reiterated its concerns over the overly-definitive nature of the findings (as above), as well as noting further changes to the research protocol and other issues:
'Quality rating scale' threshold:
There is no mention of the use of quality rating scales as a threshold prior to the second round ACC peer review. This criterion was developed and applied to the data some time in late July/ early August. The ACC advised:
NHMRC also did not release/ publish the second-round ACC feedback dated 30 August 2013, also obtained through FOI [4]. The ACC reiterated its concerns over the overly-definitive nature of the findings (as above), as well as noting further changes to the research protocol and other issues:
'Quality rating scale' threshold:
There is no mention of the use of quality rating scales as a threshold prior to the second round ACC peer review. This criterion was developed and applied to the data some time in late July/ early August. The ACC advised:
'Adapted GRADE' tool:
The 9 July 2013 ACC feedback raised specific concerns regarding the 'adapted GRADE' tool developed by the ONHMRC to assess the level of confidence in trial results [4]. This was referred to as 'the process for aggregating results across multiple outcomes in a trial' (which was not possible, since the Overview did not assess any original studies and the secondary systematic review sources did not report the required details about the original studies for this to be done.
The 30 August 2013 follow-up ACC advice reiterated these concerns, which had not been addressed:
This was reiterated in the 30 August 2013 ACC report a number of times under separate items, for example:
The 9 July 2013 ACC feedback raised specific concerns regarding the 'adapted GRADE' tool developed by the ONHMRC to assess the level of confidence in trial results [4]. This was referred to as 'the process for aggregating results across multiple outcomes in a trial' (which was not possible, since the Overview did not assess any original studies and the secondary systematic review sources did not report the required details about the original studies for this to be done.
The 30 August 2013 follow-up ACC advice reiterated these concerns, which had not been addressed:
- "Derivation of evidence statements: The process for aggregating results across multiple outcomes within a trial, and the approach taken to incorporating meta-analytical findings with those from individual studies, is still unclear."
This was reiterated in the 30 August 2013 ACC report a number of times under separate items, for example:
Expert reviewer feedback, 2014:
On 9 April 2014, the NHMRC released its Draft Information Paper and the Optum Overview Report for public consultation, inviting members of the public and other stakeholders to submit extra research evidence that may have been missed. As part of the public consultation process, NHMRC sought advice from three external expert reviewers, to fulfil the requirements of its 'quality assurance' process.
Alongside release of the final report on 11 March 2015, NHMRC published an 'Expert review comments' document, which summarised comments provided by three expert reviewers in May 2014. One expert reviewer in particular [5] was highly critical of the Review's methods and findings, however the significant majority of their feedback was withheld from disclosure in the NHMRC 'Expert review comments' document.
Following are excerpts of the expert peer reviewer's feedback, 80% of which (by word count) was not disclosed. Some of this feedback reiterates concerns raised by the ACC concerning the way in which the 'quality' of homeopathic trials were assessed using 'problematic' criteria, as well as disagreeing with the definitive nature of the Review's findings:
[1] NHMRC Administrative Report, p.8
[2] NHMRC Administrative Report, p.9
[3] NHMRC Nomination of Independent Expert Clinical Reviewers for Externally Developed Clinical Practice Guidelines v2_140219
[4] Australasian Cochrane Centre Methodological Review feedback to NHMRC, 30 August 2013. NHMRC FOI 2015-16 007-05
[5] Expert reviewer feedback to NHMRC, May 2014. NHMRC FOI 2014/15 004-Section 62
[6] Australasian Cochrane Centre Methodological Review feedback to NHMRC, 9 July 2013. NHMRC FOI 2015-16 008-13
On 9 April 2014, the NHMRC released its Draft Information Paper and the Optum Overview Report for public consultation, inviting members of the public and other stakeholders to submit extra research evidence that may have been missed. As part of the public consultation process, NHMRC sought advice from three external expert reviewers, to fulfil the requirements of its 'quality assurance' process.
Alongside release of the final report on 11 March 2015, NHMRC published an 'Expert review comments' document, which summarised comments provided by three expert reviewers in May 2014. One expert reviewer in particular [5] was highly critical of the Review's methods and findings, however the significant majority of their feedback was withheld from disclosure in the NHMRC 'Expert review comments' document.
Following are excerpts of the expert peer reviewer's feedback, 80% of which (by word count) was not disclosed. Some of this feedback reiterates concerns raised by the ACC concerning the way in which the 'quality' of homeopathic trials were assessed using 'problematic' criteria, as well as disagreeing with the definitive nature of the Review's findings:
- "The Information Paper explains how the evidence for effectiveness was examined, however on one issue I found it a little confusing. The principal approach was to review reviews (p6), however prospective controlled studies were implied as being reviewed (last para p6), but only seem to have been reviewed if they were identified by external submissions. What is not clear is how systemic or not the NHMRC review was of high quality RCTs. Importantly, for more than a decade high quality RCTs have been regarded as Level 1 evidence as per the Oxford CEBM website (http://www.cebm.net/?o=1025) (sadly our own NHMRC evidence guidelines have lagged for some time). The NHMRC decision not to adhere to a search of all Level 1 evidence, as per international standards, needs to be carefully explained and justified. It is important the Information Paper explicitly explains that the NHMRC review did not include a systematic assessment of RCT evidence. I do not agree that considering random evidence provided by interest groups offsets this deficiency (p18)."
- "High quality RCTs with narrow confidence intervals (Level 1 evidence) should have been searched for and included in this review.
Systematic reviews (SRs) have considerable weaknesses as reliable sources of evidence. Personally, I would prefer a much more reserved approach to their use as Level 1 evidence. For example, we know that SRs can come to quite contrasting conclusions pending the grading RCT scale they adopt. (See Juni et al, JAMA, 1999 http://rds.epi-ucsf.org/ticr/syllabus/courses/18/2009/04/16/Lecture/notes/Hazards_of_quality_scoring.pdf)."
- "High quality clinical trial designs in homeopathy should accommodate for tailoring of treatment but this is generally not weighted in the rating scales used by reviewers. If tailoring of treatment is critical in homeopathy then it may be that only low quality studies, as defined by the common grading metrics, will exhibit positive outcomes. [This is because tailoring of treatment may have resulted in un-blinding of the intervention group.] Some systematic reviews conclude homeopathy is more than placebo (Cucherat et al 2000; Linde & Melchart 1998; Linde et al 1997; Kleijnen 1991; and many of the reviews in the Swiss report found a trend in favour of homeopathy). It is probably unreasonable to discount this evidence on the basis that good quality trials did not show such strong evidence of efficacy, if the quality rating scale for trials is not well justified for use in homeopathy. Hence, the usual conclusion that high quality trials demonstrate less favourable outcomes than poor quality trials does not really hold. Quality rating scales have seldom been tested for inter-rater reliability, there is a lack of agreement between scales as to what is being measured, and scales differ in many respects including items for inclusion and number of items. The NHMRC Report should address this issue more explicitly."
- “The dismissal of positive systematic reviews compounded with the lack of an independent systematic review of high quality randomised controlled trials leaves me uncertain of the definitive nature of the Report’s conclusions.”
- "… if I were to dispassionately consider the evidence of efficacy, I am still left with niggling doubts that there are unanswered questions around the evidence. ..."
- "Quality rating scales have seldom been tested for inter-rater reliability, there is a lack of agreement between scales as to what is being measured, and scales differ in many respects including items for inclusion and number of items."
- "Finally, I am concerned that no homeopathic expert was appointed to the NHMRC Review Panel. I cannot imagine this being agreed in oncology, orthopaedics or other disciplines."
- "Curiously, I would also have been interested in the sub-analysis of studies where the active substance is present in considerable amounts."
[1] NHMRC Administrative Report, p.8
[2] NHMRC Administrative Report, p.9
[3] NHMRC Nomination of Independent Expert Clinical Reviewers for Externally Developed Clinical Practice Guidelines v2_140219
[4] Australasian Cochrane Centre Methodological Review feedback to NHMRC, 30 August 2013. NHMRC FOI 2015-16 007-05
[5] Expert reviewer feedback to NHMRC, May 2014. NHMRC FOI 2014/15 004-Section 62
[6] Australasian Cochrane Centre Methodological Review feedback to NHMRC, 9 July 2013. NHMRC FOI 2015-16 008-13