Why are Cochrane hepato-biliary reviews undervalued by physicians as an aid for clinical decision-making?

1. Introduction 

According to its programmatic statement [1], “The Cochrane Database of Systematic Reviews (CDSR) aims to provide the type of information that is needed by physicians to make clinical decisions.” The availability of such a tool would be invaluable for practitioners in making clinical decisions, because there is consistent evidence that the Cochrane systematic reviews are of higher methodological quality than the systematic reviews published in scientific journals [2], [3], [4], [5].

However, there are several published reports suggesting that this aim has not been achieved. Of special interest is a study from McMaster University (the PLUS Project), in which researchers found that the physicians interviewed rated the Cochrane reviews as less relevant and less newsworthy, and accessed them less often than non-Cochrane reviews published in clinical journals [6]. Furthermore, they were much less likely to move from the abstract to the full text (7% for the Cochrane reviews versus 49% for the reviews published in the journals).

That study was preceded by other investigations and surveys consistently showing that, despite their methodological quality, the Cochrane reviews are not a preferred source of evidence for physicians, who prefer different sources for answering their clinical questions regarding patient care (Table 1). A further indication of scant clinical use of the Cochrane reviews comes from the frequent discrepancy between the treatments used by physicians for liver diseases and the evidence from the relevant Cochrane reviews [17]. In a small, unpublished, survey of hepato-biliary experts conducted by one of the authors (L.P.), all respondents claimed to continue to use lactulose for hepatic encephalopathy, even after the publication of the hepato-biliary review entitled Non-absorbable Disaccharides for Hepatic Encephalopathy, which discouraged the use of disaccharides in treating this ailment.

Table 1. Rating by physicians of use of the Cochrane library for answering clinical questions.

	Reference	Physicians’ specialty	Rating of Cochrane products	Rating of other electronic sources
	Ely JW, et al. [7], [8], [9]	Internists, pediatricians, family physicians	Cochrane was not listed among the ten most used electronic or paper-based resources.	UpToDate was the most commonly used resource
	Cullen RJ [10]	Family practitioners	Cochrane Library was used by 16.7%	Medline was used by 48.9%
	McKibbon KA, Fridsma DB [11]	Primary care physicians	CDSR was used by 9.5%.	UpToDate was used by 65.9%
	Alper BS, et al. [12]	Family physicians	Cochrane Library was excluded from the databases surveyed because “not covering broad enough scope”	The combination of STAT!Ref and MDConsult could answer 85% of questions
	Schilling LM, et al. [13]	Internal medicine residents	Cochrane Library was used by 3%.	Most frequently used resources: Medline (73%) and UpToDate (70%); most helpful, UpToDate and journal articles
	McCord G, et al. [14]	Family medicine residents and faculty	Cochrane Library was not listed among the six electronic resources rated as the best.	UpToDate rated as the best electronic information resource by 54% of faculty and 46% of residents
	Lai CJ, et al. [15]	Primary care internal medicine residents	Online sources were the most frequently used; Cochrane Library was not cited among them.	UpToDate was used by 98% of residents, literature search by 44%, Google.com by 35%
	Lucas BP, et al. [16]	Internal medicine hospitalists	Treatment change due to evidence search in 18% of patients. Cochrane Library was not listed among the 4 sources of evidence rated as “helpful or very helpful” for patient management.	Sources rated as helpful or very helpful for patient management: 71% UpToDate, 46% Medline, 18% Clinical Evidence, 3% CancerNet

The present analysis is aimed at evaluating the characteristics of the Cochrane reviews that seem to limit their use in clinical decision-making. It is based on an examination of the reviews of therapeutic interventions published in the hepato-biliary subset, assumed to be representative of the general CDSR, and chosen because one of the authors of this study (L.P.) has lifelong experience in the field of liver disease.

2. Materials and methods 

We included in this study the 87 hepato-biliary reviews of therapeutic topics published through December 2008. A further review (Ribavirin with or without Alpha Interferon for Chronic Hepatitis C) was withdrawn by the editorial group and split into two separate reviews, Ribavirin Monotherapy for Chronic Hepatitis C and Ribavirin Plus Interferon versus Interferon for Chronic Hepatitis C. The reviews were examined and classified according to the following characteristics:

3. Results 

A summary of the results is shown in Table 2.

Table 2. A summary of results.

	1. Empty reviews
	No RCT identified	6/87 (7%)
	Only one RCT identified	6/87 (7%)
	2. a. Lack of newsworthiness (a);	a. 13/87 (15%);
	b. obsolete reviews (b)	b. 8/87 (9.0%)
	3. Content of reviews	12–241 pages (median 40 pages). Implications for practice, median 6 pages.
	4. Implications for practice and need for further RCTs
	Efficacy	One treatment better than its alternative in 12/87 reviews (14%)
	Further RCTs needed	60/87 (69%)
	Further RCTs may be needed or are needed for specific aspects	20/87 (23%)

3.1. Empty reviews (Table 3) 

Six reviews (7%) could find no relevant randomised trial, and six (7%) found one RCT only. Among these 12 reviews, eleven concluded by stressing the need for further RCTs. One, Cholecystectomy versus No Cholecystectomy in Patients with Silent Gallstones, suggests the need for preliminary observational follow-up studies. Eleven reviews do not advance suggestions for practice; a review, including one trial, Early versus Delayed Laparoscopic Cholecystectomy for Biliary Colic, suggested that the intervention “seems to decrease the morbidity during the waiting period for elective laparoscopic cholecystectomy.”

Table 3. Reviews with no or only one RCT.

	Title of the review	Pages of full text
	No RCTs identified
	Antacids for preventing oesophago-gastric variceal bleeding and rebleeding in cirrhosis	14
	Antifibrinolytic amino acids for acquired coagulation disorders in patients with liver disease	12
	Cholecystectomy versus no cholecystectomy in patients with silent gallstones	12
	Elective surgery for benign liver tumours	16
	Probiotics for NAFLD &/or NASH	18
	Vitamin K for upper GI bleeding in patients with liver diseases	15
	Only one RCT identified
	Bicyclol for chronic hepatitis B	16
	Bicyclol for chronic hepatitis C	14
	d-Penicillamine for primary sclerosing cholangitis	17
	Early versus delayed cholecystectomy for biliary colic	27
	Human recombinant activated factor VII for upper GI bleeding in patients with liver disease	18
	Primary closure versus T-tube drainage after laparoscopic common bile duct stone exploration.	19

3.2. Non-newsworthy and obsolete reviews (Table 4a, Table 4b) 

There were 13 (15%) non-newsworthy reviews (shown in Table 4a).

Table 4a. Lack of newsworthiness: reviews posted in the Cochrane Database of Systemic Reviews Ten years or more after the publication of the most recent RCT included.

	Title (no. of pages)	Posted in	RCTs: n.; published, from to	Interval, years	Implications for practice
	Anabolic androgenic steroids for alcoholic liver disease (51)	2006, issue 4	n. 5; 1966–91	15	No significant benefit
	Antibiotics for leptospirosis (20)	2000, issue 2	n. 3; 1984–88 (poor quality of 2/3 RCTs)	12	Indication uncertain; penicillin or doxyciclin may cause more good than harm
	Azathioprine for primary biliary cirrhosis (19)	2007, issue 3	n. 2; 1976, 1984	23	Insufficient evidence; higher risk of adverse events
	Branched chain aminoacids for hepatic encephalopathy (58)	2003, issue 1	n. 11; 1984–93	10	No convincing evidence that BCAA had a significant beneficial effect
	Cyclosporine for primary biliary cirrhosis (24)	2007, issue 3	n. 3; 1988–93	14	No effect on progression to death, OLT or histology; more adverse events
	d-Penicillamine for primary biliary cirrhosis (41)	2004, issue 4	n. 7; 1981–85	19	No benefit, harmful effects
	d-Penicillamine for primary sclerosing cholangitis (17)	2006, issue 1	n. 1 RCT, 1988	18	Insufficient evidence, not recommended
	Dopaminergic agonists for hepatic encephalopathy (22)	2004, issue 4	n. 5; 1979–82	22	No evidence to support or refute an effect
	Endoscopic retrograde cholangiopancreatography with or without stenting in patients with pancreaticobiliary malignancy, prior to surgery (30)	2007, issue 3	n. 2, high bias risk; 1987, 1994	13	Treatment cannot be supported or refuted
	Hepatitis B vaccination for patients with chronic renal failure (28)	2004, issue 3	n. 7; 1981–94	10	Effect on antibody production, unknown on infection
	Portosystemic shunts versus endoscopic therapy for variceal rebleeding in patients with cirrhosis (53)a	2006, issue 4	n. 22: 8 trials of surgical shunts, 1987–93; 14 of TIPS	13	Endoscopic treatment first line (banding only in 4 trials); selected shunting procedures after rebleeding
	Propylthiouracil for alcoholic liver disease (39)	2005, issue 4	n. 6; 1979–93	12	No significant effect
	Routine abdominal drainage for uncomplicated open cholecystectomy (130)	2007, issue 2	n. 28; 1976–92	15	Increased harm, no benefit
	Surgical versus endoscopic treatment for bile duct stones (62).	2006, issue 2	n. 13; 8 of surgical clearance versus pre- or post-operative endoscopic retrograde cholangiopancreatography; 1983–92	14	Surgical clearance during open cholecystectomy, superior to clearance by endoscopic retrograde cholangiopancreatography

a This review included 8 trials of surgical shunts (4 of porto-caval shunts, 4 of spleno-renal shunts, and 14 of transjugular intrahepatic porto-caval shunts. Only the reviews of surgical shunts lack newsworthiness and are outdated; on the other hand, all the trials of surgical shunts and 10 of the trials of transjugular intrahepatic porto-caval shunts assessed as comparator the endoscopic variceal sclerotherapy, now almost completely replaced by endoscopic variceal banding.

Due to the rapid pace of progress in some areas (e.g.: viral hepatitis), several reviews not fulfilling the time-related definition of non-newsworthiness regarded treatments that are clearly obsolete with respect to current national guidelines. Since this definition implies a judgement that is, to some extent, subjective, in this report the eight reviews judged as obsolete (9%) and not fulfilling the definition of non-newsworthiness are listed in Table 4b, separately from the 13 non-newsworthy reviews. Overall, 21 (24%) reviews were considered outdated.

Table 4b. Reviews of therapies posted in the Cochrane Database of Systematic Reviews less than ten years after the publication of the most recent RCT included, but judged as obsolete according to current guidelines.

	Title of the review (no. of pages)	Posted in	RCTs: n.; published, from–to	Interval, years	Implications for practice of the review (full text)
	Glucocorticosteroids for primary biliary cirrhosis (36)	2005, issue 2	n. 2; 1989, 1999	6	Insufficient evidence; adverse effects may be frequent
	Glucocorticosteroids for viral hepatitis C (37)	2004, issue 2	n. 8; 1987–2001	3	Insufficient evidence
	Interferon for interferon-naïve patients with chronic hepatitis C (70)	2002, issue 2; unchanged, 2008, issue 4	n. 54; 1989–98	9	Interferon effective on virological, biochemical, and histological outcomes
	Interferon for interferon non-responding and relapsing patients (52)	2002, issue 4	n. 18; 1995–99	3	Retreatment with interferon leads to sustained virologic response in a minority of patients
	Ribavirin monotherapy for chronic hepatitis C	2005,issue 4	n. 13; 1993-2003	2	No significant effect. Not recommended
	Ribavirin plus interferon versus interferon for chronic hepatitis C (116)	2005,issue 2	n. 69 of standard interferon, 3 of PEG-interferon (2 as abstracts); 1995–2004	1	Adding ribavirin to any type of interferon should be considered the treatment of choice for patients with chronic hepatitis C
	S-Adenosyl-l-methionine for alcoholic liver diseases (42)	2006, issue 2	n. 9; 1988–99	7	No evidence to support or refute S-Adenosyl-l-methionine for alcoholic liver diseases
	Sequential combination of glucocorticosteroids and alpha interferon versus interferon alone for HBeAg-positive chronic hepatitis B (87)	2005, issue 3	n. 13; 1990–98	7	Sequential combination of glucocorticosteroids and alfa interferon may be more effective than interferon monotherapy on loss HBeAg and HBV DNA

3.4. Implications for practice, and need for further RCTs (Table 5) 

According to the Implications for Practice sections, 12 out of 87 reviews concluded that one of the treatments assessed was better than the active or non-active alternative (14%, listed in Table 6). Sixty reviews (69%) affirmed that “further trials are needed,” mostly in the group of reviews on treatments found not to be better than the alternative and/or with insufficient evidence. Twenty reviews (23%) (reported separately in Table 5), concluded that further trials “may be needed,” (e.g., in the review on penicillamine for primary sclerosing cholangitis, which states that “future randomised trials on d-penicillamine may be considered, preferably using lower dosages”) or that further trials are needed for specific aspects (e.g., in the review on cholecystectomy deferral in patients with endoscopic sphincterotomy, entitled Prophylactic Cholecystectomy Following Clearance of Choledocholithiasis Should Be Studied by Randomised Clinical Trials in Patients Considered High Risk for Surgery). Overall, 80 reviews (92% of the total number of 87) in the two subsets required further trials, a figure close to the 96% found by a study analysing 1016 reviews from the CDSR [18].

Table 5. Treatments reported as better than the alternative option, or partially better, or with uncertain evidence, and need of further RCTs for each group.

	(% of the total number of 87 reviews)	Further RCTs are needed: number of reviewsa	Further RCTs may be needed, or are needed for specific aspects: number of reviewsa	No further RCTs are needed: number of reviews
	One treatment better than the alternative: 12 (14%)	1	5	6 reviews
	Uncertain evidence, or one treatment partially better than the alternativeb: 26 (30%)	23	2	1
	Insufficient evidence and/or neither of the treatments assessed better than the alternative: 50 (57%)	36	13	1
	RCTs needed, or needed for specific aspects, number of reviews (% of 87)	60 (69%)	20 (23%)	8 (9%)

a Further RCTs needed for specific end points, or for subgroups, or for one of the treatments assessed by the RCTs. b Better for surrogate end points, or for subgroups, or for one of the treatments assessed by the RCTs.

Table 6. Implications for practice: list of reviews concluding that one of the interventions assessed is better than its alternative.

	Antibiotic prophylaxis for cirrhotic patients with gastrointestinal bleeding
	Cholecystectomy deferral in patients with endoscopic sphincterotomy (prophylactic cholecystectomy better)
	Cyclosporin versus tacrolimus for liver transplanted patients (tacrolimus better)
	Early versus delayed laparoscopic cholecystectomy for acute cholecystitis
	Interferon for acute hepatitis C
	Interferon for interferon-naïve patients with chronic hepatitis C
	Interventions for paracetamol (acetaminophen) overdose
	Laparoscopic versus open cholecystectomy for patients with symptomatic cholecystolithiasis
	Ribavirin plus interferon versus interferon for chronic hepatitis C
	Surgical versus endoscopic treatment of bile duct stones
	Terlipresssin for acute oesophageal variceal hemorrhage (with a comment expressing disagreement by P.C. Gotzsche)
	Vaccines for preventing hepatitis B in health care workers

4. Discussion 

There is a striking contrast between the stated aims of the Cochrane Database of Systematic Reviews, which is “to provide the type of information that is needed by physicians to make clinical decisions [1],” on the one hand, and their low rate of consultation and use by physicians, on the other. In our view, this discrepancy can be explained, at least to a large extent, by some characteristics of the reviews that are apparent in the hepato-biliary subset of the CDSR.

First, a non-negligible proportion of the reviews included in the subgroup were empty (no RCTs, or only 1 RCT), precluding a meta-analysis. Seven others included 2 RCTs only, a number unlikely to produce helpful information [19]. The empty reviews can be of help to physicians if the authors summarise and comment on the available non-randomised evidence from the excluded studies. Examples are the review on cholecystectomy in patients with silent gallstones, and the review on elective surgery for benign liver tumours. Empty reviews can also be of interest for researchers by pointing out grey zones that need further trials [20], [21]. However, some reviews remain empty perhaps because they address questions of interest for the authors only, and assess interventions with poor pre-clinical background, implausible efficacy, and very low priority. Examples are the reviews on bicyclol for chronic hepatitis B and C; on antacids for preventing oesophageal variceal bleeding in cirrhosis; and on vitamin K for upper gastrointestinal bleeding in patients with liver disease.

Second, almost one quarter of the reviews were outdated, either because the trials considered were too old (≥10 years when the review was posted in CDSR), or because recent advances in the field make the results of the review irrelevant for today's clinical decisions. In the real world of clinical practice, a physician is unlikely to search for information on the clinical value of azathioprine or d-penicillamine for primary biliary cirrhosis; the surgical porto-caval or spleno-renal shunt versus endoscopic variceal sclerotherapy in cirrhosis; glucocorticosteroids for viral hepatitis C; propylthiouracil for alcoholic liver disease; or glucocorticoids followed by interferon for chronic hepatitis B. These treatments and several others assessed in the Cochrane Hepato-Biliary Group of reviews were never put into practice, or have long since been dismissed. Reviews of outdated treatments are frustrating for physicians who are asking clinical questions. Furthermore, physicians may be confused by reviews that still assess treatments replaced by more recent and effective options. For instance, peginterferon α, the current mainstay of treatment for chronic hepatitis C [22], [23], [24], was not assessed in any of the 54 trials included in a review entitled Interferon for Interferon Naïve Patients with Chronic Hepatitis C, and in only 5 of 72 trials included in another review entitled Ribavirin Plus Interferon versus Interferon for Chronic Hepatitis C.

Third, consultation of Cochrane reviews is often time-consuming because of their length, distracting clinical readers from the goals of their research because the oversized content is focused mostly on the description of the methodology, with very loose links to clinical practice. This may account for the low rate of access from the abstracts to the full texts of the Cochrane reviews (7%), one of the most disturbing findings of the PLUS Project [25].

Fourth, only 12 reviews established the superiority of a treatment over its alternative, and two of them are outdated (Interferon for Interferon-naïve Patients with Chronic Hepatitis C, and Ribavirin Plus Interferon versus Interferon for Chronic Hepatitis C (Table 4b, Table 6)). To be helpful for physicians when evidence from randomised trials is lacking or insufficient, clinical information from non-randomised studies could perhaps be provided in the reviews. This policy is exemplified by the review Interventions for Paracetamol (Acetaminophen) Overdose, which included 10 randomised and 48 non-randomised studies, and reported evidence of a drop in the overall mortality rate of paracetamol overdose from 3–5% to 0.7% since acetylcysteine was recommended as the drug of choice.

Surprisingly, some of the 60 reviews requiring further trials are outdated, and regard treatments that have long since been dismissed (e.g., that on Sequential Combination of Glucocorticoids and Alpha Interferon versus Alpha Interferon Alone for HBeAg-Positive Chronic Hepatitis B). Overall, the high number of reviews recommending further research may frustrate the confidence of physicians in the usefulness of the CDSR to provide answers to clinical questions, and turn them toward other sources of information.

Finally, and most important, the CDSR is not structured to provide easy consultation and help for physicians asking questions relevant to their clinical practice. In the taxonomy of the clinical questions from physicians developed by Ely et al., the first most common question is: “What is the drug of choice for condition X?” and the fifth most common is: “How should I treat condition X (not limited to drug treatment)? [8]. Both questions ask for disease-centred information, taking into account the variety of treatments available. In contrast, the Cochrane reviews address narrow questions such as, “What's the evidence that intervention A is better than intervention B?” [26], ignoring further options of interest for the physician.

In conclusion, in its present form, the CDSR is an archive of good-to-high-quality systematic reviews that is, nonetheless, burdened by a large amount of irrelevant material. By avoiding protocols that deal with irrelevant, implausible and outdated topics, the core of good-to-high-quality reviews would become an important pre-clinical starting point in the development of evidence-based, pre-appraised, summaries and guidelines for physicians. On the other hand, the structure of the CDSR is not the most suitable for providing direct answers to clinical questions, for reasons at least in part clarified by the present analysis.

Our conclusion is based on the systematic examination of just one subset of 87 reviews, and could be less applicable to other subsets of the CDSR. However, the Cochrane reviews are used less than other sources of clinical information by general practitioners and resident non-specialists in hepato-biliary disease (Table 1), suggesting a general lack of confidence. Furthermore, an exploration of a random sample of oncological and cardiovascular Cochrane reviews suggests that our observations likely reflect the distinctive attributes of the Cochrane Collaboration in providing pre-clinical evidence, and focusing on methodology and rigor rather than on clinical relevance [25].

Conflict of interest 

No conflict of interest.