Deluged by Data: The Struggle to Close the Evidence-to-Practice Gap

Download PDF

At the turn of the century, the Institute of Medicine published To Err Is Human¹ and Crossing the Quality Chasm.² “Both caught the attention of the public and policy mak­ers,” said Richard L. Abbott, MD, professor emer­itus at the University of California, San Francisco. Although the first report focused on medical er­rors and the second on health care delivery issues, they also dovetailed with an already established and growing interest in evidence-based medicine.

For instance, in the late 1980s, the Academy introduced its clinical practice guidelines. Two issues that concerned physicians at the time were significant gaps in evidence and variations in practice among specialists including cardiologists, ophthalmologists, and orthopedists. “People became really interested in focusing on evidence, rather than just on professionals’ prior experience or on expert opinion, often referred to as emi­nence-based medicine,” Dr. Abbott said.

Since then, evidence-based medicine (EBM) has become an integral part of patient care. But keeping up with and implementing the latest evidence poses myriad challenges for physicians (see “Barriers to Quick Adoption,” below). In fact, Crossing the Quality Chasm reported that the time between significant discovery and adoption into routine patient care averaged 17 years.² Although a lag between discovery and implementation is inevitable, it’s important to look at EBM and how physicians, trial investigators, and organizations can work together to speed EBM to the clinic to ensure that patients receive the best current care.

The Evolving Role of EBM

Keys to effective practice include understanding EBM and knowing how to apply it to patients.

Evolution of EBM. Initially, EBM was defined as an approach to patient care in which decisions were based on clinical studies, ideally unbiased and well-designed randomized controlled trials (RCTs), said Dr. Abbott. The goals of EBM were to improve the patient’s quality of care and outcomes and to minimize mistakes—and they still are, he said.

Today, however, a working definition of EBM has been expanded to include integration of the best available evidence with clinical expertise as well as the patient’s preferences, values, and unique set of circumstances.^3,4

EBM and your patients. This redefinition is much more than an academic exercise. “Clini­cal evidence is based largely on populations of patients, and those patients may differ a little or a lot from the patient sitting in front of you,” said Jennifer K. Sun, MD, MPH. She is chair of the DRCR Retina Network (DRCR.net), a consortium of sites funded by the NIH to research diabetic eye disease and other retinal diseases. Patients in clinical practice, for example, may be much less compliant with follow-up visits or treatment than those intentionally recruited for and guided through clinical studies.⁵

“In addition, study outcomes often refer to average results across the entire group, which may not represent what will happen in an individual patient,” said Dr. Sun. She gave the example of anti-VEGF studies conducted by the DRCR.net. “In many of our studies for diabetic macular edema [DME], patients on average did extraordi­narily well, gaining 2 to 4 lines of vision, depend­ing upon the type of agent used.” However, some patients gained even more than the average, while others gained less or even, rarely, experienced vision worsening, she said.

This may be why some EBM experts consider an N-of-1 trial—how a particular patient responds—to be the highest level of evidence, even higher than that of an RCT, said Paul P. Lee, MD, JD, at the W.K. Kellogg Eye Center in Ann Arbor, Mich­igan. “Our vocation is to know about the findings and limitations of the RCTs so we have informed judgment about how we care for each patient.”

Barriers to Quick Adoption

Although EBM has been part of the medical landscape for at least two decades, many barriers to quick adoption remain.

Physician perspective. “The biggest barrier is that physicians are extremely busy,” said Dr. Abbott.

Time crunches. The stresses on physicians include not just patient care but also third-party demands such as documentation, negotiations with insurance companies, and other regulatory burdens, added Rahul N. Khurana, MD, at Northern California Retina Vitreous Associates in Mountain View, California.

“To adopt a new practice,” said Dr. Abbott, “physicians may have to take courses and learn something new, whether gaining experience with a new drug or device.” Adding to these time pres­sures is the huge volume of information available through the internet, peer-reviewed journals, meetings, and more, to which physicians may devote significant time in order to keep current. In fact, the amount of medical knowledge is estimat­ed to double every two to three months.⁶

Access to information. In addition to volume, information’s rapid dissemination through the internet and social media can either facilitate or inhibit the adoption of EBM in medicine, said Dr. Lee. Obviously, the sooner physicians get infor­mation the sooner they can integrate it. However, it’s not uncommon for one study to contradict the next or for related findings to raise questions or create ambiguity. In the confusion, busy physi­cians may have trouble determining which new procedures are worth adopting.

Similarly, he said, “patients’ access to infor­mation may push us to do better faster, or it can create problems, as we’ve seen with the antivaccine movement.”

Practicalities of practice. Cost and medicolegal risk can also affect EBM adoption rates.

Cost. There’s no question that real-world practice in ophthalmology is influenced to some degree by billing concerns, said Dr. Sun. “Many of our eye treatments are expensive and rely on specialized care. If the best medicine available requires a $1,000 out-of-pocket expense, what is the best course of action for the patient? The only answer is true informed consent.”

Of course, costs can be an issue for physicians and their practices, as well. “Does the new evidence ask physicians to do something different that costs more?” asked Dr. Abbott. “And how will it affect their office and workflow?” For instance, take a practice that does not own a selective laser trabec­uloplasty (SLT) machine but wants to offer SLT as primary therapy for primary open-angle glaucoma patients. It could use a machine at an off-site cen­ter, but this would significantly alter workflow.

In addition, most payment systems reward volume of procedures rather than outcomes. As a result, adherence to EBM and clinical guidelines may be hindered if payment is not sufficient, Dr. Abbott said.

Medicolegal issues. Exposure to medicolegal risk may factor into adoption of new technology or of drugs that you don’t have a lot of experience with, said Dr. Abbott, “If you have a bad or less-than-desirable outcome and you end up in a lawsuit, the prosecuting attorney could argue that you did not take proper steps when adopting the new practice.”

Real-world implementation. It can be more difficult to incorporate new protocols into the clinic than in a clinical trial setting, said Dr. Sun. “In planning our clinical trials, we have the luxury of doing what clinicians prob­ably don’t, which is to sit down quietly and take the time to look at all the data together.”

Additionally, clinical trials may produce im­pressive results, said Dr. Khurana, but the investi­gators follow strict protocols and regimens, which don’t always translate well into daily practice. “For example, the patient may not want to follow up or be treated as often as the protocol recommends,” he said. And other issues may come into play: Per­haps the patient can’t afford multiple copays, isn’t able to get transportation, or may have a family illness.⁵

In fact, when Dr. Sun sees a referral patient who hasn’t responded well to a medication, she first checks whether the dosing regimen has been followed. She has found that inadequate dosing can be a common problem in real-world practice.

VISUAL ABSTRACTS. To help quickly relay important study findings, visual abstracts, such as this one, can be useful to physicians and patients.

Behavioral Blind Spots

When it comes to adoption of EBM, some barriers may have to do with the behavioral blind spots of both physicians and patients.

Skepticism. “Much Level 1 evidence has been driven by the pharmaceutical or medical device industries attempting to get a new medicine or device approved,” said Dr. Khurana. “This can lead to questions about the legitimacy of the data.”

Moreover, when evidence goes against our in­tuition, it’s hard to accept and people become very skeptical about it, said Laura Scherer, PhD, an ex­pert in medical decision-making at the University of Colorado, Denver. “Physicians either don’t want to incorporate it into practice, or they require a lot more evidence to change what they are doing. They just don’t believe that their intuition could be so wrong.”

This happens often in medicine, she said, es­pecially when new evidence contradicts practices we hadn’t previously tested because we just knew they worked. “Once a clinical trial strongly defies people’s expectations, they can have a hard time putting it into practice.”

Resistance to change. Some physicians see new evidence and say, “Great, when can I sign up for the course?” said Dr. Abbott. “Others tend to be more conservative.”

Sometimes it does make sense to wait, said Dr. Lee. “We have to be careful not to jump on a bandwagon too fast. The key is to ask, ‘Does this make sense? Are these results truly plausible? Is there a scientific basis for this to work?’”

Dr. Abbott noted that change is more likely if widely publicized, strong evidence confirms that a new approach will result in a significantly im­proved outcome. Dr. Sun gave an example: DRCR.net Protocol I—a phase 3 study of anti-VEGF medications for DME—showed a dramatic im­provement in visual acuity outcomes in the first year with ranibizumab compared to laser alone. This prompted a relatively quick change in clinical practice, she said.

Just do something? Physicians and patients are used to thinking that if something doesn’t work, they should just do more of it—or if one medicine is good, two are better, said Eve A. Kerr, MD, MPH, at the Ann Arbor Michigan VA Center for Clinical Management Research, and an expert on implementation science. “Sometimes that is not the case,” she said. “And once a treatment or practice has been established in medical care, it is much harder to think about how to stop or scale back, particularly because clinical guidelines often don’t explicitly address how to do so.”

When it’s found that a standard practice doesn’t really have the desired benefit and instead causes harm, there is still a very strong bias toward continuing with the practice, said Dr. Scherer, who tested a hypothetical scenario posed to both men and women.⁷ In her study, participants were told, “Suppose that 30 years of research showed that a breast or prostate test had absolutely no effect on reducing deaths from this particular cancer but did produce harms. Would you want the test?” About half the study participants did, largely because they simply couldn’t believe a cancer-screening test wouldn’t save lives. Even 43% of those who believed the test wouldn’t save their life still wanted it. They were convinced the informa­tion would benefit them in some way, even though they weren’t sure how, she said.

Racial and gender bias. Although physicians may not admit it, biases may affect consistency of their care. A classic cardiology study conducted at Duke University showed doctors videotapes of pa­tients with identical presenting complaints. “One set of patients were white actors and the other set were black actors,” said Dr. Lee. The doctors were asked to assess the patients and make recommen­dations. The authors wrote, “We found that the race and sex of the patient affected the physicians’ decisions about whether to refer patients with chest pain for cardiac catheterization. [This was] even after we adjusted for symptoms, the phy­sicians’ estimates of the probability of coronary disease, and clinical characteristics.”⁸

Confirmation bias. The process by which people look for information that confirms their view—known as confirmation bias—also may factor into how people interpret EBM.⁹ Although there is of­ten good reason to debate about evidence, said Dr. Scherer, we are more likely to question evidence that strongly disconfirms something we want to believe or intuitively believe. “Then it takes much longer and requires much more data to change practice.”

JOURNALS. JAMA Ophthalmology uses Key Points (top), and Ophthalmology uses a precis (bottom) to quickly communicate important information to readers.

Trial Design and Reporting

Certain steps can be taken to enhance trials and speed their translation into clinical practice, said Dr. Kerr.

Encourage community involvement. A major priority of the DRCR.net has been to encourage involvement of both academic centers and community practice physicians, leveraging the strengths of both, said Dr. Sun. “Today, retina specialists in community practice are some of the leading clinical trial investigators throughout the United States, and private practitioners are often more successful at quickly enrolling patients than physicians at academic centers.” When commu­nity centers are involved from the outset and the realities of practice are taken into account, the chances that the study results can be successfully implemented in practice increase.¹⁰

Rethink trial design. To make trials more relevant to the challenges of clinical practice, said Dr. Sun, it may help to ask this question: How do we design a trial that will be applicable not only to relatively healthy patients who are highly motivated but also to patients who may not be as motivated or may be sicker, may lack access to medical care, or may not have insurance that is open to specialized care?

It’s also important to remember that no trial can answer every relevant clinical question, said Dr. Sun. “That’s why we need to ask, ‘What are the questions this study an­swers well? Which ques­tions are still outstand­ing? And is it necessary to do another study?’”

Improve outcome reporting. To facilitate translation of research into practice, Dr. Sun noted that it may help to synthesize outcome reporting with tools such as visual abstracts (Fig. 1) and focused tables of contents, which are groupings of, say, five or six of the most important recent studies on specific topic. Over the last de­cade, the DRCR.net has tried to report outcomes in a way that helps physicians explain results to individual patients. “This means not just provid­ing the average amount of visual acuity change over the course of a trial,” said Dr. Sun, “but also trying to report outcomes such as the percentage of patients that gained 2 or 3 lines of vision or who lost vision.”

Standardize algorithms. One barrier to EBM adoption that came to light after the initial anti-VEGF studies, said Dr. Sun, was the complexity of the treatment algorithm. “We tried to find ways to articulate the broad philosophy of the algorithm to make it easier for physicians to implement,” she said. “For example: Inject into eyes that are improving or worsening but hold the injections once there is sustained stability.” Using the same algorithms across multiple studies also helps familiarize physicians with this process.

Obtain more data. Finding more ways to gain data about larger numbers of patients may also be a boon to clinical translation. For example:

Large simple trials. This form of RCT mini­mizes inclusion and exclusion criteria, making it possible to enroll thousands of people in each arm of the study, said Dr. Lee. “The large sample size allows the investigators to control for confounding variables. When results are unambiguous, uptake may happen more quickly.”

IRIS Registry. Comprehensive, large-scale reg­istries provide big pools of data, and, by tracking outcomes, they help confirm uptake of EBM, said Dr. Abbott. Dr. Lee added that registries—such as the Academy’s IRIS Registry, which leverages the collective experience of millions of patients around the United States—allow users to see which patterns of care are associated with dif­ferent outcomes. (See “New Application for IRIS Registry Users.”)

New Application for IRIS Registry Users Would you like to know how your clinical care compares to that of your peers? This is possi­ble—for cataract surgery, for now—with a new cloud-based app that became available to IRIS Registry users in October. In its initial iteration, the application (called Verana Practice Insights) allows users to: examine their own data and trends in patient outcomes and care; benchmark individual clinical care patterns against those of other ophthalmologists; and visualize deidentified aggregate data of phy­sician practice trends across the United States. With this information, physicians have a data-based foundation for determining and adopting best practices, improving outcomes, and providing better patient care. Who will benefit. Verana Practice Insights will initially provide information on practice trends related to cataract surgery and will ex­pand to other indications in early 2020. Those who do not perform cataract procedures but are interested in participating in the future should preregister. This will help determine which areas will be developed next. Who is eligible. Verana Health (www.veranahealth.com) developed Practice Insights for U.S.-based Academy members who participate in the IRIS Registry via an integrated electronic health record (EHR) system. There is no charge. How to get started. Complete the form at www.veranahealth.com/verana-practice-insights-signup. CHECK YOUR STATS. The Verana Practice Insights tool allows cataract surgeons to benchmark their outcomes against those of their peers.

CHOOSING WISELY. This campaign outlines five key areas in which physicians can help patients choose care that is supported by evidence, free from harm, and truly necessary (choosing wisely.org).

EVIDENCE RATINGS. The Summary Benchmarks, like the Preferred Practice Patterns (aao.org/ppp), which they summarize, are accompanied by evidence ratings (yellow).

Societies and Other Institutions

Education, accessible results, and open dialogue are all key in aiding in adoption of the current evidence, said Dr. Sun. “This is where professional societies can and do play a really strong role, such as with special educational initiatives.”

Highlight need-to-know information. For instance, to help JAMA Ophthalmology readers quickly understand the importance of its studies, the journal began in late 2016 to add “Key Points” (Fig. 2) of 100 words or fewer to its original investigations. Each study article includes one- or two-sentence statements encapsulating these three items: the question under investigation, the finding of the study, and the meaning that read­ers can take away from the study. Similarly, the Ophthalmology journal table of contents includes an approximately 35-word precis summarizing the main findings of each original article, without duplicating the abstract’s conclusion.

Target key initiatives. Societies can also help by popularizing key initiatives, said Dr. Khurana, adding that, in fact, this did happen a few years ago. Every specialty society picked a handful of practices that everyone should be doing—they didn’t have to be revolutionary. For example, ophthalmologists had typically given patients antibiotic drops after intravitreal injections, but the data showed it didn’t demonstrably decrease the risk of infection, he said. This became one of several important deimplementation initiatives of the Academy. Helping to promote this change was Choosing Wisely (Fig. 3), a multispecialty partner­ship that seeks to prevent the use of unnecessary medical tests, treatments, and procedures.¹¹

Simplify guidelines. Given the information overload that clinicians face, synthesis and sim­plification can make a difference. As secretary for Quality of Care and Knowledge Base Develop­ment (2002-2008), Dr. Abbott helped summarize the Academy’s approximately 25-page Preferred Practice Patterns¹² clinical guidelines for man­agement of various conditions into two-page Summary Benchmarks (Fig. 4). The guidelines include an evidence rating. For example, the qual­ity of individual studies is rated on a numerical scale, starting with I++ indicating “High-quality meta-analyses, systematic reviews of randomized controlled trials (RCTs), or RCTs with a very low risk of bias.” Recommendations for care are based on the body of evidence and are rated starting at “Good quality (GQ). Further research is very unlikely to change our confidence in the estimate of effect.” And key recommendations are graded, starting with “Strong recommendation. When the desirable effects of an intervention clearly out­weigh the undesirable effects or clearly do not.”

For a project in China, Dr. Abbott further sum­marized the benchmarks to create pocket cards, allowing key evidence to “travel” with doctors.

Learn from the VA. The VA is another institu­tion that is helping physicians. Approximately 20 years old, the VA’s Quality Enhancement Research Initiative focuses on implementation research and quality improvement. It achieves this by integrat­ing researchers into its system, and they examine all the factors that contribute to adoption of a new technique or, conversely, deimplementation of an obsolete treatment, said Dr. Kerr.

Find implementation funding. “Whether fund­ing comes from internal health systems, insurers, or the NIH, we need more resources to figure out what works and what doesn’t and to do larger-scale implementation research,” said Dr. Kerr. “We put a lot of funding into new discoveries, but very little into making sure they get to the right people at the right time.”

___________________________

1 Kohn LT et al., eds. To Err Is Human. National Academies Press; 2000.

2 Richardson WC et al. Crossing the Quality Chasm: A New Health System for the 21st Century. National Academies Press; 2001. www.nationalacademies.org/hmd/Reports/2001/Crossing-the-Quality-Chasm-A-New-Health-System-for-the-21st-Century.aspx.

3 Straus S et al. ACP J Club. 2007;146(1):A8-A9.

4 Sackett DL et al. Br Med J. 1996;312(7032):71-72.

5 Ciulla TA et al. Ophthalmol Retina. 2018;2(12):1179-1187.

6 Densen P et al. Trans Am Clin Climatol Assoc. 2011;122:48-58.

7 Scherer LD et al. J Exp Psychol Appl. 2018;25(2):149-161.

8 Schulman MD et al. NEJM. 1999;340(8):618-626.

9 Tilburt JC et al. Med Care. 2010;48(4):341-348.

10 Fleisher P, ed. Community-Engaged Research with Com­munity-Based Clinicians: A Resource Manual for Researchers. University of California, San Francisco; 2010. https://accelerate.ucsf.edu/files/CE/manual_for_researchers_clinicians.pdf.

11 Choosing Wisely: “Don’t routinely provide antibiotics before or after intravitreal injections.”

12 aao.org/preferred-practice-patterns.

Meet the Experts

Richard L. Abbott, MD Pro­fessor emeritus at University of California, San Francisco, and research associate at the Francis I. Proctor Foun­dation in San Francisco. Rel­evant financial disclosures: None.

Eve A. Kerr, MD, MPH Pro­fessor of internal medicine at the University of Michigan Medical School in Ann Arbor, Mich., and senior investigator at the Ann Arbor Veterans Affairs Center for Clinical Management Research. Relevant financial disclo­sures: Department of Veter­ans Affairs: E.

Rahul N. Khurana, MD Ret­ina specialist at Northern California Retina Vitreous Associates in Mountain View, Calif. Associate clinical pro­fessor in ophthalmology at University of California, San Francisco. Relevant financial disclosures: None

Paul P. Lee, MD, JD Professor and chair of oph­thalmology and visual sciences at the Universi­ty of Michigan Medical School and director of the W.K. Kellogg Eye Center in Ann Arbor, Mich. Relevant financial disclosures: None.

Mitesh Patel, MD, MBA, MS Director of the Penn Medi­cine Nudge Unit and assis­tant professor of medicine and health care manage­ment at the Perelman School of Medicine and The Whar­ton School at the University of Pennsylvania in Phila­delphia. He also is a senior fellow at the Leonard Davis Institute of Health Econom­ics, on faculty at the Penn Medicine Center for Health Care Innovation and the Center for Health Incentives and Behavioral Economics, and a staff physician at the Crescenz VA Medical Center in Philadelphia. Relevant financial disclosures: None.

Laura D. Scherer, PhD Assistant professor of research at the University of Colorado in Den­ver. Relevant financial disclosures: None.

Jennifer K. Sun, MD, MPH Associate professor of ophthalmology at Harvard in Boston. She is also chief of the Center for Clinical Eye Re­search and Trials of the Beetham Eye Institute, Joslin Diabetes Center, and chair of the DRCR Retina Network (DRCR.net). Relevant financial disclosures: None.