Attack the Stack: Finding the Gold in a Mountain of Research

Download PDF

Evidence from clinical trials is fundamental for informed decision making, but the sheer volume of the literature can be overwhelming. Where do you begin? Four epidemiologists provide expert guidance, with examples of key studies that changed ophthalmic practice as well as current strategies for gleaning the research findings that you can apply to your own patients.

In 1974, Alfred Sommer, MD, MHS, wrote an editorial in the American Journal of Ophthalmology decrying the lack of rigorous epidemiologic and statistical reasoning in ophthalmic clinical research. “At the time, the research amounted to ‘In my last 10 cases, I used three sutures instead of two, and my patients did much better.’” 

Clinical trials came to ophthalmology relatively late, compared with cardiology and other specialties, according to Dr. Sommer, who wrote the classic work Epidemiology and Statistics for the Ophthalmologist, published in 1980. He is now dean emeritus and professor of epidemiology and international health at the Johns Hopkins Bloomberg School of Public  Health and professor of ophthalmology at the Johns Hopkins University School of Medicine. 

He credits the advent of institutional review boards, which essentially serve as the first filter for studies, with transforming the quality of clinical research. “The ophthalmic literature has come a long way in my lifetime.” 

However, Kay Dickersin, PhD, MA, director of the U.S. Cochrane Center and an editor for the Cochrane Eyes and Vision Review Group, believes that some problems with quality persist today; she attributes these in part to the publish-or-perish demands placed on doctors. “The academic reward system and the proliferation of journals have held the quality of studies down. The average reader sees a peer-reviewed study from a reputable academic institution and should be able to assume that the quality is great. But that’s not the case,” said Dr. Dickersin, who is also professor and director for the Center for Clinical Trials at Johns Hopkins Bloomberg School of Public Health. 

“The challenge the journals have is that there aren’t enough epidemiologists and other methodologists to serve as peer reviewers,” said Dr. Dickersin. “Though peer reviewers can’t rescue a poorly conceived or conducted study, they can help to ensure that reports are clear and complete and to keep poor studies out of the journals,” she said. 

For the busy clinician tasked with filtering out the lower-quality research as well as keeping current with an overwhelming amount of literature, it’s simply not possible to read everything, said Anne L. Coleman, MD, PhD. “But it’s important to do our best to translate research insights into practice.” Dr. Coleman is secretary for Quality of Care at the Academy as well as professor of ophthalmology at the Jules Stein Eye Institute and professor of epidemiology at the University of California, Los Angeles, School of Public Health. 

Dr. Coleman’s imperative about taking research into the clinic has strong grounding both in common sense and in history. Since Dr. Sommer wrote his editorial in the 1970s, several key clinical trials have dramatically improved clinical care. Drs. Sommer, Dickersin, and Coleman, as well as Emily Y. Chew, MD, deputy director of the Division of Epidemiology and Clinical Applications and deputy clinical director at the National Eye Institute, discuss the lessons that can be learned from these landmark studies. This brief overview—followed by practical strategies for assessing the literature—may provide a renewed sense of purpose to ophthalmologists about to tackle the stack of journals on their desks.

Trials That Changed Ophthalmic Practice

A number of clinical trials have had a major impact on patient care. Some of these groundbreaking studies directly affected clinical practice, while others introduced methodological innovations that raised the standard of subsequent research, said Dr. Dickersin. This list, which is by no means exhaustive, provides exemplars to remind us of how valuable clinical trials have been—and will continue to be—in advancing clinical practice.

1. DRS: Diabetic Retinopathy Study

Arch Ophthalmol. 1973;90(5):347-348.
Arch Ophthalmol. 1985;103(12):1796-1806.

“Probably the most important study in ophthalmology is the DRS because it was one of the earliest randomized controlled clinical trials and the largest multicenter study in the history of ophthalmic clinical science at that time,” said Dr. Sommer.

The DRS showed that laser photocoagulation helps prevent severe vision loss from proliferative diabetic retinopathy. It was followed by the Early Treatment Diabetic Retinopathy Study (ETDRS), which demonstrated that photocoagulation earlier in the course of diabetic retinopathy is safe and effective.

“Because of these trials, laser photocoagulation became the standard of care for treating diabetic retinopathy,” said Dr. Chew. Prior to the DRS, the risk of going blind from proliferative diabetic retinopathy was 50 percent in five years; laser photocoagulation reduced the risk by as much as 95 percent, she said.

“The National Eye Institute organized and funded the DRS and ETDRS, so people really thought about every aspect of the trials,” said Dr. Sommer. The DRS, for example, was preceded by an NIH meeting that developed a standard set of definitions to describe the severity of diabetic retinopathy. This severity scale enabled ophthalmologists (and other medical specialists) to speak the same language. The definitions are still used today.

“The DRS had the right sample size; a strict protocol for monitoring participating centers, with people visiting the sites to make sure they were doing everything per the protocol; and strict standardization, such as one center that read all the photographs,” said Dr. Sommer. “The study had validity, precision, and applicability. It really served as the model of how to do clinical research right and was pivotal in changing the mindset in ophthalmology.”

2. OHTS: Ocular Hypertension Treatment Study

Arch Ophthalmol. 2002;120(6):714-720.

OHTS was the first large-scale study to demonstrate that topical ocular hypotensive medication was effective in delaying or preventing the onset of primary open-angle glaucoma (POAG).

The study also collected natural history data to help identify which patients are at higher risk for POAG and which patients are more likely to benefit from early treatment.

Based on the findings from OHTS, clinicians can separate patients with elevated intraocular pressure (IOP) into categories of high, medium, and low risk by a patient’s age, race, IOP, optic nerve anatomy, and central corneal thickness. The higher the risk, the greater the likelihood that early medical therapy is beneficial.

“OHTS helped our field shift from indecision regarding how to treat ocular hypertensives to insight about the risk factors for glaucoma in patients similar to those enrolled in the trial,” said Dr. Coleman, an investigator in the study.

“OHTS got ophthalmologists measuring central corneal thickness in their suspected glaucoma cases. Before the trial, we didn’t routinely order corneal pachymetry on ocular hypertensives,” said Dr. Coleman. “It also got people treating ocular hypertensives more appropriately, encouraging clinicians not to start medical treatment in patients at low risk of developing open-angle glaucoma.”

3. CRYO-ROP: Cryotherapy for Retinopathy of Prematurity

Arch Ophthalmol. 2001;119(8):1120-1125.

Most of our current understanding regarding the risk factors, natural history, and treatment of retinopathy of prematurity (ROP) is based on data from CRYO-ROP, a landmark NEI-supported, multicenter, prospective study.

CRYO-ROP was the first trial to evaluate a treatment for ROP. It led to the implementation of neonatal screening and peripheral retinal ablation for acute ROP. Prior to the trial, ablative therapy was controversial; after CRYO-ROP, it became the standard of care.

The trial was exceptionally well planned and designed, according to Dr. Chew. Study planners chose a well-considered threshold for treatment that permitted both statistically and clinically significant improvements in outcomes. They also chose pragmatic outcome measures that were efficient (fundus appearance) and relevant (visual function).

“CRYO-ROP established outcome measures as well as a classification system for the disease—both extremely important aspects of clinical trials that benefit subsequent research,” said Dr. Chew. It raised the bar on the quality of data collection.

“Since the trial, we have moved away from measures that are important to doctors and toward measures of importance to patients,” said Dr. Dickersin. “Visual function is just such an outcome—though it’s hard to measure, especially in babies.”

4. COMS: Collaborative Ocular Melanoma Study

Arch Ophthalmol. 1990;108(9):1268-1273.

The COMS is the largest study ever done in ocular oncology. Before the trial, most practitioners treated medium-sized tumors with enucleation. Interest in eye-sparing radiation therapy had increased, but the degree to which radiation could prolong survival was unknown. The COMS showed that brachytherapy was as successful as enucleation at controlling the tumor, with equivalent survival outcomes.

The COMS also showed no advantage in administering external radiation to patients with large choroidal melanomas before enucleation. Further, it established the importance of the natural history of choroidal melanoma, especially for small tumors in which treatment might not be needed, according to Dr. Chew.

This study has a long list of other achievements, as well. It established the accuracy of clinical diagnosis based on ultrasound, photography, fluorescein angiography, and clinical assessment. It also established the first classification system for choroidal melanoma size. Paul T. Finger, MD, FACS, chairman of the AJCC-UICC Ophthalmic Oncology Task Force, said that the COMS demonstrated for the first time that one staging system can be used at multiple institutions to allow treatments to be directly comparable and additive (“Updated Ocular Tumor Classification Will Improve Studies, Benefit Patients,” EyeNet, February 2012). This is particularly important in oncology because most eye cancers are rare and therefore difficult to study.

5. EMGT: Early Manifest Glaucoma Trial

Arch Ophthalmol. 2002;120(10):1268-1279.

The EMGT, cosponsored by NEI and the Swedish Research Council, was the first large randomized controlled clinical trial to compare the effects of treatment against no treatment in patients with newly diagnosed, previously untreated open-angle glaucoma. Before this study, the natural history of untreated glaucoma was little understood; thus, the true effect of IOP-lowering treatment could not be established.

The study proved that untreated patients were more likely than treated patients to show progression and that progression occurred earlier in the untreated group. The magnitude of the effect of IOP reduction was surprising even to the study investigators, said lead author Anders Heijl, MD, PhD, professor and chairman of ophthalmology at Malmö University Hospital in Sweden. “Every 1 mmHg of IOP reduction was associated with a risk reduction of 10 to 13 percent, depending on the analysis,” he said (“Landmark Glaucoma Studies,” EyeNet, March 2012).

Although the study clearly established the benefit of treatment, it also found that a substantial percentage of patients progressed despite treatment, while almost 4 out of 10 untreated patients did not progress. Further, the rate of progress was variable, even with the same group. This suggested that factors other than IOP played a role; for example, EMGT was the first study to demonstrate poorer outcomes in patients with pseudoexfoliative glaucoma.

Another important conclusion, in the words of the authors: “Because the EMGT visual field progression criterion is a sensitive indicator of deterioration, careful follow-up may allow treatment to be deferred in some patients.”

Taken as a whole, the key lessons for clinicians were 1) that even small reductions in IOP make a difference and 2) that because clinicians cannot readily predict which patients will progress, management must be tailored to the individual patient, based on close observation.

6. AREDS: Age-Related Eye Disease Study

Arch Ophthalmol. 2003;121(11):1621-1624.

AREDS was an NEI-sponsored, multicenter, prospective cohort study designed to assess the natural history, prognosis, and risk factors of age-related macular degeneration (AMD) and cataract. It also included a randomized, placebo-controlled clinical trial of the effects of high doses of antioxidants and zinc on the progression of AMD and the effects of antioxidants on the development and progression of cataracts.

The AREDS formula—500 mg of vitamin C, 400 IU of vitamin E, 15 mg of beta-carotene, 80 mg of zinc, and 2 mg of copper—delayed progression to end-stage AMD by 25 percent in those with either intermediate or advanced AMD. In the same high-risk group, the supplements reduced the risk of AMD-associated vision loss by about 19 percent. The supplements did not provide a benefit to those with early-stage AMD or with cataracts.

Micronutrient supplements represent the first effective treatment to slow the progression of AMD. “Before AREDS, nobody was prescribing vitamins,” said Dr. Coleman. “Since the study, everybody does.”

Another contribution of AREDS was the development of an AMD severity scale based on the trial’s natural history data. Scales are essential for assessing clinical trial results, and the trial leaders presented a detailed one for researchers and a simplified version that’s easy for physicians to use and for patients to understand. AREDS also taught us about the effect of cataract surgery on the retina.

7. CATT: Comparison of Age-Related Macular Degeneration Treatments Trials

N Engl J Med. 2011;364(20):1897-1908.

Sometimes off-label drug use provides such dramatic results that it becomes common practice before being tested in clinical trials. Bevacizumab (Avastin), an anti-VEGF biologic drug frequently used off label to treat wet AMD, is a good example of this.

The CATT study was designed to compare Avastin with ranibizumab (Lucentis)—an antibody fragment drug derived from the same parent molecule—that was approved for the treatment of wet AMD by the U.S. Food and Drug Administration in 2006. Prior to CATT, the two drugs had never been compared head to head.

The trial found that Avastin is equivalent to Lucentis in the treatment of AMD through two years when using similar dosing regimens. It also showed that monthly dosing produced slightly more visual gain than an as-needed regimen.

Of course, where the significant difference lies between Avastin and Lucentis is cost. One dose of Lucentis costs about $2,000; one dose of Avastin, about $50. Currently, the Centers for Medicare & Medicaid Services (CMS) does not have a specific code for reimbursement of off-label use of Avastin for AMD, and payment varies.

The Department of Health and Human Services Office of the Inspector General (OIG) has recommended that, in light of the high cost of Lucentis and the comparable efficacy of Avastin, CMS should establish a new payment code for treating wet AMD with Avastin. OIG has previously reported that if Medicare reimbursement for all beneficiaries treated for AMD had been paid at the Avastin rate, Medicare and its beneficiaries would have saved $1.4 billion.

8. IONDT: Ischemic Optic Neuropathy Decompression Trial

JAMA. 1995;273(8):625-632. 

“Because nonarteritic ischemic optic neuropathy (NAION) is a rare disease, you don’t hear about the IONDT very often,” said Dr. Dickersin, who was the principal investigator at the data center, “but it really had a tremendous impact.” 

The IONDT was designed to evaluate the safety and efficacy of optic nerve decompression surgery (ONDS) for NAION. It also aimed to elucidate the natural history of NAION, particularly second-eye involvement. 

At the time, the ophthalmic literature had reported mixed results regarding the efficacy of ONDS. The studies were plagued by ill-defined progressive disease, small sample sizes, and varying visual testing methods. No randomized controlled trial had been done. “It basically came down to 14 case studies that steered doctors toward decompression surgery,” said Dr. Dickersin. “That was a real problem!”

To its credit, the NEI had the foresight to run a trial right at the beginning of the ONDS trend, according to Dr. Dickersin. The IONDT found that the surgery not only was ineffective but also had the potential to be harmful. Forgoing treatment was safer than using the new surgical technique. Fortunately, doctors in the United States stopped performing ONDS almost immediately.

“It’s sad because there’s been no effective treatment for patients with NAION, but at least the IONDT stopped more harm from being done,” said Dr. Dickersin. The trial did report two positive findings—first, that many more patients than previously thought stabilize on their own, and, second, that the probability of second-eye involvement is lower than previously thought.

In terms of methodology, it’s always difficult to regulate surgical quality in randomized controlled trials, but the IONDT excelled in that area, which enhanced the credibility of its results. The methodology developed for the trial has served as a model for quality assurance for subsequent multicenter surgical trials.

Observational Studies Can Be Influential, Too

While not the gold standard of clinical research, observational studies are still valuable, often pointing investigators toward hypotheses to test in clinical trials. Despite their lower standard of evidence compared with randomized controlled studies, observational studies have influenced many current concepts and practices in ophthalmology.

BALTIMORE EYE STUDY (BES; Am J Epidemiol. 1991;134[10]:1102-1110). One example is the groundbreaking BES, launched by Dr. Sommer in 1985. It was the first community-based eye survey of adults and has served as the prototype for dozens that have followed.

The BES demonstrated for the first time that IOP alone is not a reliable definition or predictor of glaucoma. “We found that half of all patients with open-angle glaucoma will have a pressure of less than 21,” said Dr. Sommer. “Twenty-one is not the magic cutoff that many people believe it to be.” Pressures over 21 don’t indicate anything other than a higher risk; most people with glaucoma have an IOP of 21 or below. “This has led to the concept of target pressures. Before the BES, glaucoma treatment was considered successful if you got IOP below 21. Obviously that’s not a useful therapeutic endpoint—you’ve to get to a level that stops the progression of optic nerve damage.”

An objective of the BES was to look at racial differentials. “We suspected from prior chart review that blacks were more likely to develop open-angle glaucoma earlier and at higher rates than whites,” said Dr. Sommer. “The study supported these suspicions and led us in the Preferred Practice Patterns to suggest earlier and more frequent screening for glaucoma in blacks.”

WISCONSIN EPIDEMIOLOGICAL STUDY OF DIABETIC RETINOPATHY (WESDR; Arch Ophthalmol. 1989;107[2]:244-249). Before the WESDR, most information about prevalence, severity, incidence, and progression of diabetic retinopathy was based on specific groups of patients at specific clinics, where severe disease was likely overrepresented. The WESDR was a large cohort with a wide distribution of severity that was studied at baseline and followed up four, 10, 14, and 25 years later. The longitudinal data has been helpful in the design of clinical trials that assess interventions to prevent or slow progression of diabetic retinopathy, said Dr. Sommer.

The data from the WESDR suggested risk factors for diabetic retinopathy, helping to define screening guidelines for the disease. Just as important, the data also pointed clinicians toward several possible modifiable risk factors, such as the need for glycemic control throughout the course of the disease, improved blood pressure and cholesterol control, and smoking cessation.

Strategies for Reading Studies

Every clinician wants to base treatment decisions on strong evidence—like that found in the landmark studies above—but few have the time to keep up with the abundance of ophthalmic literature. Beyond coping with the sheer quantity, assessing the quality and applicability of research presents challenges. Drs. Chew and Coleman offer the following advice.

Focus on Potential Relevance

Effective use of current databases can help cut the mass down to size by helping you home in on relevant keywords and related research. PubMed is a good place to start—get to know the many search tools it offers. (See “What's New With PubMed: An Insider's Guide,” EyeNet, February 2013.)

“While there is no sure substitute for reading an article in its entirety, I learn a great deal from abstracts,” said Dr. Coleman, adding, “I find that even just scanning article titles can add perspective when time is short. Very often these efforts call attention to a few studies that have great relevance.

“There might be a couple of studies that are particularly applicable to your patients,” she said. “In these instances, it’s important to take time to appraise a study and understand its strengths and limitations.”

Critically Appraise Chosen Studies

Critical appraisal involves assessing the strengths and weaknesses of a study. The importance of this process cannot be overemphasized; it’s the only way to determine if a study is valid and relevant to your practice. Contrary to what many people think, it does not focus on statistics alone.

Ask the important questions. “You don’t have to be a statistician to appraise a study,” said Dr. Chew. “You just need to know what questions to ask yourself.

“The cornerstone of clinical trials is the randomization—ask yourself if it’s done without bias,” said Dr. Chew. Is the sample size large enough to test the study hypotheses? And is its statistical significance clinically applicable? “Say a study’s outcome measure is visual acuity, and it reports a two-letter difference,” said Dr. Chew. “The sample size might be so big that this small difference turns out to be statistically significant. Is it clinically meaningful, though? Not so much.”

Take it section by section. Dr. Coleman suggests the following steps in assessing each section of an article:

• Introduction. Read the introduction to figure out why the study was done and what the hypothesis was. Did the investigators ask a focused question?

• Methods. Then look at the methods to make sure you understand the implied logic. Consider the study population: What are the key clinical, demographic, and cultural characteristics of the patients? Are they similar enough to your own patients to allow you to apply the research insights? The methods section also contains other important keys to assessing the study: outcome measures, data collection procedures, and statistical tests used.

• Results. When it comes to the results, careful statistical analysis can add substantial value to an article. One indicator is whether an epidemiologist or statistician is involved as an author or mentioned in the acknowledgments. If so, that can give you more confidence in the analysis. It is also worthwhile to ask yourself whether the results make sense and are internally consistent.

• Discussion. In the discussion section, consider whether the authors’ conclusions are consistent with the results. Have they concluded anything that’s not supported by the data? Pay close attention to the discussion of the limitations of the study; they might change your mind about whether the results are relevant to your practice.

Read editorials for context. Sometimes one is fortunate enough to read an accompanying editorial from an experienced investigator in the field, Dr. Chew added. Such editorials typically explain why the study is important and help place the findings in context.

Consult Resources That Specialize in Vetting Research

So the bad news is that there is no quick and easy way to read a study. The good news is that there are trustworthy resources whose mission is to review all the existing literature on a topic and critically appraise the strength of the evidence. An additional benefit is that as new results emerge, they can be put into the context of preexisting results. All of these materials are available online; see “Further Resources” for their Internet addresses.

Preferred Practice Pattern Guidelines (PPPs). “The PPPs aim to consider all of the levels of scientific evidence available and are an excellent review of the relevant literature,” said Dr. Coleman. Just as important, the PPPs go beyond appraising the literature to offering practical recommendations. Dr. Coleman noted that most of ophthalmic practice is consensus medicine, not evidence-based medicine.

What if a question arises in your personal clinical practice that is not covered in a PPP? Take a look at the recently launched PPP Clinical Questions. “These are focused around a specific question for which one of the PPP committees, a subspecialty society, or the Cochrane Eyes and Vision Group finds all of the evidence that is available to answer the question. The PPP committee then provides consensus opinion where there is a lack of evidence to help the reader make an informed decision based on both existing evidence and opinions,” said Dr. Coleman.

Cochrane Reviews. Like the PPPs, the Cochrane Eyes and Vision Group offer reviews that synthesize all the scientific evidence available. Unlike the PPPs, they focus solely on clinical trials, and they do not issue any guidelines.

Systematic reviews have become popular in many journals, but the quality of those reviews is often poor, according to Dr. Dickersin. “People often think it’s something you can just write up in a weekend. But done properly, it’s an expensive and time-consuming process,” she said. Cochrane Reviews can each take two years or even longer to complete.

“Systematic reviews have to be well executed, trustworthy, and kept up to date,” Dr. Dickersin emphasized. “Bear in mind that even a systematic review needs to be carefully read and digested because it won’t address all the questions you have about a particular patient. You’ll need to translate it into a story that fits your patient.”

Resources for new developments. Because of their exacting methodologies, PPPs and Cochrane Reviews may lag a bit behind the latest news. However, according to Dr. Sommer, that may not be a problem: “It’s not often that really important papers are published.”

But when a study does appear that makes headlines and leads to patients coming in with questions, both the Academy and the NEI issue rapid-response statements written by expert epidemiologists and ophthalmologists who have vetted the research.

Use Common Sense

Be cautious about changing your practice based on one trial. “Every study that comes out positive has a statistically significant result, but that just means it’s unlikely that the result occurred by chance; it doesn’t prove that it didn’t occur by chance,” said Dr. Sommer. “So anything worth doing is worth doing more than once!”

This is particularly true with data from studies that were not randomized controlled trials. Other types of studies have lower standards of evidence and may not pan out when tested by clinical trials.

The new thing is not always the best thing. Intraocular lenses (IOLs) illustrate this point. “The Europeans were way ahead of us in trumpeting IOLs, but here in the United States, we waited through many iterations of the IOLs as they improved,” said Dr. Sommer. “That’s why our complication rates were much lower when we started using them, compared to Europe.”

There is no perfect study. The experts agreed that there is more than one way to collect and analyze data, and the design, conduct, and interpretation of results are all subject to bias and other limitations. Period.