Implementation of a Monitoring System
An ideal monitoring system would capture data regarding every patient of interest for a given practitioner. Doing so would collect the maximum number of cases for statistical analyses and provide maximum statistical power for reliable estimates of uncommon or rare events. In addition, a 100% analysis would minimize bias due to missing patients. For example, electronic billing data could identify every patient who had intraocular surgery (identified by specific Current Procedural Terminology [CPT] codes) in a practice during a specified period and any subsequent surgeries (again identified by CPT codes) within the next 30 or 90 days to determine a specific complication (identified by its International Classification of Diseases code), such as retinal detachment or endophthalmitis. This would reflect outcome quality.
In contrast, questions about process quality—such as whether a target pressure range was set for every patient with glaucoma—are not amenable to a 100% analysis because that type of data may not be entered in current administrative databases; instead, it may need to be extracted from medical records by a trained reviewer. Information obtained from billing databases may allow assessment of other process quality measures, such as gonioscopy.
An organization’s next step is to review its records. What standards should be used for such a review? What criteria should be used? Explicit criteria, which have a yes/no outcome or limited categories (eg, optic nerve documentation could include a statement regarding the nerve’s condition, the vertical cup–disc ratio, or a drawing or photograph), have higher reliability than implicit criteria (the reviewer’s judgment that overall quality was good or not good), particularly for interrater reliability. For ophthalmology practices, the American Academy of Ophthalmology (AAO) provides Preferred Practice Pattern guidelines and a summary benchmarks series, both of which can be used to obtain explicit criteria. Similarly, the American Board of Ophthalmology includes explicit criteria in its Practice Improvement Modules for maintenance of board certification. These are available at www.aao.org/ppp and at https://abop.org, respectively.
Record reviews may reveal that some medical records are unavailable or are missing data or information on visits. Every effort should be made to obtain unavailable records. If these records remain unavailable, the number of unavailable records should be recorded, and replacement records from the randomization should be reviewed. A high proportion of missing medical records may suggest bias. For records with missing visits, it may be possible to capture important data from other available visits. If the review criteria require that every visit is checked, the options are to (1) exclude that patient, (2) exclude that patient only for analyses needing that missing-visit data, (3) impute the missing values through statistical modeling of available data, or (4) treat the missing visit as either meeting or not meeting the criteria (generally the latter). The key steps are to decide what to do and then apply that decision consistently over time (and report the decision with the data and results).
An important element of establishing a system for monitoring quality of care is performing power calculations to determine sample sizes, because these calculations provide confidence that a nonsignificant difference is truly nonsignificant (and is not due to having an insufficient sample size). See the section “Was the sample large enough to detect a difference?” earlier in this chapter.
One final consideration is the external validity of the method used to determine whether a quality measure was met. Using chart review, McGlynn and colleagues determined that the rate of annual dilated eye examination among patients with diabetes mellitus was only 19%. But when they used billing codes, they found that the rate was 50%. Was the discrepancy due to poor documentation of the procedure or inaccurate billing practices? The data may have been recorded incorrectly, by either the observer or the person abstracting the data from the data source. Other errors could be related to coding issues, data entry problems, and incorrect diagnoses. In summary, each discrepancy needs to be examined further, and the whole monitoring system may need to be redesigned. As stated previously, conducting a pilot study with a handful of participants can help with study design, increase validity, and save time when developing a monitoring system.
In 2013, AAO introduced its IRIS (Intelligent Research in Sight) Registry, a centralized system for collecting electronic eye care data from ophthalmology practices. This registry automatically abstracts data from modern electronic medical record systems. In comparison to large claims-based registries, which collect data about electronic billing, IRIS contains specific information related to the medical treatment of eye disease (eg, visual acuity, IOP) and is agnostic to insurance status. This allows ophthalmologists to examine results of medical treatment of eye disease and develop improvement activities. AAO’s other goals for the registry include the provision of benchmark reports for quality of care and identify opportunities for improvement. The registry can also create large data sets for diseases such as AMD, cataract, and glaucoma, facilitating exploration of trends in treatment, costs, and other factors. It also allows collection of data regarding rare diseases, such as retinitis pigmentosa. For example, Rao and colleagues compared the visual acuity after 1 year in patients undergoing treatment for neovascular age-related macular degeneration (nAMD). The researchers found no difference in visual acuity results when comparing 3 common medications for nAMD (bevacizumab, ranibizumab, and aflibercept). This result suggested that all tested medications could be used, and providers may consider other distinguishing characteristics of the treatments such as cost when deciding which medication to recommend. This may be an important tool to help ophthalmologists continually monitor and improve their performance quality.
McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635–2645.
Rao P, Lum F, Wood K, et al. Real-world vision in age-related macular degeneration patients treated with single anti-VEGF drug type for 1 year in the IRIS Registry. Ophthalmology. 2018;125(4):522–528.
Excerpted from BCSC 2020-2021 series: Section 1 - Update on General Medicine. For more information and to purchase the entire series, please visit https://www.aao.org/bcsc.