Reprinted, with permission, from Archives of Ophthalmology, Multicenter trial of cryotherapy for retinopathy of prematurity: preliminary results, Volume 106, No. 4. Copyright @ 1988, American Medical Association. All rights reserved.
Accurate assessment of plus disease is critical during clinical ROP management. This is particularly relevant because the Early Treatment for ROP (ETROP) study recently demonstrated that presence of plus disease is sufficient for meeting the definition of type-1 ROP, which benefits from early treatment with laser photocoagulation or cryotherapy.3
Several studies have raised concerns about the reliability and accuracy of plus disease diagnosis among clinical experts. For example, the multicenter Cryotherapy for ROP (Cryo-ROP) trial involved experienced pediatric and retinal ophthalmologists who were trained and certified in ROP diagnosis and calibrated during regular training sessions. However, 12% of infants in this study diagnosed with threshold disease after ophthalmoscopy by one certified investigator were diagnosed as less-than-threshold during confirmatory examination by a second unmasked certified investigator.4
More recently, a study was performed in which 22 national and international experts independently reviewed 34 wide-angle retinal photographs from infants with ROP on a secure website. Using a 3-level categorization (plus, preplus, neither), all 22 experts agreed on the same diagnosis in only 4 of the 34 images (12%). Using a 2-level categorization (plus, not plus), all experts agreed on the same diagnosis in 7 of the 34 images (21%) (Figure 2).5
Reprinted, with permission, from Archives of Ophthalmology, Interexpert agreement of plus disease diagnosis in retinopathy of prematurity, Volume 125, No. 7. Copyright @ 2007, American Medical Association. All rights reserved.
Representative images shown to 22 expert participants in published study of plus disease diagnostic agreement.5
(A) was classified as plus by 2 (9.5%), preplus by 9 (42.9%), neither plus nor preplus by 10 (47.6%), and "cannot determine" by 1 expert. (B) was classified as plus by 3 (14.3%), preplus by 9 (42.9%), neither by 9 (42.9%), and "cannot determine" by 1 expert. (C) was classified as plus by 11 (52.4%), preplus by 10 (47.6%), and "cannot determine" by 1 expert. (D) was classified as plus by all 22 (100%) experts.
Potential Subjectivity in Diagnosis
Inconsistent identification of plus disease can lead to errors in overtreatment or undertreatment of severe ROP. For that reason, it is essential that plus disease be detected correctly. The above studies suggest that plus disease diagnosis may be subjective and variable even by recognized authorities. A likely explanation is that plus disease is defined by a photographic standard with descriptive qualifiers rather than by quantitative measurable parameters.
Because the standard photographic definition of plus disease is a single narrow-angle image (Figure 1), different physicians may reach different conclusions when faced with numerous real-world clinical scenarios. For example:
- Which particular vessels in the standard photograph represent the requisite amount of dilation and tortuosity?
- Do dilated and tortuous vessels need to be present in the same two retinal quadrants?
- Is severe tortuosity in the absence of significant dilation, or significant dilation in the absence of severe tortuosity, sufficient for diagnosing presence of plus disease?
- Should venous tortuosity and arterial dilation be considered as well?
- Are other vascular characteristics, such as branching pattern or retinal location, important in identifying infants with poor prognosis if left to their untreated natural history?
- What is the best way to ensure that clinicians are not misled by the differences in field-of-view and magnification between the standard plus disease photograph and that of indirect ophthalmoscopy?
The Future: Toward Quantitative Diagnostic Methods
Computer-based image analysis has potential to provide quantifiable, objective measurements to support the diagnosis of plus disease. Several groups of investigators have explored the possibility of automated plus disease detection by measuring the performance of image analysis systems compared to a reference standard of dilated ophthalmoscopy by one or more experienced examiners. These systems rely on specialized segmentation techniques to identify retinal vessels in images and on different algorithms to represent parameters such as vascular dilation and tortuosity.
In several limited studies, these computer-based systems have performed comparably or better than expert ophthalmologists in identifying changes consistent with plus disease from retinal images.6-8 For example, one study determined that the sensitivity of an automated system for detecting tortuosity sufficient for plus disease in 185 wide-angle retinal images was 0.97, compared to 0.65 to 0.81 by three independent pediatric ophthalmologists.6 Another group of researchers found that a group of 11 expert pediatric ophthalmologists and retinal specialists reviewing a series of 20 wide-angle images had sensitivity for plus disease diagnosis from 0.167 to 1.000 and specificity from 0.714 to 1.000, whereas a computer-based system analyzing curvature of retinal arteries had sensitivity 1.000 and specificity 0.846.8
For findings from landmark randomized controlled trials such as Cryo-ROP and ETROP to be applied properly during patient management, identification of plus disease must be made accurately and consistently. Future development of an objective definition of plus disease, based on automated methods for quantifying vascular features such as dilation and tortuosity, could eventually result in improved diagnostic validity and reliability. This would be analogous to widely used tools for computer-based interpretation of electrocardiograms and Papanicolaou smears.9-10 Establishing and validating these quantitative techniques in clinical studies could greatly improve the quality of care provided to premature infants who are at risk for ROP.
1. International Committee for the Classification of Retinopathy of Prematurity. The international classification of retinopathy of prematurity revisited. Arch Ophthalmol. 2005;123(7):991-999.
2. Cryotherapy for Retinopathy of Prematurity Cooperative Group. Multicenter trial of cryotherapy for retinopathy of prematurity. Preliminary results. Arch Ophthalmol. 1988;106(4):471-479.
3. Early Treatment for Retinopathy of Prematurity Cooperative Group. Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol. 2003;121(12):1684-1694.
4. Reynolds JD, Dobson V, Quinn GE, et al. Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies. Arch Ophthalmol. 2002;120(11):1470-1476.
5. Chiang MF, Jiang L, Gelman R, Du YE, Flynn JT. Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol. 2007;125(7):875-880.
6. Wallace DK, Freedman SF, Zhao Z, Jung SH. Accuracy of ROPtool vs. individual examiners in assessing retinal vascular tortuosity. Arch Ophthalmol. 2007;125(11):1523-1530.
7. Gelman R, Jiang L, Du YE, Martinez-Perez ME, Flynn JT, Chiang MF. Plus disease in retinopathy of prematurity: pilot study of computer-based and expert diagnosis. J AAPOS. 2007;11(6):532-540.
8. Koreen S, Gelman R, Martinez-Perez ME, Jiang L, Berrocal AM, Hess DJ, Flynn JT, Chiang MF. Evaluation of a computer-based system for plus disease diagnosis in retinopathy of prematurity. Ophthalmology. 2007;114(12):e59-67.
9. Hongo RH, Goldschlager N. Status of computerized electrocardiography. Cardiol Clin. 2006;24(3):491-504.
10. Ku NN. Automated Papanicolaou smear analysis as a screening tool for female lower genital tract malignancies. Curr Opin Obstet Gynecol. 1999;11(1):41-43.
Dr. Chiang states that he has no financial relationship with the manufacturer of any product discussed in this article or with the manufacturer of any competing product. Dr. Chiang is an unpaid member of the Scientific Advisory Board for Clarity Medical Systems, Pleasanton, CA.