Agreement on ROP severity poor among expert photograph readers

This Danish study in the April issue of the British Journal of Ophthalmology investigated inter-reader agreement on five severity levels of central vascular changes and aggressive posterior retinopathy of prematurity (ROP). Among four international ROP readers, agreement was poor on grading of central vascular changes in RetCam photographs from a preterm population born between 1997 and 2006.

The authors note that the subjective nature of diagnosing such vascular changes possibly resulted in earlier treatment of preterm infants in Denmark between 1997 and 2006. They conclude that the recent increased incidence of treated infants in Denmark is, at least in part, explained by a significant shift in indication for treatment. They point out that two new terms were defined by the revisited International Classification of ROP (ICROP): pre-plus disease, a predictor of sight-threatening plus disease development; and aggressive posterior ROP (APROP), at high risk of rapidly developing into retinal detachment.

Four expert ROP readers - pediatric ophthalmologists or vitreoretinal specialists from the United Kingdom, Sweden and Denmark - participated in the study. They were blinded to any information about the infants with the exception of being told that a high proportion of advanced ROP cases were included. They examined 243 photographs/948 quadrants from 136 infants.

They agreed on the quadrant scores of only 70 (7.38 percent) of the quadrants. Mean scores differed systematically between the readers (permutation test, P < 0.0001). Agreement of all four readers on the presence of aggressive posterior ROP was not achieved for any of the photographs. Readers scored plus disease in at least two quadrants in 95.5 percent of the eyes for which treatment was indicated.

All four readers agreed on whether treatment was indicated in 195 eyes (80.2 percent). However, they unanimously recommended treatment only in 18 eyes (7.4 percent). One reader was found to differ systematically from the others in indicating treatment (Rasch analysis; P = 0.0001). The authors identified a significant shift in indication for treatment between the birth periods of 2000 and 2002, and 2003 and 2006 (Mann-Whitney rank sum test, P < 0.001).

They conclude that inter-reader agreement on central vascular changes, especially when based on more than two rating categories, is poor. They say that further refinement of the revisited ICROP guidelines - or even better, a reliable and valid computer-based image analysis system to further standardize quantification of central vascular changes - is urgently needed.