Do Experts Agree on the Diagnoses Assigned to Uveitis Cases?
American Journal of Ophthalmology, February 2018
Jabs et al. conducted an interobserver study to ascertain the level of expert agreement on diagnoses assigned to cases of uveitis. They found that independent assessment yielded only moderate agreement, which improved greatly after conference calls with colleagues.
For their study, 5 committees (each with 9 uveitis experts) reviewed a total of 5,766 cases from a preliminary database representing 25 uveitic diseases. Initially, the experts voted online, independently, on whether each case coincided with its assigned diagnosis. The agreement statistic (κ) was calculated for 36 pairwise comparisons per disease, and the mean κ was calculated for each disease. After independent voting, committees held consensus conference calls to discuss the cases that lacked supermajority agreement, defined as > 75%. Nominal group techniques were applied to attempt to reach the targeted level of agreement.
The mean κ achieved from independent voting was 0.39, denoting moderate agreement. Disease-specific variation ranged from 0.23 (for toxoplasmic retinitis) to 0.79 (for cytomegalovirus anterior uveitis). After the conference calls, supermajority agreement was attained for approximately 99% of cases, with disease-specific variations ranging from 96% to 100%. The remaining cases (approximately 1%) were permanently “tabled.” Ultimately, 71% of the cases evaluated were accepted into the final database and 28% were rejected. Acceptance rates ranged from 42% for herpes simplex anterior uveitis to 92% for serpiginous-like tuberculous choroiditis. Throughout the study, perfect agreement (κ = 1.00) was achieved by only 1 pair of experts. For several diseases, the agreement of at least 1 pair of experts was essentially “chance alone.”
Although diagnostic agreement was only moderate early in the study, it was improved by collaborative discussion. Only during the conference calls did many essential disease-specific acceptance/rejection criteria begin to emerge. The obstacles to consensus that arose in this study indicate the need for clear, validated, widely accepted classification criteria for uveitic conditions. With better criteria, the data derived from case series, cohort studies, and multicenter trials should become more homogeneous and thus more useful for establishing accurate diagnoses.
The original article can be found here.