Issues in Designing a Measurement System

A useful measurement system may include quality indicators. For example, one question ophthalmologists commonly face is whether they have dilated a diabetic patient’s eyes at least once every 2 years. How might one validate that a dilated eye examination was performed? One cumbersome validation method would be to video record ophthalmologists while they perform patient examinations and then review that recording to confirm the completion of the dilated eye examination. Other validation options may include (1) written documentation indicating that dilating drops were placed in the eye; (2) notations in the medical record indicating that a peripheral dilated eye examination was performed, such as noting whether it was normal or abnormal; and (3) a diagram or drawing of the retina periphery. All of these would be valid measures.

Once a valid measure has been chosen, reliability needs to be determined. First, the analysis should yield the same results if performed by the same clinician at a different time. For example, are the same results obtained when a measure is made with the same instrument the second time and the third? Measures that minimize errors when repeated have good test-retest reliability, or reproducibility. Second, the analysis should yield the same results if performed on the same subject multiple times. If the person doing the measurement gets the same results on the same subject, after multiple attempts, there is good intrarater reliability. Third, the analysis should yield the same results if performed by different clinicians. Organizations should design measures (as well as a training system for the staff and a support system that will capture and analyze the data) to allow different people to use the same measure and obtain similar results. Measures that have this characteristic are said to have good interrater reliability.

In research, 2 statistical techniques are commonly used to determine the degree of agreement between 2 different tests that detect a particular disease in a group of patients. One method is to simply tally the number of times that the results of the 2 tests agree (ie, both tests indicate disease present, both tests indicate no disease) and then divide that number by the total number of items being assessed, thereby yielding the percent agreement. Another method, the к (kappa) statistic, measures the agreement between 2 or more individuals or entities while taking into account the potential for agreement via chance alone. Kappas greater than 0.75 represent excellent agreement; those from 0.40 to 0.75, fair to good agreement; and those below 0.40, poor agreement. However, not all experts agree on these kappa cutoff points; and other cutoffs for agreement may be recommended.

Once a valid and reliable measure has been established, the organization should first consider the population of interest and the inclusion and exclusion criteria. For example, a study of the quality of cataract surgery might exclude retina specialists (exclusion criterion) and include only comprehensive ophthalmologists who spend at least 50% of their time seeing patients (inclusion criterion).

Second, the organization needs to determine whether the studied event occurs at a frequency that will allow meaningful differences to be found. Are the events so rare (“floor effect”) or so common (“ceiling effect”) that little value is to be gained in using such a measurement system? Organizations should consider conducting a pilot study to investigate these issues before implementing a system.

Third, whenever possible, organizations should use measures that are easily obtained, yet are valid and reliable. Systems that do not require much additional work are more practical for the purpose of monitoring practices. Thus, billing files may provide sufficient information on the completion of specific process quality steps, such as the performance of regular visual field testing in patients with glaucoma, and outcomes, such as incidence of suprachoroidal hemorrhage after intraocular surgery.

Excerpted from BCSC 2020-2021 series: Section 1 - Update on General Medicine. For more information and to purchase the entire series, please visit https://www.aao.org/bcsc.