Skip to main content

  • ChatGPT-4 needs some work before it can be useful in clinical applications

    Comprehensive Ophthalmology, Retina/Vitreous

    Add to My Bookmarks

    Review of: Appropriateness and readability of ChatGPT-4 generated responses for surgical treatment of retinal diseases

    Momenaei B, Wakabayashi T, Shahlaee A, et al. Ophthalmology Retina, in press 2023

    The online artificial intelligence (AI) platform GhatGPT-4 appears to offer appropriate responses to questions about managing retinal detachment, macular hole, and epiretinal membrane. However, the readability of these responses is less than ideal.

    Study design

    This retrospective cross-sectional study evaluated the readability and appropriateness of online artificial-intelligence (AI)–generated responses to common questions about the surgical management of retinal detachment, macular hole, and epiretinal membrane. The ChatGPT-4 platform was asked to respond to a list of common retina surgery–related questions 3 separate times. Two independent retina specialists graded the appropriateness of the responses (i.e., accuracy and alignment with what these specialists would recommend based on their clinical experience). Readability was assessed using 5 established indices (Flesch Reading Ease score, Flesch Kincaid Grade Level, Gunning Fog Index, Coleman–Liau Index, and Simple Measure of Gobbledygook Index) and an online application to generate an overall reading level score.

    Outcomes

    Responses were deemed appropriate in 84.6% (33/39), 92% (23/25), and 91.7% (22/24) of the questions related to retinal detachment, macular hole, and epiretinal membrane, respectively. However, the readability scores indicated that the answers were difficult or very difficult for the average lay person to read.

    Limitations

    There was no indication of whether the ChatGPT-4 program was restarted with each question or whether AI learning may have occurred in response to prior questions.

    Clinical significance

    This analysis shows that while AI programs like ChatGPT-4 may offer appropriate and fairly consistent responses to common retina surgical questions, the responses are still not easy to read and interpret. The data suggest that this application of AI is not yet ready for clinical use.

    Financial Disclosures: Dr. Rebecca Soares discloses no financial relationships.