Evaluating Large Language Models as Medical Consultation Tools for Double Eyelid Surgery: A Cross-Language Study in English and Chinese.

[BACKGROUND] Double eyelid surgery is a common cosmetic procedure that creates a crease in the upper eyelid. Due to insufficient understanding of the procedure, numerous consultations have emerged, placing a heavy burden on plastic surgeons. The rise of large language models (LLMs) offers a potential solution to this issue.

[METHODS] This study collected sixteen questions commonly of concern to individuals seeking the surgery via an online questionnaire and assessed the efficacy of fifteen popular LLMs in answering these questions with both English and Chinese inputs. All responses from the LLMs were scored multidimensionally by three expert eyelid plastic surgeons across dimensions including professionalism, patient friendliness, informativeness, practicality, and logical clarity. The scoring results were statistically analyzed using the Friedman test and Nemenyi post-hoc test.

[RESULTS] With English input, ERNIE-Bot, ChatGPT-4o, and Gemini-2.0-Flash consistently ranked among the top three across most evaluation dimensions. In contrast, Claude-3.7-Sonnet, HuatuoGPT, ZoeGPT, CompliantGPT, and BastionGPT ranked lower across all dimensions, with performance significantly lagging behind the top performers. For Chinese input, DeepSeek-R1 maintained a leading position across all dimensions, forming the first tier alongside DeepSeek-V3, Gemini-2.0-Flash, and ERNIE-Bot. Meanwhile, Claude-3.5-Haiku, ZoeGPT, Llama3.3-70B-Instruct, CompliantGPT, HuatuoGPT, and BastionGPT ranked lower in multiple dimensions, with a significant gap relative to first-tier models.

[CONCLUSION] This study demonstrated LLMs' potential as medical consultation tools for double eyelid surgery, providing useful guidance for both English and Chinese users. Future research should focus on fine-tuning LLMs with more specialized medical data and exploring workflows for surgeon-LLM collaboration to validate their clinical utility.

[LEVEL OF EVIDENCE V] This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .