Large Language Models and Medical Board Certification: Risks and Opportunities for Physician Learning

Description: Large language models (LLMs) such as OpenAI’s ChatGPT have raised alarms by passing some medical school examinations without any specialized fine-tuning. In the US, licensed physicians demonstrate their competence and professionalism by maintaining board certification with their specialty-specific certifying boards. A fundamental component of the certification process is the board certification exam. Recently, many medical boards have moved from a closed-book, single-day exam taken every 6-10 years to an open-book, longitudinal assessment format with questions delivered online every quarter. Although most evidence suggests that the longitudinal assessment format is better for maintaining competence and ensuring high-quality continuing workforce education, these exams’ formats are vulnerable to unwanted LLM assistance.

We plan to discuss questions around the risks and opportunities for medical education and physician certification presented by LLMs’ increased spread. For example, should certifying boards and medical schools redesign exams with a focus on multi-dimensional problem-solving instead of memorization? How can physicians and physicians-in-training demonstrate their competence, professionalism, and value? What are appropriate ways to use and evaluate AI tools as supports for human decision-making? We will discuss our own experiments assessing LLM performance on certification activities from the American Board of Family Medicine.

Citations and Literature References (if none, write n/a): 1. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, Madriaga M, Aggabao R, Diaz-Candido G, Maningo J, Tseng V. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health. 2023 Feb 9;2(2):e0000198.

2. Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, Scales N, Tanwani A, Cole-Lewis H, Pfohl S, Payne P. Large Language Models Encode Clinical Knowledge. arXiv preprint arXiv:2212.13138. 2022 Dec 26.