Implications of ChatGPT for Certifying Board Examinations and Scholarship

Description: A new artificial intelligence (AI) model called "ChatGPT" was introduced in November 2022 and early studies found that ChatGPT performed near the passing standard of all three United States Medical Licensing Examination (USMLE) exams, a level comparable to a third-year medical student. Medical specialty board certification examinations are designed to assess residents’ clinical decision-making with clinically applied patient scenarios. Little is known about ChatGPT’s performance with a more advanced set of questions. In addition, most of the certifying boards have been utilizing longitudinal assessment programs for continuous certification in recent years. This new emerging AI-assisted search may cause unprecedented ethical and exam security issues for the Member Boards of the American Board of Medical Specialties. Lastly, ChatGPT suffers from "AI hallucination," which is a confident response by an AI that does not seem to be justified by its training data. However, the implication of this risk has been rarely explored in medical scholarship.

In this session, we plan to include three parts to share our experience with ChatGPT and reflect on its implications. In the first part, we will report on ChatGPT’s performance on the American Board of Family Medicine’s (ABFM) In-Training Exam (ITE). Secondly, we will share our current policy and considerations regarding ChatGPT and other AI-assisted search tools for the Family Medicine Certification Longitudinal Assessment (FMCLA). In the third part, we will demonstrate how the pilot versions of ChatGPT used fragments of existing citations to fabricate references that appeared to be real articles, and the continued implications of that for medical research.

Citations and Literature References (if none, write n/a): 1. Gilson A, Safranek C, Huang T, et al. How Does ChatGPT Perform on the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. Medical Education; 2022. doi:10.1101/2022.12.23.22283901
2. Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health. 2023;2(2):e0000198.