OpenAI’s ChatGPT, the popular large language model, has once again demonstrated its capabilities by successfully completing the United States Medical Licensing Exam (USMLE).
The USMLE is a standardized set of three tests that assess expert-level knowledge required for medical licensure in the United States. Researchers evaluated ChatGPT’s performance on this exam to assess the AI’s medical knowledge compared to that of expert human clinicians.
The Step 1 exam focuses on basic science, pharmacology, and pathophysiology and is typically taken by medical students who’ve finished two years of didactic and problem-based learning. Medical students spend around 300-400 hours preparing for this exam. The Step 2CK exam is primarily taken by fourth-year medical students who have completed 1.5 to 2 years of clinical rotations, and it focuses on clinical reasoning, medical management, and bioethics. Lastly, the Step 3 exam is taken by physicians who have usually completed at least 0.5 to 1 year of postgraduate medical education.
ChatGPT achieved an accuracy level at or very close to the passing threshold of 60%. Its accuracy for USMLE Steps 1, 2CK, and 3 was 75.0%/45.4%, 61.5%/54.1%, and 68.8%/61.5%, respectively. According to the researchers, this is the first time that an AI system has achieved this benchmark without any specialised inputs from human trainers.
The abstract paper published by the researchers notes that the viral AI chatbot passed the tests without any special training or prompts from human trainers. In addition, the AI bot added a “high level of concordance and insight in its explanations” during the tests.
According to the study, ChatGPT displayed comprehensible reasoning and valid clinical insights, highlighting the potential for trust and explainability in AI-assisted medical education and clinical decision-making. The researchers suggest that large language models like ChatGPT may assist human learners in a medical education setting and lead to further integration into clinical decision-making.
ALSO READ l Love in time of ChatGPT: AI wrote this breakup letter
