PDF(Pubmed)
Abstract：
BACKGROUND: ChatGPT has gained global attention recently owing to its high performance in generating a wide range of information and retrieving any kind of data instantaneously. ChatGPT has also been tested for the United States Medical Licensing Examination (USMLE) and has successfully cleared it. Thus, its usability in medical education is now one of the key discussions worldwide.
OBJECTIVE: The objective of this study is to evaluate the performance of ChatGPT in medical biochemistry using clinical case vignettes.
METHODS: The performance of ChatGPT was evaluated in medical biochemistry using 10 clinical case vignettes. Clinical case vignettes were randomly selected and inputted in ChatGPT along with the response options. We tested the responses for each clinical case twice. The answers generated by ChatGPT were saved and checked using our reference material.
RESULTS: ChatGPT generated correct answers for 4 questions on the first attempt. For the other cases, there were differences in responses generated by ChatGPT in the first and second attempts. In the second attempt, ChatGPT provided correct answers for 6 questions and incorrect answers for 4 questions out of the 10 cases that were used. But, to our surprise, for case 3, different answers were obtained with multiple attempts. We believe this to have happened owing to the complexity of the case, which involved addressing various critical medical aspects related to amino acid metabolism in a balanced approach.
CONCLUSIONS: According to the findings of our study, ChatGPT may not be considered an accurate information provider for application in medical education to improve learning and assessment. However, our study was limited by a small sample size (10 clinical case vignettes) and the use of the publicly available version of ChatGPT (version 3.5). Although artificial intelligence (AI) has the capability to transform medical education, we emphasize the validation of such data produced by such AI systems for correctness and dependability before it could be implemented in practice.
OBJECTIVE: The objective of this study is to evaluate the performance of ChatGPT in medical biochemistry using clinical case vignettes.
METHODS: The performance of ChatGPT was evaluated in medical biochemistry using 10 clinical case vignettes. Clinical case vignettes were randomly selected and inputted in ChatGPT along with the response options. We tested the responses for each clinical case twice. The answers generated by ChatGPT were saved and checked using our reference material.
RESULTS: ChatGPT generated correct answers for 4 questions on the first attempt. For the other cases, there were differences in responses generated by ChatGPT in the first and second attempts. In the second attempt, ChatGPT provided correct answers for 6 questions and incorrect answers for 4 questions out of the 10 cases that were used. But, to our surprise, for case 3, different answers were obtained with multiple attempts. We believe this to have happened owing to the complexity of the case, which involved addressing various critical medical aspects related to amino acid metabolism in a balanced approach.
CONCLUSIONS: According to the findings of our study, ChatGPT may not be considered an accurate information provider for application in medical education to improve learning and assessment. However, our study was limited by a small sample size (10 clinical case vignettes) and the use of the publicly available version of ChatGPT (version 3.5). Although artificial intelligence (AI) has the capability to transform medical education, we emphasize the validation of such data produced by such AI systems for correctness and dependability before it could be implemented in practice.
摘要：
背景：ChatGPT由于其在生成广泛的信息和即时检索任何类型的数据方面的高性能，最近引起了全球关注。ChatGPT还通过了美国医学执照考试（USMLE）的测试，并成功清除了它。因此，它在医学教育中的可用性现在是全球范围内的关键讨论之一。
目的：本研究的目的是使用临床案例插图评估ChatGPT在医学生物化学中的表现。
方法：使用10个临床病例小插曲在医学生物化学中评估了ChatGPT的性能。随机选择临床病例插图，并与反应选项一起输入ChatGPT。我们测试了每个临床病例的反应两次。ChatGPT生成的答案被保存并使用我们的参考资料进行检查。
结果：ChatGPT在第一次尝试时就产生了4个问题的正确答案。对于其他情况，ChatGPT在第一次和第二次尝试中产生的应答存在差异.在第二次尝试中，ChatGPT为使用的10个案例中的6个问题提供了正确答案，而4个问题提供了错误答案。但是,令我们惊讶的是，对于病例3,通过多次尝试获得了不同的答案.我们认为这是由于案件的复杂性而发生的，其中涉及以平衡的方法解决与氨基酸代谢相关的各种关键医学方面。
结论：根据我们的研究结果，ChatGPT可能不被视为用于医学教育以改善学习和评估的准确信息提供者。然而,我们的研究受到样本量小(10例临床病例小插曲)和使用公开版本的ChatGPT(3.5版)的限制.尽管人工智能(AI)有能力改变医学教育，我们强调，在实际实施之前，要验证由此类AI系统产生的此类数据的正确性和可靠性。
目的：本研究的目的是使用临床案例插图评估ChatGPT在医学生物化学中的表现。
方法：使用10个临床病例小插曲在医学生物化学中评估了ChatGPT的性能。随机选择临床病例插图，并与反应选项一起输入ChatGPT。我们测试了每个临床病例的反应两次。ChatGPT生成的答案被保存并使用我们的参考资料进行检查。
结果：ChatGPT在第一次尝试时就产生了4个问题的正确答案。对于其他情况，ChatGPT在第一次和第二次尝试中产生的应答存在差异.在第二次尝试中，ChatGPT为使用的10个案例中的6个问题提供了正确答案，而4个问题提供了错误答案。但是,令我们惊讶的是，对于病例3,通过多次尝试获得了不同的答案.我们认为这是由于案件的复杂性而发生的，其中涉及以平衡的方法解决与氨基酸代谢相关的各种关键医学方面。
结论：根据我们的研究结果，ChatGPT可能不被视为用于医学教育以改善学习和评估的准确信息提供者。然而,我们的研究受到样本量小(10例临床病例小插曲)和使用公开版本的ChatGPT(3.5版)的限制.尽管人工智能(AI)有能力改变医学教育，我们强调，在实际实施之前，要验证由此类AI系统产生的此类数据的正确性和可靠性。
