PDF(Pubmed)
Abstract：
BACKGROUND: Artificial intelligence (AI) is a rapidly developing field with the potential to transform various aspects of health care and public health, including medical training. During the \"Hygiene and Public Health\" course for fifth-year medical students, a practical training session was conducted on vaccination using AI chatbots as an educational supportive tool. Before receiving specific training on vaccination, the students were given a web-based test extracted from the Italian National Medical Residency Test. After completing the test, a critical correction of each question was performed assisted by AI chatbots.
OBJECTIVE: The main aim of this study was to identify whether AI chatbots can be considered educational support tools for training in public health. The secondary objective was to assess the performance of different AI chatbots on complex multiple-choice medical questions in the Italian language.
METHODS: A test composed of 15 multiple-choice questions on vaccination was extracted from the Italian National Medical Residency Test using targeted keywords and administered to medical students via Google Forms and to different AI chatbot models (Bing Chat, ChatGPT, Chatsonic, Google Bard, and YouChat). The correction of the test was conducted in the classroom, focusing on the critical evaluation of the explanations provided by the chatbot. A Mann-Whitney U test was conducted to compare the performances of medical students and AI chatbots. Student feedback was collected anonymously at the end of the training experience.
RESULTS: In total, 36 medical students and 5 AI chatbot models completed the test. The students achieved an average score of 8.22 (SD 2.65) out of 15, while the AI chatbots scored an average of 12.22 (SD 2.77). The results indicated a statistically significant difference in performance between the 2 groups (U=49.5, P<.001), with a large effect size (r=0.69). When divided by question type (direct, scenario-based, and negative), significant differences were observed in direct (P<.001) and scenario-based (P<.001) questions, but not in negative questions (P=.48). The students reported a high level of satisfaction (7.9/10) with the educational experience, expressing a strong desire to repeat the experience (7.6/10).
CONCLUSIONS: This study demonstrated the efficacy of AI chatbots in answering complex medical questions related to vaccination and providing valuable educational support. Their performance significantly surpassed that of medical students in direct and scenario-based questions. The responsible and critical use of AI chatbots can enhance medical education, making it an essential aspect to integrate into the educational system.
OBJECTIVE: The main aim of this study was to identify whether AI chatbots can be considered educational support tools for training in public health. The secondary objective was to assess the performance of different AI chatbots on complex multiple-choice medical questions in the Italian language.
METHODS: A test composed of 15 multiple-choice questions on vaccination was extracted from the Italian National Medical Residency Test using targeted keywords and administered to medical students via Google Forms and to different AI chatbot models (Bing Chat, ChatGPT, Chatsonic, Google Bard, and YouChat). The correction of the test was conducted in the classroom, focusing on the critical evaluation of the explanations provided by the chatbot. A Mann-Whitney U test was conducted to compare the performances of medical students and AI chatbots. Student feedback was collected anonymously at the end of the training experience.
RESULTS: In total, 36 medical students and 5 AI chatbot models completed the test. The students achieved an average score of 8.22 (SD 2.65) out of 15, while the AI chatbots scored an average of 12.22 (SD 2.77). The results indicated a statistically significant difference in performance between the 2 groups (U=49.5, P<.001), with a large effect size (r=0.69). When divided by question type (direct, scenario-based, and negative), significant differences were observed in direct (P<.001) and scenario-based (P<.001) questions, but not in negative questions (P=.48). The students reported a high level of satisfaction (7.9/10) with the educational experience, expressing a strong desire to repeat the experience (7.6/10).
CONCLUSIONS: This study demonstrated the efficacy of AI chatbots in answering complex medical questions related to vaccination and providing valuable educational support. Their performance significantly surpassed that of medical students in direct and scenario-based questions. The responsible and critical use of AI chatbots can enhance medical education, making it an essential aspect to integrate into the educational system.
摘要：
背景：人工智能（AI）是一个快速发展的领域，具有改变医疗保健和公共卫生各个方面的潜力，包括医学培训。在为五年级医学生开设的“卫生与公共卫生”课程期间，使用AI聊天机器人作为教育支持工具，进行了有关疫苗接种的实用培训课程。在接受疫苗接种的特定培训之前，对学生进行了基于网络的测试，该测试是从意大利国家医学住院医师测试中提取的。完成测试后，AI聊天机器人协助对每个问题进行了严格的更正。
目的：这项研究的主要目的是确定AI聊天机器人是否可以被视为公共卫生培训的教育支持工具。次要目标是评估不同AI聊天机器人在意大利语复杂的多项选择医学问题上的表现。
方法：从意大利国家医疗居住权测试中提取了由15个关于疫苗接种的多项选择题组成的测试，使用有针对性的关键词，并通过GoogleForms和不同的AI聊天机器人模型（BingChat，ChatGPT,Chatsonic,谷歌吟游诗人，和YouChat）。考试的纠正是在教室里进行的,专注于对聊天机器人提供的解释进行批判性评估。进行了Mann-WhitneyU测试，以比较医学生和AI聊天机器人的表现。在培训体验结束时匿名收集学生反馈。
结果：总计，36名医学生和5个AI聊天机器人模型完成了测试。学生在15分中的平均得分为8.22（SD2.65），而AI聊天机器人的平均得分为12.22（SD2.77）。结果表明，两组之间的性能差异具有统计学意义（U=49.5，P<.001），具有较大的效应大小(r=0.69)。当按问题类型划分时(直接，基于场景的，和否定)，在直接(P<.001)和基于情景(P<.001)的问题上观察到显著差异，但不是在否定的问题（P=.48）。学生报告对教育经历的满意度很高（7.9/10），表达强烈的重复体验的愿望（7.6/10）。
结论：这项研究证明了AI聊天机器人在回答与疫苗接种相关的复杂医学问题和提供有价值的教育支持方面的有效性。在直接和基于情景的问题上，他们的表现大大超过了医学生。负责任和批判性地使用人工智能聊天机器人可以增强医学教育，使其成为融入教育系统的一个重要方面。
目的：这项研究的主要目的是确定AI聊天机器人是否可以被视为公共卫生培训的教育支持工具。次要目标是评估不同AI聊天机器人在意大利语复杂的多项选择医学问题上的表现。
方法：从意大利国家医疗居住权测试中提取了由15个关于疫苗接种的多项选择题组成的测试，使用有针对性的关键词，并通过GoogleForms和不同的AI聊天机器人模型（BingChat，ChatGPT,Chatsonic,谷歌吟游诗人，和YouChat）。考试的纠正是在教室里进行的,专注于对聊天机器人提供的解释进行批判性评估。进行了Mann-WhitneyU测试，以比较医学生和AI聊天机器人的表现。在培训体验结束时匿名收集学生反馈。
结果：总计，36名医学生和5个AI聊天机器人模型完成了测试。学生在15分中的平均得分为8.22（SD2.65），而AI聊天机器人的平均得分为12.22（SD2.77）。结果表明，两组之间的性能差异具有统计学意义（U=49.5，P<.001），具有较大的效应大小(r=0.69)。当按问题类型划分时(直接，基于场景的，和否定)，在直接(P<.001)和基于情景(P<.001)的问题上观察到显著差异，但不是在否定的问题（P=.48）。学生报告对教育经历的满意度很高（7.9/10），表达强烈的重复体验的愿望（7.6/10）。
结论：这项研究证明了AI聊天机器人在回答与疫苗接种相关的复杂医学问题和提供有价值的教育支持方面的有效性。在直接和基于情景的问题上，他们的表现大大超过了医学生。负责任和批判性地使用人工智能聊天机器人可以增强医学教育，使其成为融入教育系统的一个重要方面。
