背景:跨语言评估基于人工智能(AI)的模型对于确保多语言环境中信息的公平访问和准确性至关重要。这项研究旨在比较英语和阿拉伯语中用于传染病查询的AI模型效率。
方法:该研究采用了METRICS清单来设计和报告基于AI的医疗保健研究。测试的AI模型包括ChatGPT-3.5,ChatGPT-4,Bing,还有Bard.询问包括15个关于艾滋病毒/艾滋病的问题,结核病,疟疾,COVID-19和流感。人工智能生成的内容由两名双语专家使用经过验证的CLEAR工具进行评估。
结果:在比较AI模型在传染病查询中的英语和阿拉伯语性能时,注意到变异性。英语查询显示出一贯优异的性能,在巴德的带领下,其次是Bing,ChatGPT-4和ChatGPT-3.5(P=0.012)。在阿拉伯语中也观察到了同样的趋势,尽管没有统计学意义(P=.082)。分层分析显示,在大多数清晰的组成部分中,英语的分数更高,特别是在完整性方面,准确度,适当性,和相关性,尤其是ChatGPT-3.5和Bard.在五个传染病主题中,英语胜过阿拉伯语,除了Bing和Bard的流感查询。四个人工智能模型的英语表现被评为“优秀”,显著优于“高于平均水平”的阿拉伯同行(P=0.002)。
结论:在应对传染病查询时,英语和阿拉伯语在AI模型性能方面存在差异。这种语言差异可能会对以阿拉伯语为母语的AI模型提供的健康内容的质量产生负面影响。建议AI开发人员解决此问题,以增强健康结果为最终目标。
BACKGROUND: Assessment of artificial intelligence (AI)-based models across languages is crucial to ensure equitable access and accuracy of information in multilingual contexts. This study aimed to compare AI model efficiency in English and Arabic for infectious disease queries.
METHODS: The study employed the METRICS checklist for the design and reporting of AI-based studies in healthcare. The AI models tested included ChatGPT-3.5, ChatGPT-4, Bing, and Bard. The queries comprised 15 questions on HIV/AIDS, tuberculosis, malaria, COVID-19, and influenza. The AI-generated content was assessed by two bilingual experts using the validated CLEAR tool.
RESULTS: In comparing AI models\' performance in English and Arabic for infectious disease queries, variability was noted. English queries showed consistently superior performance, with Bard leading, followed by Bing, ChatGPT-4, and ChatGPT-3.5 (P = .012). The same trend was observed in Arabic, albeit without statistical significance (P = .082). Stratified analysis revealed higher scores for English in most CLEAR components, notably in completeness, accuracy, appropriateness, and relevance, especially with ChatGPT-3.5 and Bard. Across the five infectious disease topics, English outperformed Arabic, except for flu queries in Bing and Bard. The four AI models\' performance in English was rated as \"excellent\", significantly outperforming their \"above-average\" Arabic counterparts (P = .002).
CONCLUSIONS: Disparity in AI model performance was noticed between English and Arabic in response to infectious disease queries. This language variation can negatively impact the quality of health content delivered by AI models among native speakers of Arabic. This issue is recommended to be addressed by AI developers, with the ultimate goal of enhancing health outcomes.