背景:大型语言模型,例如OpenAI的ChatGPT,在各种应用中表现出了潜力,包括医学教育。以前的研究已经评估了ChatGPT在大学或专业环境中的表现。然而,该模型在标准化入学考试背景下的潜力仍未得到探索。
目的:本研究评估了ChatGPT在英国标准化入学测试中的表现,包括生物医学准入测试(BMAT),大学入学数学考试(TMUA),法律国家能力测验(LNAT),和思维能力评估(TSA),了解其作为教育和考试准备创新工具的潜力。
方法:使用最近的公共资源(2019-2022年)来编译来自BMAT的509个问题的数据集,TMUA,LNAT,和TSA涵盖不同的能力主题,科学知识和应用,数学思维和推理,批判性思维,解决问题,阅读理解,和逻辑推理。本次评估使用传统GPT-3.5模型评估了ChatGPT的性能,注重多项选择题的一致性。根据问题难度对模型的性能进行了分析,在汇总所有年份的考试时,正确回答的比例,并使用二项分布和配对样本(2尾)t检验比较同一考试的论文之间的考试成绩。
结果:在BMAT第2部分(P<.001),TMUA论文1(P<.001)和论文2(P<.001)中,正确回答的比例明显低于错误回答。在BMAT第1部分中没有观察到显著差异(P=2),TSA第1节(P=.7),或LNAT论文1和2,A节(P=3)。ChatGPT在BMAT第1节中的表现优于第2节(P=.047),候选人的最高排名为73%,而最低排名为1%。在TMUA中,它处理问题,但准确性有限,论文之间没有性能差异(P=.6),候选人排名低于10%。在LNAT中,它表现出适度的成功,尤其是在论文2的问题中;然而,学生成绩数据不可用。TSA的表现因年份而异,结果通常适中,候选人排名波动。对于容易到中等难度的问题,观察到类似的趋势(BMAT第1节,P=.3;BMAT第2节,P=.04;TMUA论文1,P<.001;TMUA论文2,P=.003;TSA第1节,P=.8;以及LNAT论文1和2,第A节,P>.99)和难以挑战的(BMAT第1节,P=.7;BMAT第2节,P<.001;TMUA论文1,P=.007;TMUA论文2,P<.001;TSA第1节,P=.3;以及LNAT论文1和2,A节,P=.2)。
结论:ChatGPT有望作为评估能力的学科领域和测试格式的补充工具,解决问题和批判性思维,阅读理解。然而,它在科学和数学知识和应用等领域的局限性突出表明,需要不断发展和与传统学习策略相结合,以充分发挥其潜力。
BACKGROUND: Large language models, such as ChatGPT by OpenAI, have demonstrated potential in various applications, including medical education. Previous studies have assessed ChatGPT\'s performance in university or professional settings. However, the model\'s potential in the context of standardized admission tests remains unexplored.
OBJECTIVE: This study evaluated ChatGPT\'s performance on standardized admission tests in the United Kingdom, including the BioMedical Admissions Test (BMAT), Test of Mathematics for University Admission (TMUA), Law National Aptitude Test (LNAT), and Thinking Skills Assessment (TSA), to understand its potential as an innovative tool for education and test preparation.
METHODS: Recent public resources (2019-2022) were used to compile a data set of 509 questions from the
BMAT, TMUA, LNAT, and TSA covering diverse topics in aptitude, scientific knowledge and applications, mathematical thinking and reasoning, critical thinking, problem-solving, reading comprehension, and logical reasoning. This evaluation assessed ChatGPT\'s performance using the legacy GPT-3.5 model, focusing on multiple-choice questions for consistency. The model\'s performance was analyzed based on question difficulty, the proportion of correct responses when aggregating exams from all years, and a comparison of test scores between papers of the same exam using binomial distribution and paired-sample (2-tailed) t tests.
RESULTS: The proportion of correct responses was significantly lower than incorrect ones in BMAT section 2 (P<.001) and TMUA paper 1 (P<.001) and paper 2 (P<.001). No significant differences were observed in
BMAT section 1 (P=.2), TSA section 1 (P=.7), or LNAT papers 1 and 2, section A (P=.3). ChatGPT performed better in
BMAT section 1 than section 2 (P=.047), with a maximum candidate ranking of 73% compared to a minimum of 1%. In the TMUA, it engaged with questions but had limited accuracy and no performance difference between papers (P=.6), with candidate rankings below 10%. In the LNAT, it demonstrated moderate success, especially in paper 2\'s questions; however, student performance data were unavailable. TSA performance varied across years with generally moderate results and fluctuating candidate rankings. Similar trends were observed for easy to moderate difficulty questions (
BMAT section 1, P=.3;
BMAT section 2, P=.04; TMUA paper 1, P<.001; TMUA paper 2, P=.003; TSA section 1, P=.8; and LNAT papers 1 and 2, section A, P>.99) and hard to challenging ones (BMAT section 1, P=.7;
BMAT section 2, P<.001; TMUA paper 1, P=.007; TMUA paper 2, P<.001; TSA section 1, P=.3; and LNAT papers 1 and 2, section A, P=.2).
CONCLUSIONS: ChatGPT shows promise as a supplementary tool for subject areas and test formats that assess aptitude, problem-solving and critical thinking, and reading comprehension. However, its limitations in areas such as scientific and mathematical knowledge and applications highlight the need for continuous development and integration with conventional learning strategies in order to fully harness its potential.