BACKGROUND: ChatGPT has attracted a lot of interest worldwide for its versatility in a range of natural language tasks, including in the education and evaluation industry. It can automate time- and labor-intensive tasks with clear economic and efficiency gains.
METHODS: This study evaluated the potential of ChatGPT to automate psychometric analysis of test questions from the 2020 Portuguese National Residency Selection Exam (PNA). ChatGPT was queried 100 times with the 150 MCQ from the exam. Using ChatGPT\'s responses, difficulty indices were calculated for each question based on the proportion of correct answers. The predicted difficulty levels were compared to the actual difficulty levels of the 2020 exam MCQ\'s using methods from classical test theory.
RESULTS: ChatGPT\'s predicted item difficulty indices positively correlated with the actual item difficulties (r (148) = -0.372, p < .001), suggesting a general consistency between the real and the predicted values. There was also a moderate significant negative correlation between the difficulty index predicted by ChatGPT and the number of challenges (r (148) = -0.302, p < .001), highlighting ChatGPT\'s potential for identifying less problematic questions.
CONCLUSIONS: These findings unveiled ChatGPT\'s potential as a tool for assessment development, proving its capability to predict the psychometric characteristics of high-stakes test items in automated item calibration without pre-testing in real-life scenarios.

Despite the increasing implementation of formative assessment in medical education, its\' effect on learning behaviour remains questionable. This effect may depend on how students value formative, and summative assessments differently. Informed by Expectancy Value Theory, we compared test preparation, feedback use, and test-taking motivation of medical students who either took a purely formative progress test (formative PT-group) or a progress test that yielded study credits (summative PT-group). In a mixed-methods study design, we triangulated quantitative questionnaire data (n = 264), logging data of an online PT feedback system (n = 618), and qualitative interview data (n = 21) to compare feedback use, and test-taking motivation between the formative PT-group (n = 316), and the summative PT-group (n = 302). Self-reported, and actual feedback consultation was higher in the summative PT-group. Test preparation, and active feedback use were relatively low and similar in both groups. Both quantitative, and qualitative results showed that the motivation to prepare and consult feedback relates to how students value the assessment. In the interview data, a link could be made with goal orientation theory, as performance-oriented students perceived the formative PT as not important due to the lack of study credits. This led to low test-taking effort, and feedback consultation after the formative PT. In contrast, learning-oriented students valued the formative PT, and used it for self-study or self-assessment to gain feedback. Our results indicate that most students are less motivated to put effort in the test, and use feedback when there are no direct consequences. A supportive assessment environment that emphasizes recognition of the value of formative testing is required to motivate students to use feedback for learning.

This comprehensive review critically examines the UK medical curriculum, with a particular focus on progress testing as an innovative assessment strategy. The curriculum, evolving from foundational sciences to practical applications, is encapsulated in the integrated curriculum model (ICM). This model adeptly combines theoretical knowledge with clinical practice, fostering cognitive, affective, and psychomotor skills among medical students. Central to this review is an exploration of progress testing. This method, grounded in constructivist learning theories, emphasises continuous assessment and professional development. Progress testing\'s regular, comprehensive examinations are instrumental in guiding students through the progressive stages of competence, as outlined in Miller\'s pyramid, from foundational knowledge to clinical proficiency. The review also addresses the broader impacts of progress testing on teaching approaches, student feedback, academic and pastoral support, and quality assurance. By aligning with the dynamic requirements of 21st-century medical training, progress testing not only nurtures well-rounded professionals but also ensures compliance with regulatory bodies like the General Medical Council. Its emphasis on continuous evaluation aligns with the practical realities of a medical career, driving curricular innovation and aligning with regulatory standards. The implementation of progress testing marks a significant advancement in medical education. Its continuous, holistic nature benefits both students and educators, nurturing a more engaged learning attitude and meeting evolving medical needs. The adoption of this assessment strategy is seen as pivotal in shaping competent medical professionals, ready to face the challenges of modern medical practice.

BACKGROUND: Progress testing is an assessment method in which an examination reflecting competencies at graduation is regularly administered to students over multiple years, thereby facilitating self-directed learning. However, the significance of the objective structured clinical examination as a progress test in undergraduate education, needs to be determined. This study provides evidence of the role of the objective structured clinical examination for progress testing and optimal scoring methods for assessing students in different academic years.
METHODS: We conducted a sequential explanatory mixed-methods pilot study. Participants were assessed using the Item Rating Scale, the year-adjusted Global Rating Scale, and the Training Level Rating Scale. The characteristics of each scale were compared quantitatively. In addition, the influence of the objective structured clinical examination as a progress test on learning attitudes was examined. Qualitative data from a post-examination questionnaire were analyzed, using content analysis to explore influences on self-directed learning.
RESULTS: Sixth and fifth year clinical students (n = 235) took the objective structured clinical examination progress test. The total Item Rating Scales were recorded (%) as 59.03 ± 5.27 and 52.64 ± 5.08 (p < 0.01); Training Level Rating Scale was 3.94 ± 0.39 vs 3.22 ± 0.42 (p < 0.01); and the year-adjusted Global Rating Scale was 4.25 ± 0.44 vs 4.32 ± 0.52 (no significant difference), for the sixth and fifth year students, respectively. The correlations across stations and the reliability of each station were satisfactory. Four categories were identified in the qualitative analysis: \"motivation to learn during the clinical clerkship was promoted,\" \"dissatisfied with being asked about things they had not experienced,\" \"confusion about being unable to use conventional test-taking strategies,\" and \"insufficient understanding of competencies at graduation.\" The scores indicated significant differences in performance according to training year.
CONCLUSIONS: This study provides evidence that the objective structured clinical examination can be used as a progress testing tool for undergraduate clinical clerkships. Further enhancement of training opportunities and dissemination of performance competency goals in clerkship curricula are required if we intend to promote self-directed learning through progress testing.

This article was migrated. The article was marked as recommended. McMaster University has introduced the Problem-Based Learning (PBL) in medical education over half a century ago. Since then, hundreds of reviews and study reports have identified many critical issues affecting the success of the PBL unit. Nonetheless, we are still debating the efficacy and success of the PBL program. Over half of all medical schools globally have introduced various versions of PBL pedagogy in their medical education program achieving assorted modifications of outcomes. In this paper, I have reviewed from many scholars and through their publications; I have identified eight important Fundamental Roots for a successful PBL program. The success of any PBL program must be evaluated as a whole from the perspective of Standardization, Reliability and Validity working synchronously between these fundamental roots and not the individual PBL unit. The importance of Standardization must consider all the critical issues identified in many reviews and study reports. These issues are one of many factors when incorporated into the fundamental framework of a PBL program will regulate all PBL units in a unified outcome. The educational objective principles will guide the reliability of a PBL program by meeting the institutional mission and students\' career success goals. The \"Assessment as Learning\" should incorporate the \"Holistic and Divergent Approach\" and the longitudinal \"Progress Testing\". These are the principle methods of evaluation to achieve a reliable and valid outcome assessment in a successful PBL program.

Adaptive testing has a long but largely unrecognized history. The advent of computer-based testing has created new opportunities to incorporate adaptive testing into conventional programmes of study. Relatively recently software has been developed that can automate the delivery of summative assessments that adapt by difficulty or content. Both types of adaptive testing require a large item bank that has been suitably quality assured. Adaptive testing by difficulty enables more reliable evaluation of individual candidate performance, although at the expense of transparency in decision making, and requiring unidirectional navigation. Adaptive testing by content enables reduction in compensation and targeted individual support to enable assurance of performance in all the required outcomes, although at the expense of discovery learning. With both types of adaptive testing, candidates are presented a different set of items to each other, and there is the potential for that to be perceived as unfair. However, when candidates of different abilities receive the same items, they may receive too many they can answer with ease, or too many that are too difficult to answer. Both situations may be considered unfair as neither provides the opportunity to demonstrate what they know. Adapting by difficulty addresses this. Similarly, when everyone is presented with the same items, but answer different items incorrectly, not providing individualized support and opportunity to demonstrate performance in all the required outcomes by revisiting content previously answered incorrectly could also be considered unfair; a point addressed when adapting by content. We review the educational rationale behind the evolution of adaptive testing and consider its inherent strengths and limitations. We explore the continuous pursuit of improvement of examination methodology and how software can facilitate personalized assessment. We highlight how this can serve as a catalyst for learning and refinement of curricula; fostering engagement of learner and educator alike.

UNASSIGNED: Progress testing in education is an assessment principle for the measurement of students\' progress over time, e.g., from start to graduation. Progress testing offers valid longitudinal formative measurement of the growth in the cognitive skills of the individual students within the subjects of the test as well as a tool for educators to monitor potential educational gaps and mismatches within the curriculum in relation to the basic veterinary learning outcomes.
UNASSIGNED: Six veterinary educational establishments in Denmark, Finland, Germany (Hannover), the Netherlands, Norway, and Sweden established in cooperation with the European Association of Establishments for Veterinary Education (EAEVE) a common veterinary item repository that can be used for progress testing in European Veterinary Education Establishments (VEEs), linear as well as computer adaptive, covering the EAEVE veterinary subjects and theoretical \"Day One Competencies.\" First, a blueprint was created, suitable item formats were identified, and a quality assurance process for reviewing and approving items was established. The items were trialed to create a database of validated and calibrated items, and the responses were subsequently psychometrically analyzed according to Modern Test Theory.
UNASSIGNED: In total, 1,836 items were submitted of which 1,342 were approved by the reviewers for trial testing. 1,119 students from all study years and all partners VEEs participated in one or more of six item trials, and 1,948 responses were collected. Responses were analyzed using Rasch Modeling (analysis of item-fit, differential item function, item-response characteristics). A total of 821 calibrated items of various difficulty levels matching the veterinary students\' abilities and covering the veterinary knowledge domains have been banked.
UNASSIGNED: The item bank is now ready to be used for formative progress testing in European veterinary education. This paper presents and discusses possible pitfalls, problems, and solutions when establishing an international veterinary progress test.

UNASSIGNED: The fundamental role of medical education is the transformation of students to doctors, through a process of education and professional identity formation (PIF), which can be informed by several educational, behavioural and emotional factors. PIF has been deemed to be of equal importance to the acquisition of clinical knowledge and skills and includes constructs such as professionalism, leadership and resilience. We aimed to assess professional identity formation, professionalism, leadership and resilience (PILLAR) in the junior years of medical school in the 2020/2021 academic year and illustrate the potential role of quantitative assessment to demonstrate progression in these areas. In this research, we provide the methods and baseline results for the PILLAR study.
UNASSIGNED: We implemented a compulsory assessment in pre-clinical years of graduate entry and direct entry medicine at the Royal College of Surgeons in Ireland University of Medicine and Health Sciences, Dublin, Ireland. Validated scales were used to assess students\' PILLAR. Descriptive and univariable statistical techniques were used to compare student scores between respective years.
UNASSIGNED: A total of 1311 students (92% response rate) provided their consent for research. For the psychometric scales, there were no evident trends among the years on these assessment measures. Results indicated significant differences in all measures, however, these did not correspond to ascending years of seniority.
UNASSIGNED: The PILLAR methodology provides important information on the challenges of quantitatively assessing medical students in the four key areas of PIF, professionalism, leadership, and resilience. Our cross-sectional results point to cohort effects, without the expected progression per year in the cross-sectional data, or suggest that the chosen quantitative measures may be problematic for these constructs in pre-clinical students. Therefore, while we believe that PILLAR has potential as a progress test for these constructs, this will only truly be elucidated by repeated measures of each cohort over time.

Progress tests (PTs) assess applied knowledge, promote knowledge integration, and facilitate retention. Clinical attachments catalyse learning through an appropriate learning context. The relationship between PT results and clinical attachment sequence and performance are under-explored. Aims: (1) Determine the effect of Year 4 general surgical attachment (GSA) completion and sequence on overall PT performance, and for surgically coded items; (2) Determine the association between PT results in the first 2 years and GSA assessment outcomes.
All students enrolled in the medical programme, who started Year 2 between January 2013 and January 2016, were included; with follow up until December 2018. A linear mixed model was applied to study the effect of undertaking a GSA on subsequent PT results. Logistic regressions were used to explore the effect of past PT performance on the likelihood of a student receiving a distinction grade in the GSA.
965 students were included, representing 2191 PT items (363 surgical items). Sequenced exposure to the GSA in Year 4 was associated with increased performance on surgically coded PT items, but not overall performance on the PT, with the difference decreasing over the year. PT performance in Years 2-3 was associated with an increased likelihood of being awarded a GSA distinction grade (OR 1.62, p < 0.001), with overall PT performance a better predictor than performance on surgically coded items.
Exposure to a surgical attachment improves PT results in surgically coded PT items, although with a diminishing effect over time, implying clinical exposure may accelerate subject specific learning. Timing of the GSA did not influence end of year performance in the PT. There is some evidence that students who perform well on PTs in preclinical years are more likely to receive a distinction grade in a surgical attachment than those with lower PT scores.

In March 2020, the first diagnostic radiography degree apprenticeship programme in England was launched at the authors\' institution. As part of the programme development and design, the programme development team explored and then implemented progress testing into a strand of the programme. The objective of this educational perspective is to scrutinise the literature around the use of progress testing in higher education programmes, namely medicine, to explain how and why this decision was reached.
The initial search strategy was developed using the electronic databases CINHAL Complete and SCOPUS. Key words included \'progress test\' and \'medicine\' or \'health\' or \'education\' or \'higher education\'. Eliminating articles that were not relevant, and also identifying and adding additional articles by key authors and experts resulted in thirty-three key articles being considered for review.
The thirty-three articles were a mixture of review articles, empirical research, case studies and conference presentations. Five key themes were identified which are discussed in this article; the evolution of progress testing; advantages of progress testing, disadvantages of progress testing, developing a test framework and academic progression and student feedback.
Progress testing is now well established in pre-registration medical programmes globally. The advantages of progress testing and the use of frequent look rapid remediation appear to be undisputed. Key disadvantages with progress testing were identified as it being an administrative heavy assessment process as well as a perceived bias towards male students undertaking this type of assessment.
Now this assessment practice is established within medicine, it seems reasonable to explore its use in other areas of healthcare, such as radiography.