OBJECTIVE: In transitioning to competency-based surgical training, the need to clearly define competency is paramount. The purpose of this study is to define the well-prepared foundational resident using the ACGME General Surgery Milestones as our conceptual framework.
METHODS: Participants reflected on their expectations of a well-prepared resident at the end of PGY1, then assigned milestone levels reflecting this level of competence for General Surgery Milestones 1.0 and 2.0. Subcompetency scores were averaged among residents and faculty. The level of the well-prepared foundational resident was determined based on the highest level within one standard deviation of faculty, resident, and total group averages.
METHODS: This took place during a dedicated education retreat at a single, large academic general surgery residency program.
METHODS: Key faculty stakeholders and a representative sample of residents (PGY 1-5) within our institution participated.
RESULTS: Eight faculty and five residents completed Milestones 1.0 and 2.0 scoring. Mean scores between faculty and residents were compared. For 1.0, mean scores for Practice-Based Learning and Improvement 3 (PBLI 3) and Interpersonal Communication Skills 3 (ICS 3) were discernably lower for residents than for faculty (PBLI 3 1.3 (0.3) v 0.9 (0.2), p = 0.01; ICS3 1.6 (0.6) v 1.1 (1), p = 0.01). Scores of 2.0 were comparable across all subcompetency domains. With this broad agreement, Milestone-based competency standards were determined. Descriptive narratives of the KSAs were created for each subcompetency, combining the determined Milestones 1.0 and 2.0 levels.
CONCLUSIONS: We were able to clearly define the competent foundational resident using the ACGME Milestones as a conceptual framework. These Milestone levels reflect the culture and expectations in our department, providing a foundation upon which to build a program of assessment. This methodology can be readily replicated in other programs to reflect specific expectations of the program within the larger ACGME frameworks of competency.

To describe one institution\'s approach to transformation of high-stakes objective structure clinical examinations (OSCEs) from norm-referenced to criterion-referenced standards setting and to evaluate the impact of these changes on OSCE performance and pass rates.
The OSCE writing team at the college selected a modified Angoff method appropriate for high-stakes assessments to replace the two standard deviation method previously used. Each member of the OSCE writing team independently reviewed the analytical checklist and calculated a passing score for active stations on OSCEs. Then the group met to determine a final pass score for each station. The team also determined critical cut points for each station, when indicated. After administration of the OSCEs, scores, pass rates, and need for remediation were compared to the previous norm-referenced method. Descriptive statistics were used to summarize the data.
OSCE scores remained relatively unchanged when switched to a criterion-referenced method, but the number of remediators increased up to 2.6 fold. In the first year, the average score increased from 86.8% to 91.7% while the remediation rate increased from 2.8% to 7.4%. In the third year, the average increased from 90.9% to 92% while the remediation rate increased from 6% to 15.6%. Likewise, the fourth-year average increased from 84.9% to 87.5% while the remediation rate increased from 4.4% to 9%.
Transition to a modified Angoff method did not impact average OSCE score but did increase the number of remediations.

This article was migrated. The article was marked as recommended. Context: We challenge the philosophical acceptability of the Angoff method, and propose an alternative method of standard setting based on how important it is for candidates to know the material each test item assesses, and not how difficult it is for a subgroup of candidates to answer each item. Methods: The practicalities of an alternative method of standard setting are evaluated here, for the first time, with direct comparison to an Angoff method. To negate bias due to any leading effects, a prospective cross-over design was adopted involving two groups of judges (n=7 and n=8), both of which set the standards for the same two 100 item multiple choice question tests, by the two different methods. Results: Overall, we found that the two methods took a similar amount of time to complete. The alternative method produced a higher cut-score (by 12-14%), and had a higher degree of variability between judges\' cut-scores (by 5%). When using the alternative method, judges reported a small, but statistically significant, increase in their confidence to decide accurately the standard (by 3%). Conclusion: This is a new approach to standard setting where the quantitative differences are slight, but there are clear qualitative advantages associated with use of the alternative method.

BACKGROUND: The Upper Digestive Disease (UDD) Tool™ is used to monitor symptom frequency, intensity, and interference across nine symptom domains and includes two Patient-Reported Outcome Measurement Information System (PROMIS) domains assessing physical and mental health. This study aimed to establish cut scores for updated symptom domains through standard setting exercises and evaluate the effectiveness and acceptability of virtual standard setting.
METHODS: The extended Angoff method was employed to determine cut scores. Subject matter experts refined performance descriptions for symptom control categories and achieved consensus. Domains were categorized into good, moderate, and poor symptom control. Two cut scores were established, differentiating good vs. moderate and moderate vs. poor. Panelists estimated average scores for 100 borderline patients per item. Cut scores were computed based on the sum of the average ratings for individual questions, converted to 0-100 scale.
RESULTS: Performance descriptions were refined. Panelists discussed that interpretation of the scores should take into account the timing of symptoms after surgery and patient populations, and the importance of items asking symptom frequency, severity, and interference with daily life. The good/moderate cut scores ranged from 21.3 to 35.0 (mean 28.6, SD 3.6) across domains, and moderate/poor ranged from 47.5 to 71.3 (mean 54.5, SD 7.0).
CONCLUSIONS: Panelists were confident in the virtual standard setting process, expecting valid cut scores. Future studies can further validate the cut scores using patient perspectives and collect patient and physician preferences for displaying contextual items on patient- and physician-facing dashboard.

This article was migrated. The article was marked as recommended. The Angoff standard setting method depends fundamentally on the conceptualisation of an anchor statement. The precise wording and consequent interpretation of anchor statements varies in practice. Emphasis is often placed on standard setting judges\' perceptions of difficulty for a candidate subgroup. The current review focusses on the meaning of anchor statements and argues that when determining the required standard of performance it is more appropriate to consider: (1) what it is important to achieve, and not how difficult it is to achieve it; (2) what all candidates should achieve, and not what a subgroup of candidates would achieve. In summary, current practice should be refined by using an anchor statement which refers to estimating the \'minimum acceptable performance by every candidate\' for each item being tested, and then requiring each judge to score the relevant aspects of importance which could then be combined to derive a cut-score.

This article was migrated. The article was marked as recommended. McMaster University has introduced the Problem-Based Learning (PBL) in medical education over half a century ago. Since then, hundreds of reviews and study reports have identified many critical issues affecting the success of the PBL unit. Nonetheless, we are still debating the efficacy and success of the PBL program. Over half of all medical schools globally have introduced various versions of PBL pedagogy in their medical education program achieving assorted modifications of outcomes. In this paper, I have reviewed from many scholars and through their publications; I have identified eight important Fundamental Roots for a successful PBL program. The success of any PBL program must be evaluated as a whole from the perspective of Standardization, Reliability and Validity working synchronously between these fundamental roots and not the individual PBL unit. The importance of Standardization must consider all the critical issues identified in many reviews and study reports. These issues are one of many factors when incorporated into the fundamental framework of a PBL program will regulate all PBL units in a unified outcome. The educational objective principles will guide the reliability of a PBL program by meeting the institutional mission and students\' career success goals. The \"Assessment as Learning\" should incorporate the \"Holistic and Divergent Approach\" and the longitudinal \"Progress Testing\". These are the principle methods of evaluation to achieve a reliable and valid outcome assessment in a successful PBL program.

This article was migrated. The article was marked as recommended. Introduction Medical educators need to demonstrate that their trainees meet expected competency levels when progressing through medical education. This study aimed to develop competency-based pass/fail cut-scores for a graduation required Objective Structured Clinical Examination (OSCE), and examine validity evidence for new standards. Methods Six clinicians used the modified Angoff method to determine the cut-scores for an 8-station OSCE. The clinicians estimated the percentage of minimally competent students who would answer each checklist item correctly. Inter-rater reliability, differences in other academic achievements between pass/fail groups, educational impact, and response process were examined. Results One hundred seventy-four rising 4th-year medical students participated in the OSCE. The cut-scores determined for the OSCE resulted in a substantially lower failure rate (5% vs. 29% of the previous year). The inter-rater reliability across domains and cases was .98 (95% CI = .97 - .99). The pass/fail groups significantly differed in six of the eight measures of academic achievements included in the study. Discussion The impact of the standards setting was substantial as it significantly reduced the failure rate and burdens of remediation for both students and faculty. The very high inter-rater reliability indicates that the modified Angoff method produced reliable cut-scores. The significant differences between the pass/fail groups in other measures support external validity of the standards and ensure no false passes. The study also supports response process validity by including discussion among judges and check of previous student performances, as well as recruiting and training multiple clinician educators experienced in medical student teaching. Conclusion Findings of the study provide strong evidence supporting validity of the new cut-scores from a wide spectrum of validity metrics, including response process, internal structure, relations to other variables, and consequences. The study also added to the literature the value of the modified Angoff method in determining competency-based standards for OSCEs.

Quantitative measures of systematic differences in OSCE scoring across examiners (often termed examiner stringency) can threaten the validity of examination outcomes. Such effects are usually conceptualised and operationalised based solely on checklist/domain scores in a station, and global grades are not often used in this type of analysis. In this work, a large candidate-level exam dataset is analysed to develop a more sophisticated understanding of examiner stringency. Station scores are modelled based on global grades-with each candidate, station and examiner allowed to vary in their ability/stringency/difficulty in the modelling. In addition, examiners are also allowed to vary in how they discriminate across grades-to our knowledge, this is the first time this has been investigated. Results show that examiners contribute strongly to variance in scoring in two distinct ways-via the traditional conception of score stringency (34% of score variance), but also in how they discriminate in scoring across grades (7%). As one might expect, candidate and station account only for a small amount of score variance at the station-level once candidate grades are accounted for (3% and 2% respectively) with the remainder being residual (54%). Investigation of impacts on station-level candidate pass/fail decisions suggest that examiner differential stringency effects combine to give false positive (candidates passing in error) and false negative (failing in error) rates in stations of around 5% each but at the exam-level this reduces to 0.4% and 3.3% respectively. This work adds to our understanding of examiner behaviour by demonstrating that examiners can vary in qualitatively different ways in their judgments. For institutions, it emphasises the key message that it is important to sample widely from the examiner pool via sufficient stations to ensure OSCE-level decisions are sufficiently defensible. It also suggests that examiner training should include discussion of global grading, and the combined effect of scoring and grading on candidate outcomes.

BACKGROUND: Automated Item Generation (AIG) uses computer software to create multiple items from a single question model. There is currently a lack of data looking at whether item variants to a single question result in differences in student performance or human-derived standard setting. The purpose of this study was to use 50 Multiple Choice Questions (MCQs) as models to create four distinct tests which would be standard set and given to final year UK medical students, and then to compare the performance and standard setting data for each.
METHODS: Pre-existing questions from the UK Medical Schools Council (MSC) Assessment Alliance item bank, created using traditional item writing techniques, were used to generate four \'isomorphic\' 50-item MCQ tests using AIG software. Isomorphic questions use the same question template with minor alterations to test the same learning outcome. All UK medical schools were invited to deliver one of the four papers as an online formative assessment for their final year students. Each test was standard set using a modified Angoff method. Thematic analysis was conducted for item variants with high and low levels of variance in facility (for student performance) and average scores (for standard setting).
RESULTS: Two thousand two hundred eighteen students from 12 UK medical schools participated, with each school using one of the four papers. The average facility of the four papers ranged from 0.55-0.61, and the cut score ranged from 0.58-0.61. Twenty item models had a facility difference > 0.15 and 10 item models had a difference in standard setting of > 0.1. Variation in parameters that could alter clinical reasoning strategies had the greatest impact on item facility.
CONCLUSIONS: Item facility varied to a greater extent than the standard set. This difference may relate to variants causing greater disruption of clinical reasoning strategies in novice learners compared to experts, but is confounded by the possibility that the performance differences may be explained at school level and therefore warrants further study.

The objective was to determine the utilization, frequency, characteristics, and standard-setting methods of progression assessments in pharmacy education.
A survey was sent to 139 United States schools/colleges of pharmacy having an identifiable assessment lead and students enrolled in the doctor of pharmacy program. The survey examined programs\' use, frequency, and characteristics of progression assessments within their curriculum. Respondents also reported any changes made due to the COVID-19 pandemic and which, if any, would be maintained in future years. Analysis consisted of descriptive statistics and thematic coding. This research was deemed exempt by the university\'s institutional review board.
Seventy-eight programs responded to the survey (response rate = 56%). Sixty-seven percent of programs administered at least one progression assessment in 2019-2020. There was some variability in assessment practice, including professional year(s) administered, course(s) involved, and content. Approximately 75% of programs used assessments to ensure student competency in the programs\' learning outcomes and to identify individual student learning deficiencies. Diversity was seen in validity and reliability practices, and most programs used pre-determined cut scores without formal standard setting. Because of the pandemic, 75% of programs changed the assessment delivery mode and 20 programs planned to maintain at least one pandemic-related change in future iterations.
Most pharmacy programs utilize some type of progression assessment within their curriculum. While many schools administer progression assessments, there is little agreement on their purpose, development, and use. The pandemic changed the mode of delivery, which numerous programs will continue with in the future.