背景:推荐信被认为是骨科住院医师应用整体排名的重要因素。针对传统LOR的固有偏见和有限的比较价值,2017年引入了标准化推荐信(SLOR)。它包括关于申请人骨科资格的标准化问题,旨在实现候选人之间的公平比较。然而,不同和不一致的发现使得很难从关于SLOR和叙述性推荐信的个别研究中得出有意义的结论.
目的:在这篇系统综述中,我们问:(1)申请人的SLOR评分在九个领域和总结分数中的分布如何?(2)是否有与SLOR评分相关的申请人特征?(3)SLOR和叙述性推荐信中是否存在性别和种族偏见?
方法:PubMed,EBSCO,和GoogleScholar电子数据库于2023年3月20日进行了查询,以确定在2017年1月1日至2023年3月20日之间评估SLOR和叙述性推荐信的所有研究。包括评估骨科SLOR或叙述性LOR的文章。系统审查,病例报告,数据库之间的重复研究,和灰色文献,如摘要和预印本服务器上的文章被排除在外。最终分析中包括了8项研究,其中包括6179名申请人和18,987封推荐信。申请人的课程从2014年到2020年不等。两名评审员使用JoannaBriggs研究所(JBI)的横断面研究工具独立评估了纳入研究的质量。纳入研究的平均JBI评分为7.4分,最高为8分,分数越高表明质量越好。主要结果是确定申请人SLOR排名在其总结分数中的分布。总结性分数被分成排名以匹配,前三分之一,中间三分之一,低三分之一,也不适合。四项研究报告了申请人的SLOR得分。我们的次要研究目标是评估与SLOR求和分数相关的申请人特征,并评估SLOR或叙事建议中存在的任何偏倚。五项研究比较了包括性别在内的申请人特征的SLOR评分,种族,USMLE第1步得分,USMLE第2步得分,AlphaOmegaAlpha(AΩA)会员资格,职员等级,和出版物。在五项研究中还评估了性别和种族偏见。四项研究利用语言分析软件搜索推荐信,以了解性别和种族在单词类别使用方面的差异。
结果:研究一致发现,与数学上可能的情况相比,在最高百分位数块中识别出的候选人百分比更高。例如,两项研究统计了排名前10%的候选人比例,发现36%和42%被评为前10%。同样,文章发现,超过87%的申请人得分位居前三分之一。研究结果好坏参半,但总的来说,他们认为AΩA状态,更高的第一步得分,和更多的研究出版物与增加的SLOR评分有轻微的正相关.我们没有发现对女性有性别偏见的证据,事实上,大多数评估单词选择的研究发现,女性候选人的单词选择通常更强。同样,根据申请人种族,在SLOR中使用的单词类别中未发现一致的差异.
结论:最值得注意的是,在推荐信中,数学上不可能的申请人比例被计入最高百分位数。这损害了读者区分候选人的能力。AOA状态和研究出版物等因素与SLOR得分呈适度正相关。对妇女或少数族裔候选人的性别偏见并不明显。
结论:我们建议SLOR组作者,根据书记员或居住董事的共识评估,应该考虑。我们还建议将每个字母作者(或小组)的平均值和SD分数与他们的字母一起合并。此外,研究表明,提交所有四封SLOR信可以大大提高申请人的成功机会。我们建议学生有策略地选择并提交最大允许数量的SLOR。
BACKGROUND: Letters of recommendation are considered an important factor in the holistic ranking of orthopaedic residency applications. The standardized letter of recommendation (SLOR) was introduced in 2017 in response to the inherent bias and limited comparative value of traditional LORs. It includes standardized questions about an applicant\'s orthopaedic qualifications, designed to enable fair comparisons among candidates. However, disparate and inconsistent findings have made it difficult to draw meaningful conclusions from individual studies on the SLOR and narrative letter of recommendation.
OBJECTIVE: In this systematic
review we asked: (1) What is the distribution of applicant SLOR rating among nine domains and summative scores? (2) Are there applicant characteristics associated with SLOR ratings? (3) Is there gender and racial bias in the SLOR and narrative letter of recommendation?
METHODS: PubMed, EBSCO, and Google Scholar electronic databases were queried on March 20, 2023, to identify all studies evaluating SLOR and narrative letter of recommendations between January 1, 2017, and March 20, 2023. Articles that evaluated orthopaedic SLOR or narrative LORs were included. Systematic reviews, case reports, duplicate studies between databases, and grey literature such as abstracts and articles on preprint servers were excluded. Eight studies including 6179 applicants and 18,987 letters of recommendation were included in the final analysis. The applicant classes ranged from years 2014 to 2020. Two reviewers independently evaluated the quality of the included studies using the Joanna Briggs Institute (JBI) tool for cross-sectional studies. The mean JBI score of included studies was 7.4 out of a maximum of 8, with higher scores indicating better quality. The primary outcome was to determine the distribution of applicant SLOR rankings in their summative score. Summative scores were separated into ranked to match, top one-third, middle one-third, lower one-third, and not a fit. Four studies reported the summative SLOR scores of applicants. Our secondary study goal was to assess applicant characteristics associated with SLOR summative scores and assess any bias present in the SLOR or narrative recommendation. Five studies compared SLOR ratings across applicant characteristics including gender, race, USMLE Step 1 score, USMLE Step 2 score, Alpha Omega Alpha (AΩA) membership, clerkship grades, and publications. Gender and racial bias were also assessed across five studies. Four studies utilized a linguistic analysis software to search letters of recommendation for differences in word category use by gender and race.
RESULTS: Studies consistently found that a higher percentage of candidates were identified in the top percentile blocks than is mathematically possible. For example, the two studies that tallied the proportion of candidates ranking in the top 10% of applicants found that 36% and 42% were rated as being in the top 10%. Similarly, articles found more than 87% of applicants scored in the top one-third. Studies had mixed results, but in general, they suggested that AΩA status, higher Step 1 scores, and more research publications have a slightly positive association with increased SLOR scores. We found no evidence of gender bias against women, and in fact, most studies evaluating word choices found word choices for women candidates generally were stronger. Similarly, no consistent disparities were identified in word categories utilized in SLORs based on applicant race.
CONCLUSIONS: Most notably, a mathematically impossible proportion of applicants were counted in the top percentiles in letters of recommendation. This compromises readers\' abilities to differentiate candidates. Factors like AOA status and research publications displayed a modest positive correlation with SLOR scores. Gender bias against women or candidates from racial minority groups was not evident.
CONCLUSIONS: We suggest that group SLOR authorship, with a consensus assessment by clerkship or residency directors, should be considered. We also recommend the incorporation of mean and SD scores for each letter writer (or group) alongside their letters. Furthermore, studies indicate that submitting all four SLOR letters can substantially improve an applicant\'s chances of success. We advise students to choose strategically and submit the maximum allowable number of SLORs.