BACKGROUND: Although safety climate, teamwork, and other non-technical skills in operating rooms probably influence clinical outcomes, direct associations have not been shown, at least partially due to sample size considerations. We report data from a retrospective cohort of anesthesia evaluations that can simplify the design of prospective observational studies in this area. Associations between non-technical skills in anesthesia, specifically anesthesiologists\' quality of clinical supervision and nurse anesthetists\' work habits, and patient and operational factors were examined.
METHODS: Eight fiscal years of evaluations and surgical cases from one hospital were included. Clinical supervision by anesthesiologists was evaluated daily using a nine-item scale. Work habits of nurse anesthetists were evaluated daily using a six-item scale. The dependent variables for both groups of staff were binary, whether all items were given the maximum score or not. Associations were tested with patient and operational variables for the entire day.
RESULTS: There were 40,718 evaluations of faculty anesthesiologists by trainees, 53,772 evaluations of nurse anesthetists by anesthesiologists, and 296,449 cases that raters and ratees started together. Cohen\'s d values were small (≤0.10) for all independent variables, suggesting a lack of any clinically meaningful association between patient and operational factors and evaluations given the maximum scores. For supervision quality, the day\'s count of orthopedic cases was a significant predictor of scores (P = 0.0011). However, the resulting absolute marginal change in the percentage of supervision scores equal to the maximum was only 0.8% (99% confidence interval: 0.2% to 1.4%), i.e., too small to be of clinical or managerial importance. Neurosurgical cases may have been a significant predictor of work habits (P = 0.0054). However, the resulting marginal change in the percentage of work habits scores equal to the maximum, an increase of 0.8% (99% confidence interval: 0.1% to 1.6%), which was again too small to be important.
CONCLUSIONS: When evaluating the effect of assigning anesthesiologists and nurse anesthetists with different clinical performance quality on clinical outcomes, supervision quality and work habits scores may be included as independent variables without concern that their effects are confounded by association with the patient or case characteristics. Clinical supervision and work habits are measures of non-technical skills. Hence, these findings suggest that non-technical performance can be judged by observing the typical small sample size of cases. Then, associations can be tested with administrative data for a far greater number of patients because there is unlikely to be a confounding association between patient and case characteristics and the clinicians\' non-technical performance.

Introduction Whenever a department implements the evaluation of professionals, a reasonable operational goal is to request as few evaluations as possible. In anesthesiology, evaluations of anesthesiologists (by trainees) and nurse anesthetists (by anesthesiologists) with valid and psychometrically reliable scales have been made by requesting daily evaluations of the ratee\'s performance on the immediately preceding day. However, some trainees or nurse anesthetists are paired with the same anesthesiologist for multiple days of the same week. Multiple evaluations from the same rater during a given week may contribute little incremental information versus one evaluation from that rater for the week. We address whether daily evaluation requests could be adjusted adaptively to be made once per week, hopefully substantively reducing the number of evaluation requests. Methods Every day since 1 July 2013 at the studied department, anesthesia residents and fellows have been requested by email to evaluate anesthesiologists\' quality of supervision provided during the preceding day using the De Oliveira Filho supervision scale. Every day since 29 March 2015, the anesthesiologists have been requested by email to evaluate the work habits of the nurse anesthetists during the preceding day. Both types of evaluations were made for interactions throughout the workday together, not for individual cases. The criterion for an electronic request to be sent is that the pair worked together for at least one hour that day. The current study was performed using evaluations of anesthesiologists\' supervision and nurse anesthetists\' work habits through 30 June 2023. Results If every evaluation request were completed by trainees on the same day it was requested, trainees would have received 13.5% fewer requests to evaluate anesthesiologists (9367/69,420), the maximum possible reduction. If anesthesiologists were to do the same for their evaluations of nurse anesthetists, the maximum possible reduction would be 7.1% fewer requests (4794/67,274). However, because most evaluations were completed after the day of the request (71%, 96,451/136,694), there would be fewer requests only if the evaluation were completed before or on the day of the next pairing. Consequently, in actual practice, there would have been only 2.4% fewer evaluation requests to trainees and 1.5% fewer to anesthesiologists, both decreases being significantly less than 5% (both adjusted P <0.0001). Among the trainees\' evaluations of faculty anesthesiologists, there were 1.4% with very low scores, specifically, a mean score of less than three out of four (708/41,778). Using Bernoulli cumulative sum (CUSUM) among successive evaluations, 72 flags were raised over the 10 years. Among those, there were 36% with more than one rater giving an exceptionally low score during the same week (26/72). There were 97% (70/72) with at least one rater contributing more than one score to the recent cumulative sum. Conclusion Conceptually, evaluation requests could be skipped if a rater has already evaluated the ratee that week during an earlier day working together. Our results show that the opportunity for reductions in evaluation requests is significantly less than 5%. There may also be impaired monitoring for the detection of sudden major decreases in ratee performance. Thus, the simpler strategy of requesting evaluations daily after working together is warranted.

Daily evaluations of certified registered nurse anesthetists\' (CRNAs\') work habits by anesthesiologists should be adjusted for rater leniency. The current study tested the hypothesis that there is a pairwise association by rater between leniencies of evaluations of CRNAs\' daily work habits and of didactic lectures. The historical cohorts were anesthesiologists\' evaluations over 53 months of CRNAs\' daily work habits and 65 months of didactic lectures by visiting professors and faculty. The binary endpoints were the Likert scale scores for all 6 and 10 items, respectively, equaling the maximums of 5 for all items, or not. Mixed effects logistic regression estimated the odds of each ratee performing above or below average adjusted for rater leniency. Bivariate errors in variables least squares linear regression estimated the association between the leniency of the anesthesiologists\' evaluations of work habits and didactic lectures. There were 29/107 (27%) raters who were more severe in their evaluations of CRNAs\' work habits than other anesthesiologists (two-sided P < 0.01); 34/107 (32%) raters were more lenient. When evaluating lectures, 3/81 (4%) raters were more severe and 8/81 (10%) more lenient. Among the 67 anesthesiologists rating both, leniency (or severity) for work habits was not associated with that for lectures (P = 0.90, unitless slope between logits 0.02, 95% confidence interval -0.34 to 0.30). Rater leniency is of large magnitude when making daily clinical evaluations, even when using a valid and psychometrically reliable instrument. Rater leniency was context dependent, not solely a reflection of raters\' personality or rating style.

OBJECTIVE: Anesthesiologists can provide psychometrically reliable daily evaluations of certified registered nurse anesthetist (CRNA) work habits for purposes of the mandatory ongoing professional practice evaluation (OPPE). Our goal was to evaluate the validity of assessing CRNA work habits.
METHODS: Observational study.
METHODS: Large teaching hospital.
METHODS: N=77 anesthesiologists evaluated work habits of N=67 CRNAs. The non-technical attribute of work habits was measured on a 6-item scale (e.g., 1=\"Only assumed responsibility when forced to, and failed to follow through consistently\" versus 5=\"Consistently identified tasks and completed them efficiently and thoroughly\"). One year of scores were used to assess validity. Each daily evaluation could also be accompanied by a written comment. Content analysis of comments was performed using two years of data. Statistical analyses were performed using mixed effects logistic regression, treating each anesthesiologist as a fixed effect to compensate for the leniency of their ratings.
RESULTS: The N=77 anesthesiologists\' response rate was 97.3%, obtained at a mean 2.93days after the request. The internal consistency of the scale was large: Cronbach\'s alpha 0.952. Controlling the false discovery rate at 5.0%, among the 67 CRNAs, 8 were significantly below average (each P≤0.0048) and 6 were above average (each P≤0.0018). During the 6months after CRNAs knew that their work habits scores would be used for OPPE, there were significant increases in the scores compared with the preceding 6months (odds ratio 1.93, P<0.0001). Greater CRNA\'s qualitative annual evaluation scores made by the chief CRNA, without knowledge of the work habit scores or comments, were associated with greater odds of the CRNA\'s leniency-adjusted work habit scores equaling 5.00 (odds ratio 1.53, P=0.0004). Comments of negative sentiment made by the anesthesiologists were associated with greater odds of the leniency-adjusted work habit scores being <5.00 (odds ratio 54.5, P<0.0001). Even though the anesthesiologists were already providing information about work habits using the work habits scale, approximately half the comments of negative sentiment included the theme of work habits (92/153, 60.1%).
CONCLUSIONS: Reporting OPPE metrics are mandatory for the maintenance of clinical privileges of anesthesia practitioners in the USA. Basing such peer review on work habits can be quantitative, psychometrically reliable, and valid.

OBJECTIVE: Work habits are non-technical skills that are an important part of job performance. Although non-technical skills are usually evaluated on a relative basis (i.e., \"grading on a curve\"), validity of evaluation on an absolute basis (i.e., \"minimum passing score\") needs to be determined.
METHODS: Survey and observational study.
METHODS: None.
METHODS: None.
METHODS: The theme of \"work habits\" was assessed using a modification of Dannefer et al.\'s 6-item scale, with scores ranging from 1 (lowest performance) to 5 (highest performance). E-mail invitations were sent to all consultant and fellow anesthesiologists at Mayo Clinic in Florida, Arizona, and Minnesota. Because work habits expectations can be generational, the survey was designed for adjustment based on all invited (responding or non-responding) anesthesiologists\' year of graduation from residency.
RESULTS: The overall mean±standard deviation of the score for anesthesiologists\' minimum expectations of nurse anesthetists\' work habits was 3.64±0.66 (N=48). Minimum acceptable scores were correlated with the year of graduation from anesthesia residency (linear regression P=0.004). Adjusting for survey non-response using all N=207 anesthesiologists, the mean of the minimum acceptable work habits adjusted for year of graduation was 3.69 (standard error 0.02). The minimum expectations for nurse anesthetists\' work habits were compared with observational data obtained from the University of Iowa. Among 8940 individual nurse anesthetist work habits scores, only 2.6% were <3.69. All N=65 of the Iowa nurse anesthetists\' mean work habits scores were significantly greater than the Mayo estimate (3.69) for the minimum expectations; all P<0.00024.
CONCLUSIONS: Our results suggest that routinely evaluated work habits of nurse anesthetists within departments should not be compared with an appropriate minimum score (i.e., of 3.69). Instead, work habits scores should be analyzed based on relative reporting among anesthetists.