This paper reiterates the importance of the role of multisource feedback (MSF) in continuing medical education/continuing professional development (CME/CPD) and its impact on doctors\' performance and patient experience globally. It summarises a unique initiative of robust utilisation of internationally recognised multisource feedback tools in an outpatient child and adolescent mental health service (CAMHS) in Qatar. The process involved the effective adoption and administering of the General Medical Council\'s (GMC) self-assessment questionnaire (SQ), patient questionnaire (PQ), and colleague questionnaire (CQ) followed by the successful incorporation of these tools in CME/CPD. The original version of the PQ questionnaire and the instructions to the patient document were translated into Arabic through the blind back-translation technique. This initiative of introducing gold-standard MSF tools and processes into clinical practice, among other quality-improvement projects, has contributed to the improvement of service standards and doctors\' clinical practice. Patient satisfaction was measured through the annual patient experience analysis using the Experience of Service Questionnaire (ESQ) whereas changes in doctors\' performance were evaluated by comparing annual appraisal scores before and after implementation of this initiative. We have demonstrated that when MSF is obtained impartially and transparently using recognised and valid tools, it can improve patient experience and enhance doctors\' performance.

UNASSIGNED: Interpersonal skills are crucial for successful clinician-patient interactions. To prepare future optometrists for clinical practice, pedagogical evaluation is important to support the implementation of new strategies for teaching and evaluating interpersonal skills.
UNASSIGNED: Optometry students largely develop their interpersonal skills through in-person patient interactions. Telehealth is increasing, yet strategies to develop the interpersonal skills of students for teleconsulting have not been explored. This study aimed to assess the feasibility, effectiveness and perceived usefulness of an online, multisource (patients, clinicians and students) evaluation and feedback program for developing interpersonal skills.
UNASSIGNED: Via an online teleconferencing platform, optometry students (n = 40) interacted with a volunteer patient, observed by a teaching clinician. Patients and clinicians evaluated the interpersonal skills of the student in two ways: (1) qualitative written feedback, and (2) quantitative rating (Doctors\' Interpersonal Skills Questionnaire). All students received written patient and clinician feedback after the session, but not their quantitative ratings. A subset of students (n = 19) completed two sessions, self-ratings, and were provided with their written feedback and an audiovisual recording from their first interaction before completing the second session. All participants were invited to complete an anonymous survey at program completion.
UNASSIGNED: Patient and clinician overall interpersonal skills ratings were positively correlated (Spearman\'s r = 0.35, p = 0.03) and showed moderate agreement (Lin\'s concordance coefficient = 0.34). Student self-ratings did not match patient ratings (r = 0.01, p = 0.98), whereas there was moderate agreement between clinician and student ratings (Lin\'s concordance coefficient = 0.30). Ratings improved at the second visit (p = 0.01). Patient ratings were higher than clinicians (p = 0.01) and students (p = 0.03). All participants agreed that the program was feasible, useful and effective at fostering good interpersonal skills.
UNASSIGNED: Multisource feedback about interpersonal skills contributes to improvement in student performance. Patients and clinicians can evaluate and provide useful feedback to optometry students about their interpersonal skills using online methods.

BACKGROUND: Medical boards and healthcare providers internationally are coming under increasing pressure to attract international medical graduates (IMGs) and overseas trained doctors (OTDs) to cope with predicted general practice (GP) doctor shortages. Various pathways to registration are made available for this purpose. There is very little understanding of the effects of different training pathways to licensing and registration on the ability of IMGs and OTDs, as well as locally trained doctors, to acquire the desirable professional skills deemed necessary for working effectively in the primary care sector.
METHODS: Feedback from patients was collected at the end of their scheduled consultation with their doctor using a questionnaire consisting of 13 Likert scale items that asked them to rate their experience of the consultation. Feedback was obtained for doctors going through the Royal Australian College of General Practice (RACGP) Practice Experience Program (PEP) and the Australian General Practice Training Program (AGPT), with the former intended primarily for IMGs and OTDs, and the latter for local medical graduates including from New Zealand. Patient feedback was also obtained for patients visiting already Fellowed and experienced GPs for comparative purposes, resulting in data for three groups of doctors (two trainee, one already Fellowed). Rater consistency and agreement measures, analysis of variance, principal component analysis, t-tests and psychometric network analysis were undertaken between and within groups to identify similarities and differences in patient experience and professionalism of doctors.
RESULTS: There was a small but significant difference in average patient raw scores given to PEP and AGPT doctors (90.25, 90.97%), with the highest scores for \'Respect shown\' (92.24, 93.15%) and the lowest for \'Reassurance\' 89.38, 89.84%). Male patients gave lower scores (89.56%) than female patients (91.23%) for both groups of doctors. In comparison, patients gave experienced GPs an average 91.38% score, with male patients giving a lower average score than female patients (90.62, 91.93%). Two components were found in the patient data (interpersonal communication, caring/empathy) that account for over 80% of the variance. When patient scores were aggregated by doctor, the average PEP and AGPT doctor scores received were 90.27 and 90.99%, in comparison to the average experienced GP score of 91.43%. Network analysis revealed differences in the connectedness of items between these two groups as well as in comparison with experienced GPs, suggesting that PEP doctors\' skills are less cohesively developed in the areas of listening ability, explaining and providing reassurance.
CONCLUSIONS: The small but statistically significant differences between doctor groups reported in this preliminary study are supplemented by percentile analysis, network analysis and principal component analysis to identify areas for further exploration and study. There is scope for improving the integration of interpersonal communication skills of GPs in Training with their caring and empathy skills, when compared with experienced GPs as a benchmark. Suggestions are made for enhancing professional skills from a patients\' perspective in future training programs.

UNASSIGNED: How to evaluate clinical educators is an important question in faculty development. The issue of who are best placed to evaluate their performance is also critical. However, the whos and the hows of clinical educator evaluation may differ culturally. This study aims to understand what comprises suitable evaluation criteria, alongside who is best placed to undertake the evaluation of clinical educators in medicine within an East Asian culture: specifically Taiwan.
UNASSIGNED: An 84-item web-based questionnaire was created based on a literature review and medical educational experts\' opinions focusing on potential raters (i.e., who) and domains (i.e., what) for evaluating clinical educators. Using purposive sampling, we sent 500 questionnaires to clinical educators, residents, Post-Graduate Year Trainees (PGYs), Year-4~6/Year-7 medical students (M4~6/M7) and nurses.
UNASSIGNED: We received 258 respondents with 52% response rate. All groups, except nurses, chose \"teaching ability\" as the most important domain. This contrasts with research from Western contexts that highlights role modeling, leadership and enthusiasm. The clinical educators and nurses have the same choices of the top five items in the \"personal qualities\" domain, but different choices in \"assessment ability\" and \"curriculum planning\" domains. The best fit rater groups for evaluating clinical educators were educators themselves and PGYs.
UNASSIGNED: There may well be specific suitable domains and populations for evaluating clinical educators\' competence in East Asian culture contexts. Further research in these contexts is required to examine the reach of these findings.

BACKGROUND: Multisource feedback is an evidence-based and validated tool used to provide clinicians, including those in training, feedback on their professional and interpersonal skills. Multisource feedback is mandatory for participants in the Royal Australian College of General Practitioners Practice Experience Program and for some Australian General Practice Training Registrars. Given the recency of the Practice Experience Program, there are currently no benchmarks available for comparison within the program and to other comparable cohorts including doctors in the Australian General Practice Training program. The aim of this study is to evaluate and compare colleague feedback within and across General Practice trainee cohorts.
METHODS: Colleague feedback, from multisource feedback of Practice Experience Program participants and Australian General Practice Training Registrars, collected between January 2018 and April 2020, was compared to identify similarities and differences. Analyses entailed descriptive statistics, between and within groups rater consistency and agreement measures, principal component analysis, t-tests, analysis of variance, and psychometric network analysis.
RESULTS: Colleague ratings of Practice Experience Program participants (overall average 88.58%) were lower than for Registrars (89.08%), although this difference was not significant. \'Communication with patients\' was rated significantly lower for Practice Experience Program participants (2.13%) while this group was rated significantly better for their \'Ability to say no\' (1.78%). Psychometric network analyses showed stronger linkages between items making up the behavioural component (compared to the items of the performance and self-management components, as found by principal component analysis) for Practice Experience Program participants as compared to Registrars. Practice Experience Program participants were stronger in clinical knowledge and skills as well as confidentiality, while Registrars were stronger in communicating with patients, managing their own stress, and in their management and leadership skills.
CONCLUSIONS: The multisource feedback scores of doctors undertaking the Practice Experience Program suggests that, while all mean values are \'very good\' to \'excellent\', there are areas for improvement. The linkages between skills suggests that Practice Experience Program doctors\' skills are somewhat isolated and have yet to fully synthesise. We now have a better understanding of how different groups of General Practitioners in training compare with respect to professional and interpersonal skills. Based on the demonstrated differences, the Practice Experience Program might benefit from the addition of educational activities to target the less developed skills.

OBJECTIVE: Multisource feedback (MSF) is a workplace-based assessment tool that offers 360-degree evaluation of the trainee doctor. Little is known about its receptiveness among stakeholders in Pakistan. This study explores house officers\' perceptions regarding MSF since its implementation in Eye Unit-II, Institute of Ophthalmology, King Edward Medical University/ Mayo Hospital, Lahore.
METHODS: A qualitative case study was conducted from July 2019 to February 2020 in Eye Unit II. A purposive (maximum variation) sample of 12 house surgeons was taken. Two focus group discussions were conducted. Data were transcribed and analyzed thematically.
RESULTS: The study identified the impact of MSF on house surgeons. Most participants reported positive experiences. The feedback they received increased their motivation, management skills and team working. A number of factors affecting the receptiveness of MSF were also identified which mainly included characteristics of raters and emotional response to MSF.
CONCLUSIONS: Multisource Feedback is a useful tool for feedback that impacts the young doctors in many ways. It contributes to increasing their sense of responsibility, management skills and self-directed learning. The improvement in individual abilities and teamwork also helped in improving patient care.

Since the introduction of competency-based frameworks into postgraduate medical curricula, educators have struggled to implement robust assessment tools that document the progression of necessary skills. The global movement towards competency-based medical education demands validated assessment tools. Our objective was to provide validity evidence for the Ottawa CanMEDS Competency Assessment Tool (OCCAT), designed to assess clinical performance in the communicator, professional, and health advocate CanMEDS roles.
We developed the OCCAT, a 29-item questionnaire informed by specialty-specific Entrustable Professional Activities and consultation with stakeholders, including patients. Our sample included nine neonatal-perinatal medicine and maternal fetal medicine fellows rotating through antenatal high-risk clinics at the Ottawa Hospital. Following 70 unique encounters, the OCCAT was completed by patients and learners. Generalizability theory was used to determine overall reliability of scores. Differences in self and patient ratings were assessed using analyses of variance.
Generalizability analysis demonstrated that both questionnaires produced reliable scores (G-coefficient > 0.9). Self-scores were significantly lower than patient scores across all competencies, F(1, 6) = 13.9, p = .007. Variability analysis demonstrated that trainee scores varied across all competencies, suggesting both groups were able to recognize competencies as distinct and discriminate favorable behaviors belonging to each.
Our findings lend support to the movement to integrate self-assessment and patient feedback in formal evaluations for the purpose of enriched learner experiences and improved patient outcomes. We anticipate that the OCCAT will facilitate bridging to competency-based medical education.

BACKGROUND: Undergraduate medical education recognises that patient feedback is potentially valuable for student learning and development as a component of multi-source feedback. However greater exploration of how patient feedback perspectives differ to clinical educators is required for curriculum development and improving student feedback literacy. This study aimed to determine how two sources of feedback, patients and clinical tutors, compare on the same patient-centred, interpersonal criteria.
METHODS: A patient feedback instrument designed for the undergraduate medical education setting was used to compare patients\' feedback with clinical tutors\' feedback following a student-patient consultation in the learning context. Assessments from 222 learning consultations involving 40 medical students were collected. Descriptive statistics for tutors and patients for each question were calculated and correlations between patient and tutor were explored using Spearman\'s rank-order correlation. Mixed effects ordered logistic regression was used to compare each question with an overall rating for tutor and patients in addition to comparing patient with tutor ratings.
RESULTS: Clinical tutor and patient assessments had a weak but significant positive correlation in all areas except questions related to respect and concern. When making judgements compared with overall assessment, patients\' ratings of respect, concern, communication and being understood in the consultation have a greater effect. After eliminating the effect of generally higher ratings by patients compared with tutors using comparative ordered logistic regression, patients rated students relatively less competent in areas of personal interaction.
CONCLUSIONS: This study provides insight about patient feedback, which is required to continue improving the use and acceptability of this multisource feedback to students as a valuable component of their social learning environment. We have revealed the different perspective-specific judgement that patients bring to feedback. This finding contributes to building respect for patient feedback through greater understanding of the elements of consultations for which patients can discriminate performance.

BACKGROUND: In medical settings, multisource feedback (MSF) is a recognised method of formative assessment. It collects feedback on a doctor\'s performance from several perspectives in the form of questionnaires. Yet, no validated MSF questionnaire has been publicly available in German. Thus, we aimed to develop a German MSF questionnaire based on the CanMEDS roles and to investigate the evidence of its validity.
METHODS: We developed a competency-based MSF questionnaire in German, informed by the literature and expert input. Four sources of validity evidence were investigated: (i) Content was examined based on MSF literature, blueprints of competency, and expert-team discussions. (ii) The response process was supported by analysis of a think-aloud study, narrative comments, \"unable to comment\" ratings and evaluation data. (iii) The internal structure was assessed by exploratory factor analysis, and inter-rater reliability by generalisability analysis. Data were collected during two runs of MSF, in which 47 residents were evaluated once (first run) or several times (second and third run) on 81 occasions of MSF. (iv) To investigate consequences, we analysed the residents\' learning goals and the progress as reported via MSF.
RESULTS: Our resulting MSF questionnaire (MSF-RG) consists of 15 items and one global rating, which are each rated on a scale and accompanied by a field for narrative comments and cover a construct of a physician\'s competence. Additionally, there are five open questions for further suggestions. Investigation of validity evidence revealed that: (i) The expert group agreed that the content comprehensively addresses clinical competence; (ii) The response processes indicated that the questions are understood as intended and supported the acceptance and usability; (iii) For the second run, factor analysis showed a one-factor solution, a Cronbach\'s alpha of 0.951 and an inter-rater reliability of 0.797 with 12 raters; (iv) There are indications that residents benefitted, considering their individual learning goals and based on their ratings reported via MSF itself.
CONCLUSIONS: To support residency training with multisource feedback, we developed a German MSF questionnaire (MSF-RG), which is supported by four sources of validity evidence. This MSF questionnaire may be useful to implement MSF in residency training in German-speaking regions.

BACKGROUND: Multisource feedback (MSF) is increasingly being used to assess trainee performance, with different assessor groups fulfilling a crucial role in utility of assessment data. However, in health professions education, research on assessor behaviors in MSF is limited. When assessing trainee performance in work settings, assessors use multidimensional conceptualizations of what constitutes effective performance, also called personal performance theories, to distinguish between various behaviors and sub competencies., This may not only explain assessor variability in Multi Source Feedback, but also result in differing acceptance (and use) of assessment data for developmental purposes. The purpose of this study was to explore performance theories of various assessor groups (residents and nurses) when assessing performance of residents.
METHODS: A constructivist, inductive qualitative research approach and semi-structured interviews following MSF were used to explore performance theories of 14 nurses and 15 residents in the department of internal medicine at Aga Khan University (AKU). Inductive thematic content analysis of interview transcripts was used to identify and compare key dimensions in residents\' and nurses\' performance theories used in evaluation of resident performance.
RESULTS: Seven major themes, reflecting key dimensions of assessors\' performance theories, emerged from the qualitative data, namely; communication skills, patient care, accessibility, teamwork skills, responsibility, medical knowledge and professional attitude. There were considerable overlaps, but also meaningful differences in the performance theories of residents and the nurses, especially with respect to accessibility, teamwork and medical knowledge.
CONCLUSIONS: Residents\' and nurses\' performance theories for assessing resident performance overlap to some extent, yet also show meaningful differences with respect to the performance dimensions they pay attention to or consider most important. In MSF, different assessor groups may therefore hold different performance theories, depending on their role. Our results further our understanding of assessor source effects in MSF. Implications of our findings are related to implementation of MSF, design of rating scales as well as interpretation and use of MSF data for selection and performance improvement.