BACKGROUND: Although history taking is fundamental for diagnosing medical conditions, teaching and providing feedback on the skill can be challenging due to resource constraints. Virtual simulated patients and web-based chatbots have thus emerged as educational tools, with recent advancements in artificial intelligence (AI) such as large language models (LLMs) enhancing their realism and potential to provide feedback.
OBJECTIVE: In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students\' performance in history taking with a simulated patient.
METHODS: We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients\' responses and provide immediate feedback on the comprehensiveness of the students\' history taking. Students\' interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback.
RESULTS: Most of the study\'s participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4\'s role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed \"almost perfect\" agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model\'s assessments were overly specific or diverged from human judgement.
CONCLUSIONS: The GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context.

OBJECTIVE: Autoimmune liver diseases (AILDs) are rare and require precise evaluation, which is often challenging for medical providers. Chatbots are innovative solutions to assist healthcare professionals in clinical management. In our study, ten liver specialists systematically evaluated four chatbots to determine their utility as clinical decision support tools in the field of AILDs.
METHODS: We constructed a 56-question questionnaire focusing on AILD evaluation, diagnosis, and management of Autoimmune Hepatitis (AIH), Primary Biliary Cholangitis (PBC), and Primary Sclerosing Cholangitis (PSC). Four chatbots -ChatGPT 3.5, Claude, Microsoft Copilot, and Google Bard- were presented with the questions in their free tiers in December 2023. Responses underwent critical evaluation by ten liver specialists using a standardized 1 to 10 Likert scale. The analysis included mean scores, the number of highest-rated replies, and the identification of common shortcomings in chatbots performance.
RESULTS: Among the assessed chatbots, specialists rated Claude highest with a mean score of 7.37 (SD = 1.91), followed by ChatGPT (7.17, SD = 1.89), Microsoft Copilot (6.63, SD = 2.10), and Google Bard (6.52, SD = 2.27). Claude also excelled with 27 best-rated replies, outperforming ChatGPT (20), while Microsoft Copilot and Google Bard lagged with only 6 and 9, respectively. Common deficiencies included listing details over specific advice, limited dosing options, inaccuracies for pregnant patients, insufficient recent data, over-reliance on CT and MRI imaging, and inadequate discussion regarding off-label use and fibrates in PBC treatment. Notably, internet access for Microsoft Copilot and Google Bard did not enhance precision compared to pre-trained models.
CONCLUSIONS: Chatbots hold promise in AILD support, but our study underscores key areas for improvement. Refinement is needed in providing specific advice, accuracy, and focused up-to-date information. Addressing these shortcomings is essential for enhancing the utility of chatbots in AILD management, guiding future development, and ensuring their effectiveness as clinical decision-support tools.

暂无摘要。

UNASSIGNED: With the increasing application of large language models like ChatGPT in various industries, its potential in the medical domain, especially in standardized examinations, has become a focal point of research.
UNASSIGNED: The aim of this study is to assess the clinical performance of ChatGPT, focusing on its accuracy and reliability in the Chinese National Medical Licensing Examination (CNMLE).
UNASSIGNED: The CNMLE 2022 question set, consisting of 500 single-answer multiple choices questions, were reclassified into 15 medical subspecialties. Each question was tested 8 to 12 times in Chinese on the OpenAI platform from April 24 to May 15, 2023. Three key factors were considered: the version of GPT-3.5 and 4.0, the prompt\'s designation of system roles tailored to medical subspecialties, and repetition for coherence. A passing accuracy threshold was established as 60%. The χ2 tests and κ values were employed to evaluate the model\'s accuracy and consistency.
UNASSIGNED: GPT-4.0 achieved a passing accuracy of 72.7%, which was significantly higher than that of GPT-3.5 (54%; P<.001). The variability rate of repeated responses from GPT-4.0 was lower than that of GPT-3.5 (9% vs 19.5%; P<.001). However, both models showed relatively good response coherence, with κ values of 0.778 and 0.610, respectively. System roles numerically increased accuracy for both GPT-4.0 (0.3%-3.7%) and GPT-3.5 (1.3%-4.5%), and reduced variability by 1.7% and 1.8%, respectively (P>.05). In subgroup analysis, ChatGPT achieved comparable accuracy among different question types (P>.05). GPT-4.0 surpassed the accuracy threshold in 14 of 15 subspecialties, while GPT-3.5 did so in 7 of 15 on the first response.
UNASSIGNED: GPT-4.0 passed the CNMLE and outperformed GPT-3.5 in key areas such as accuracy, consistency, and medical subspecialty expertise. Adding a system role insignificantly enhanced the model\'s reliability and answer coherence. GPT-4.0 showed promising potential in medical education and clinical practice, meriting further study.

Large Language Models (LLMs) like ChatGPT 4 (OpenAI), Claude 2 (Anthropic), and Llama 2 (Meta AI) have emerged as novel technologies to integrate artificial intelligence (AI) into everyday work. LLMs in particular, and AI in general, carry infinite potential to streamline clinical workflows, outsource resource-intensive tasks, and disburden the healthcare system. While a plethora of trials is elucidating the untapped capabilities of this technology, the sheer pace of scientific progress also takes its toll. Legal guidelines hold a key role in regulating upcoming technologies, safeguarding patients, and determining individual and institutional liabilities. To date, there is a paucity of research work delineating the legal regulations of Language Models and AI for clinical scenarios in plastic and reconstructive surgery. This knowledge gap poses the risk of lawsuits and penalties against plastic surgeons. Thus, we aim to provide the first overview of legal guidelines and pitfalls of LLMs and AI for plastic surgeons. Our analysis encompasses models like ChatGPT, Claude 2, and Llama 2, among others, regardless of their closed or open-source nature. Ultimately, this line of research may help clarify the legal responsibilities of plastic surgeons and seamlessly integrate such cutting-edge technologies into the field of PRS.

BACKGROUND: Chronic hepatitis B (CHB) imposes substantial economic and social burdens globally. The management of CHB involves intricate monitoring and adherence challenges, particularly in regions like China, where a high prevalence of CHB intersects with health care resource limitations. This study explores the potential of ChatGPT-3.5, an emerging artificial intelligence (AI) assistant, to address these complexities. With notable capabilities in medical education and practice, ChatGPT-3.5\'s role is examined in managing CHB, particularly in regions with distinct health care landscapes.
OBJECTIVE: This study aimed to uncover insights into ChatGPT-3.5\'s potential and limitations in delivering personalized medical consultation assistance for CHB patients across diverse linguistic contexts.
METHODS: Questions sourced from published guidelines, online CHB communities, and search engines in English and Chinese were refined, translated, and compiled into 96 inquiries. Subsequently, these questions were presented to both ChatGPT-3.5 and ChatGPT-4.0 in independent dialogues. The responses were then evaluated by senior physicians, focusing on informativeness, emotional management, consistency across repeated inquiries, and cautionary statements regarding medical advice. Additionally, a true-or-false questionnaire was employed to further discern the variance in information accuracy for closed questions between ChatGPT-3.5 and ChatGPT-4.0.
RESULTS: Over half of the responses (228/370, 61.6%) from ChatGPT-3.5 were considered comprehensive. In contrast, ChatGPT-4.0 exhibited a higher percentage at 74.5% (172/222; P<.001). Notably, superior performance was evident in English, particularly in terms of informativeness and consistency across repeated queries. However, deficiencies were identified in emotional management guidance, with only 3.2% (6/186) in ChatGPT-3.5 and 8.1% (15/154) in ChatGPT-4.0 (P=.04). ChatGPT-3.5 included a disclaimer in 10.8% (24/222) of responses, while ChatGPT-4.0 included a disclaimer in 13.1% (29/222) of responses (P=.46). When responding to true-or-false questions, ChatGPT-4.0 achieved an accuracy rate of 93.3% (168/180), significantly surpassing ChatGPT-3.5\'s accuracy rate of 65.0% (117/180) (P<.001).
CONCLUSIONS: In this study, ChatGPT demonstrated basic capabilities as a medical consultation assistant for CHB management. The choice of working language for ChatGPT-3.5 was considered a potential factor influencing its performance, particularly in the use of terminology and colloquial language, and this potentially affects its applicability within specific target populations. However, as an updated model, ChatGPT-4.0 exhibits improved information processing capabilities, overcoming the language impact on information accuracy. This suggests that the implications of model advancement on applications need to be considered when selecting large language models as medical consultation assistants. Given that both models performed inadequately in emotional guidance management, this study highlights the importance of providing specific language training and emotional management strategies when deploying ChatGPT for medical purposes. Furthermore, the tendency of these models to use disclaimers in conversations should be further investigated to understand the impact on patients\' experiences in practical applications.

Although chatbots are used a lot for customer relationship management (CRM), there needs to be more data security and privacy control strategies in chatbots, which has become a security concern for financial services institutions. Chatbots gain access to large amounts of vital company information and clients\' personal information, which makes them a target of security attacks. The loss of data stored in chatbots can cause major harm to companies and customers. In this study, STRIDE (viz. Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege) modelling was applied to identify the data security vulnerabilities and threats that pertain to chatbots used in the insurance industry. To do this, we conducted a case study of a South African insurance organisation. The adopted methodology involved data collection from stakeholders in the insurance organisation to identify chatbot use cases and understand chatbot operations. After that, we conducted a STRIDE-based analysis of the chatbot use cases to elicit security threats and vulnerabilities in the insurance chatbots in the organisation. The results reveal that security vulnerabilities associated with Spoofing, Denial of Service, and Elevation of privilege are more relevant to insurance chatbots. The most security threats stem from Tampering, Elevation of privilege, and Spoofing. The study extends the discussion on chatbot security. It fosters an understanding of security threats and vulnerabilities that pertain to insurance chatbots, which is beneficial for security researchers and practitioners working on the security of chatbots and the insurance industry.

The advent of large language models (LLMs) such as ChatGPT has potential implications for psychological therapies such as cognitive behavioral therapy (CBT). We systematically investigated whether LLMs could recognize an unhelpful thought, examine its validity, and reframe it to a more helpful one. LLMs currently have the potential to offer reasonable suggestions for the identification and reframing of unhelpful thoughts but should not be relied on to lead CBT delivery.

BACKGROUND: To evaluate the accuracy, reliability, quality, and readability of responses generated by ChatGPT-3.5, ChatGPT-4, Gemini, and Copilot in relation to orthodontic clear aligners.
METHODS: Frequently asked questions by patients/laypersons about clear aligners on websites were identified using the Google search tool and these questions were posed to ChatGPT-3.5, ChatGPT-4, Gemini, and Copilot AI models. Responses were assessed using a five-point Likert scale for accuracy, the modified DISCERN scale for reliability, the Global Quality Scale (GQS) for quality, and the Flesch Reading Ease Score (FRES) for readability.
RESULTS: ChatGPT-4 responses had the highest mean Likert score (4.5 ± 0.61), followed by Copilot (4.35 ± 0.81), ChatGPT-3.5 (4.15 ± 0.75) and Gemini (4.1 ± 0.72). The difference between the Likert scores of the chatbot models was not statistically significant (p > 0.05). Copilot had a significantly higher modified DISCERN and GQS score compared to both Gemini, ChatGPT-4 and ChatGPT-3.5 (p < 0.05). Gemini\'s modified DISCERN and GQS score was statistically higher than ChatGPT-3.5 (p < 0.05). Gemini also had a significantly higher FRES compared to both ChatGPT-4, Copilot and ChatGPT-3.5 (p < 0.05). The mean FRES was 38.39 ± 11.56 for ChatGPT-3.5, 43.88 ± 10.13 for ChatGPT-4 and 41.72 ± 10.74 for Copilot, indicating that the responses were difficult to read according to the reading level. The mean FRES for Gemini is 54.12 ± 10.27, indicating that Gemini\'s responses are more readable than other chatbots.
CONCLUSIONS: All chatbot models provided generally accurate, moderate reliable and moderate to good quality answers to questions about the clear aligners. Furthermore, the readability of the responses was difficult. ChatGPT, Gemini and Copilot have significant potential as patient information tools in orthodontics, however, to be fully effective they need to be supplemented with more evidence-based information and improved readability.

Orthopedic clinics are becoming strained with clinical volume outpacing resources and personnel. Patient engagement platforms can help bridge the communication and engagement gaps between patients and their healthcare teams as total hip and knee arthroplasty transitions to the outpatient setting. These platforms provide a digital infrastructure that allows patients to participate in their healthcare journey while alleviating the burdens on clinic staff. Multiple forms of patient engagement platforms exist but typically fall into one of 3 groups: patient portals, mobile health applications, and chatbots. They all play an important role in enhancing postoperative rehabilitation, patient engagement, and patient care overall. This article explores the spectrum of available patient engagement platforms and examines their advantages, limitations, and documented benefits on clinical outcomes.