Gynecologic cancer requires personalized care to improve outcomes. Large language models (LLMs) hold the potential to provide intelligent question-answering with reliable information about medical queries in clear and plain English, which can be understood by both healthcare providers and patients. We aimed to evaluate two freely available LLMs (ChatGPT and Google\'s Bard) in answering questions regarding the management of gynecologic cancer. The LLMs\' performances were evaluated by developing a set questions that addressed common gynecologic oncologic findings from a patient\'s perspective and more complex questions to elicit recommendations from a clinician\'s perspective. Each question was presented to the LLM interface, and the responses generated by the artificial intelligence (AI) model were recorded. The responses were assessed based on the adherence to the National Comprehensive Cancer Network and European Society of Gynecological Oncology guidelines. This evaluation aimed to determine the accuracy and appropriateness of the information provided by LLMs. We showed that the models provided largely appropriate responses to questions regarding common cervical cancer screening tests and BRCA-related questions. Less useful answers were received to complex and controversial gynecologic oncology cases, as assessed by reviewing the common guidelines. ChatGPT and Bard lacked knowledge of regional guideline variations, However, it provided practical and multifaceted advice to patients and caregivers regarding the next steps of management and follow up. We conclude that LLMs may have a role as an adjunct informational tool to improve outcomes.

BACKGROUND: Patient education plays a crucial role in improving the quality of life for patients with heart failure. As artificial intelligence continues to advance, new chatbots are emerging as valuable tools across various aspects of life. One prominent example is ChatGPT, a widely used chatbot among the public. Our study aims to evaluate the readability of ChatGPT answers for common patients\' questions about heart failure.
METHODS: We performed a comparative analysis between ChatGPT responses and existing heart failure educational materials from top US cardiology institutes. Validated readability calculators were employed to assess and compare the reading difficulty and grade level of the materials. Furthermore, blind assessment using The Patient Education Materials Assessment Tool (PEMAT) was done by four advanced heart failure attendings to evaluate the readability and actionability of each resource.
RESULTS: Our study revealed that responses generated by ChatGPT were longer and more challenging to read compared to other materials. Additionally, these responses were written at a higher educational level (undergraduate and 9-10th grade), similar to those from the Heart Failure Society of America. Despite achieving a competitive PEMAT readability score (75%), surpassing the American Heart Association score (68%), ChatGPT\'s actionability score was the lowest (66.7%) among all materials included in our study.
CONCLUSIONS: Despite its current limitations, artificial intelligence chatbots has the potential to revolutionize the field of patient education especially given theirs ongoing improvements. However, further research is necessary to ensure the integrity and reliability of these chatbots before endorsing them as reliable resources for patient education.

BACKGROUND: Strabismus is a common eye condition affecting both children and adults. Effective patient education is crucial for informed decision-making, but traditional methods often lack accessibility and engagement. Chatbots powered by AI have emerged as a promising solution.
OBJECTIVE: This study aims to evaluate and compare the performance of three chatbots (ChatGPT, Bard, and Copilot) and a reliable website (AAPOS) in answering real patient questions about strabismus.
METHODS: Three chatbots (ChatGPT, Bard, and Copilot) were compared to a reliable website (AAPOS) using real patient questions. Metrics included accuracy (SOLO taxonomy), understandability/actionability (PEMAT), and readability (Flesch-Kincaid). We also performed a sentiment analysis to capture the emotional tone and impact of the responses.
RESULTS: The AAPOS achieved the highest mean SOLO score (4.14 ± 0.47), followed by Bard, Copilot, and ChatGPT. Bard scored highest on both PEMAT-U (74.8 ± 13.3) and PEMAT-A (66.2 ± 13.6) measures. Flesch-Kincaid Ease Scores revealed the AAPOS as the easiest to read (mean score: 55.8 ± 14.11), closely followed by Copilot. ChatGPT, and Bard had lower scores on readability. The sentiment analysis revealed exciting differences.
CONCLUSIONS: Chatbots, particularly Bard and Copilot, show promise in patient education for strabismus with strengths in understandability and actionability. However, the AAPOS website outperformed in accuracy and readability.

BACKGROUND: With the rapid evolution of artificial intelligence (AI), particularly large language models (LLMs) such as ChatGPT-4 (OpenAI), there is an increasing interest in their potential to assist in scholarly tasks, including conducting literature reviews. However, the efficacy of AI-generated reviews compared with traditional human-led approaches remains underexplored.
OBJECTIVE: This study aims to compare the quality of literature reviews conducted by the ChatGPT-4 model with those conducted by human researchers, focusing on the relational dynamics between physicians and patients.
METHODS: We included 2 literature reviews in the study on the same topic, namely, exploring factors affecting relational dynamics between physicians and patients in medicolegal contexts. One review used GPT-4, last updated in September 2021, and the other was conducted by human researchers. The human review involved a comprehensive literature search using medical subject headings and keywords in Ovid MEDLINE, followed by a thematic analysis of the literature to synthesize information from selected articles. The AI-generated review used a new prompt engineering approach, using iterative and sequential prompts to generate results. Comparative analysis was based on qualitative measures such as accuracy, response time, consistency, breadth and depth of knowledge, contextual understanding, and transparency.
RESULTS: GPT-4 produced an extensive list of relational factors rapidly. The AI model demonstrated an impressive breadth of knowledge but exhibited limitations in in-depth and contextual understanding, occasionally producing irrelevant or incorrect information. In comparison, human researchers provided a more nuanced and contextually relevant review. The comparative analysis assessed the reviews based on criteria including accuracy, response time, consistency, breadth and depth of knowledge, contextual understanding, and transparency. While GPT-4 showed advantages in response time and breadth of knowledge, human-led reviews excelled in accuracy, depth of knowledge, and contextual understanding.
CONCLUSIONS: The study suggests that GPT-4, with structured prompt engineering, can be a valuable tool for conducting preliminary literature reviews by providing a broad overview of topics quickly. However, its limitations necessitate careful expert evaluation and refinement, making it an assistant rather than a substitute for human expertise in comprehensive literature reviews. Moreover, this research highlights the potential and limitations of using AI tools like GPT-4 in academic research, particularly in the fields of health services and medical research. It underscores the necessity of combining AI\'s rapid information retrieval capabilities with human expertise for more accurate and contextually rich scholarly outputs.

UNASSIGNED: NLP models like ChatGPT promise to revolutionize text-based content delivery, particularly in medicine. Yet, doubts remain about ChatGPT\'s ability to reliably support evaluations of cognitive performance, warranting further investigation into its accuracy and comprehensiveness in this area.
UNASSIGNED: A cohort of 60 cognitively normal individuals and 30 stroke survivors underwent a comprehensive evaluation, covering memory, numerical processing, verbal fluency, and abstract thinking. Healthcare professionals and NLP models GPT-3.5 and GPT-4 conducted evaluations following established standards. Scores were compared, and efforts were made to refine scoring protocols and interaction methods to enhance ChatGPT\'s potential in these evaluations.
UNASSIGNED: Within the cohort of healthy participants, the utilization of GPT-3.5 revealed significant disparities in memory evaluation compared to both physician-led assessments and those conducted utilizing GPT-4 (P < 0.001). Furthermore, within the domain of memory evaluation, GPT-3.5 exhibited discrepancies in 8 out of 21 specific measures when compared to assessments conducted by physicians (P < 0.05). Additionally, GPT-3.5 demonstrated statistically significant deviations from physician assessments in speech evaluation (P = 0.009). Among participants with a history of stroke, GPT-3.5 exhibited differences solely in verbal assessment compared to physician-led evaluations (P = 0.002). Notably, through the implementation of optimized scoring methodologies and refinement of interaction protocols, partial mitigation of these disparities was achieved.
UNASSIGNED: ChatGPT can produce evaluation outcomes comparable to traditional methods. Despite differences from physician evaluations, refinement of scoring algorithms and interaction protocols has improved alignment. ChatGPT performs well even in populations with specific conditions like stroke, suggesting its versatility. GPT-4 yields results closer to physician ratings, indicating potential for further enhancement. These findings highlight ChatGPT\'s importance as a supplementary tool, offering new avenues for information gathering in medical fields and guiding its ongoing development and application.

BACKGROUND: Large Language Models (LLM; e.g., ChatGPT) may be used to assist clinicians and form the basis of future clinical decision support (CDS) for colon cancer. The objectives of this study were to (1) evaluate the response accuracy of two LLM-powered interfaces in identifying guideline-based care in simulated clinical scenarios and (2) define response variation between and within LLMs.
METHODS: Clinical scenarios with \"next steps in management\" queries were developed based on National Comprehensive Cancer Network guidelines. Prompts were entered into OpenAI ChatGPT and Microsoft Copilot in independent sessions, yielding four responses per scenario. Responses were compared to clinician-developed responses and assessed for accuracy, consistency, and verbosity.
RESULTS: Across 108 responses to 27 prompts, both platforms yielded completely correct responses to 36% of scenarios (n = 39). For ChatGPT, 39% (n = 21) were missing information and 24% (n = 14) contained inaccurate/misleading information. Copilot performed similarly, with 37% (n = 20) having missing information and 28% (n = 15) containing inaccurate/misleading information (p = 0.96). Clinician responses were significantly shorter (34 ± 15.5 words) than both ChatGPT (251 ± 86 words) and Copilot (271 ± 67 words; both p < 0.01).
CONCLUSIONS: Publicly available LLM applications often provide verbose responses with vague or inaccurate information regarding colon cancer management. Significant optimization is required before use in formal CDS.

BACKGROUND: Although history taking is fundamental for diagnosing medical conditions, teaching and providing feedback on the skill can be challenging due to resource constraints. Virtual simulated patients and web-based chatbots have thus emerged as educational tools, with recent advancements in artificial intelligence (AI) such as large language models (LLMs) enhancing their realism and potential to provide feedback.
OBJECTIVE: In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students\' performance in history taking with a simulated patient.
METHODS: We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients\' responses and provide immediate feedback on the comprehensiveness of the students\' history taking. Students\' interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback.
RESULTS: Most of the study\'s participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4\'s role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed \"almost perfect\" agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model\'s assessments were overly specific or diverged from human judgement.
CONCLUSIONS: The GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context.

Purpose: This study aims to assess the effectiveness of ChatGPT in remote learning among medical students. Methods: This cross-sectional survey study recruited 386 medical students from three public universities in Saudi Arabia. Participants completed an online questionnaire designed to assess perceptions of ChatGPT\'s effectiveness in remote learning. The questionnaire included Likert scale questions to evaluate various aspects of ChatGPT\'s support in remote learning, such as personalized learning, language and communication skills, and interactive quizzing. Data were analyzed using SPSS, employing descriptive statistics, independent samples t-tests, one-way ANOVA, and Cronbach\'s alpha to evaluate reliability. Results: Participants mostly used ChatGPT on a weekly (43.2%) or daily (48.7%) basis, primarily on personal computers (62.5%). Mean scores for ChatGPT\'s support in remote learning were high for personalized learning (4.35), language and communication skills (4.23), and interactive quizzing and assessments (4.01). Statistically significant differences were found based on gender for interactive quizzing (p = .0177) and continuity of education (p = .0122). Conclusion: Despite certain challenges and variations in perceptions based on gender and education level, the overwhelmingly positive attitudes toward ChatGPT highlight its potential as a valuable tool in medical education.

OBJECTIVE: This study aims to assess the effectiveness of ChatGPT in critical thinking skills among medical students.
METHODS: This cross-sectional survey study recruited 392 medical students from three public universities in Saudi Arabia. Participants completed an online questionnaire assessing perceptions of ChatGPT\'s impact on critical thinking skills. Data were analyzed using SPSS, employing descriptive statistics, t-tests, analysis of variance, and Cronbach\'s alpha to evaluate reliability.
RESULTS: Significant gender-based differences were found in perceptions of ChatGPT\'s efficacy, particularly in generating diverse perspectives (P = 0.0407*) and encouraging questioning (P = 0.0277*). Reflective practice perceptions varied significantly by age (P = 0.0302*), while academic backgrounds yielded significant differences across all factors assessed (P < 0.0001*). Overall, 92.6% believed integrating ChatGPT would benefit critical thinking skills. Most participants (N = 174) strongly agreed that ChatGPT improved critical thinking.
CONCLUSIONS: Integrating ChatGPT into medical education could offer valuable opportunities for fostering critical thinking abilities, albeit with the need for addressing associated challenges and ensuring inclusivity.

UNASSIGNED: To assess the quality and alignment of ChatGPT\'s cancer treatment recommendations (RECs) with National Comprehensive Cancer Network (NCCN) guidelines and expert opinions.
UNASSIGNED: Three urologists performed quantitative and qualitative assessments in October 2023 analyzing responses from ChatGPT-4 and ChatGPT-3.5 to 108 prostate, kidney, and bladder cancer prompts using two zero-shot prompt templates. Performance evaluation involved calculating five ratios: expert-approved/expert-disagreed and NCCN-aligned RECs against total ChatGPT RECs plus coverage and adherence rates to NCCN. Experts rated the response\'s quality on a 1-5 scale considering correctness, comprehensiveness, specificity, and appropriateness.
UNASSIGNED: ChatGPT-4 outperformed ChatGPT-3.5 in prostate cancer inquiries, with an average word count of 317.3 versus 124.4 (p < 0.001) and 6.1 versus 3.9 RECs (p < 0.001). Its rater-approved REC ratio (96.1% vs. 89.4%) and alignment with NCCN guidelines (76.8% vs. 49.1%, p = 0.001) were superior and scored significantly better on all quality dimensions. Across 108 prompts covering three cancers, ChatGPT-4 produced an average of 6.0 RECs per case, with an 88.5% approval rate from raters, 86.7% NCCN concordance, and only a 9.5% disagreement rate. It achieved high marks in correctness (4.5), comprehensiveness (4.4), specificity (4.0), and appropriateness (4.4). Subgroup analyses across cancer types, disease statuses, and different prompt templates were reported.
UNASSIGNED: ChatGPT-4 demonstrated significant improvement in providing accurate and detailed treatment recommendations for urological cancers in line with clinical guidelines and expert opinion. However, it is vital to recognize that AI tools are not without flaws and should be utilized with caution. ChatGPT could supplement, but not replace, personalized advice from healthcare professionals.