UNASSIGNED: Teaching medical students the skills required to acquire, interpret, apply, and communicate clinical information is an integral part of medical education. A crucial aspect of this process involves providing students with feedback regarding the quality of their free-text clinical notes.
UNASSIGNED: The goal of this study was to assess the ability of ChatGPT 3.5, a large language model, to score medical students\' free-text history and physical notes.
UNASSIGNED: This is a single-institution, retrospective study. Standardized patients learned a prespecified clinical case and, acting as the patient, interacted with medical students. Each student wrote a free-text history and physical note of their interaction. The students\' notes were scored independently by the standardized patients and ChatGPT using a prespecified scoring rubric that consisted of 85 case elements. The measure of accuracy was percent correct.
UNASSIGNED: The study population consisted of 168 first-year medical students. There was a total of 14,280 scores. The ChatGPT incorrect scoring rate was 1.0%, and the standardized patient incorrect scoring rate was 7.2%. The ChatGPT error rate was 86%, lower than the standardized patient error rate. The ChatGPT mean incorrect scoring rate of 12 (SD 11) was significantly lower than the standardized patient mean incorrect scoring rate of 85 (SD 74; P=.002).
UNASSIGNED: ChatGPT demonstrated a significantly lower error rate compared to standardized patients. This is the first study to assess the ability of a generative pretrained transformer (GPT) program to score medical students\' standardized patient-based free-text clinical notes. It is expected that, in the near future, large language models will provide real-time feedback to practicing physicians regarding their free-text notes. GPT artificial intelligence programs represent an important advance in medical education and medical practice.

UNASSIGNED: ANCA-associated vasculitis (AAV) is a rare but serious disease. Traditional case-identification methods using claims data can be time-intensive and may miss important subgroups. We hypothesized that a deep learning model analyzing electronic health records (EHR) can more accurately identify AAV cases.
UNASSIGNED: We examined the Mass General Brigham (MGB) repository of clinical documentation from 12/1/1979 to 5/11/2021, using expert-curated keywords and ICD codes to identify a large cohort of potential AAV cases. Three labeled datasets (I, II, III) were created, each containing note sections. We trained and evaluated a range of machine learning and deep learning algorithms for note-level classification, using metrics like positive predictive value (PPV), sensitivity, F-score, area under the receiver operating characteristic curve (AUROC), and area under the precision and recall curve (AUPRC). The deep learning model was further evaluated for its ability to classify AAV cases at the patient-level, compared with rule-based algorithms in 2,000 randomly chosen samples.
UNASSIGNED: Datasets I, II, and III comprised 6,000, 3,008, and 7,500 note sections, respectively. Deep learning achieved the highest AUROC in all three datasets, with scores of 0.983, 0.991, and 0.991. The deep learning approach also had among the highest PPVs across the three datasets (0.941, 0.954, and 0.800, respectively). In a test cohort of 2,000 cases, the deep learning model achieved a PPV of 0.262 and an estimated sensitivity of 0.975. Compared to the best rule-based algorithm, the deep learning model identified six additional AAV cases, representing 13% of the total.
UNASSIGNED: The deep learning model effectively classifies clinical note sections for AAV diagnosis. Its application to EHR notes can potentially uncover additional cases missed by traditional rule-based methods.

UNASSIGNED: In many countries, health care professionals are legally obliged to share information from electronic health records with patients. However, concerns have been raised regarding the sharing of notes with adolescents in mental health care, and health care professionals have called for recommendations to guide this practice.
UNASSIGNED: The aim was to reach a consensus among authors of scientific papers on recommendations for health care professionals\' digital sharing of notes with adolescents in mental health care and to investigate whether staff at child and adolescent specialist mental health care clinics agreed with the recommendations.
UNASSIGNED: A Delphi study was conducted with authors of scientific papers to reach a consensus on recommendations. The process of making the recommendations involved three steps. First, scientific papers meeting the eligibility criteria were identified through a PubMed search where the references were screened. Second, the results from the included papers were coded and transformed into recommendations in an iterative process. Third, the authors of the included papers were asked to provide feedback and consider their agreement with each of the suggested recommendations in two rounds. After the Delphi process, a cross-sectional study was conducted among staff at specialist child and adolescent mental health care clinics to assess whether they agreed with the recommendations that reached a consensus.
UNASSIGNED: Of the 84 invited authors, 27 responded. A consensus was reached on 17 recommendations on areas related to digital sharing of notes with adolescents in mental health care. The recommendations considered how to introduce digital access to notes, write notes, and support health care professionals, and when to withhold notes. Of the 41 staff members at child and adolescent specialist mental health care clinics, 60% or more agreed with the 17 recommendations. No consensus was reached regarding the age at which adolescents should receive digital access to their notes and the timing of digitally sharing notes with parents.
UNASSIGNED: A total of 17 recommendations related to key aspects of health care professionals\' digital sharing of notes with adolescents in mental health care achieved consensus. Health care professionals can use these recommendations to guide their practice of sharing notes with adolescents in mental health care. However, the effects and experiences of following these recommendations should be tested in clinical practice.

OBJECTIVE: In a growing list of countries, patients are granted access to their clinical notes (\"open notes\") as part of their online record access. Especially in the field of mental health, open notes remain controversial with some clinicians perceiving open notes as a tool for improving therapeutic outcomes by increasing patient involvement, while others fear that patients might experience psychological distress and perceived stigmatization, particularly when reading clinicians\' notes. More research is needed to optimize the benefits and mitigate the risks.
METHODS: Using a qualitative research design, we conducted semi-structured interviews with psychiatrists practicing in Germany, to explore what conditions they believe need to be in place to ensure successful implementation of open notes in psychiatric practice as well as expected subsequent changes to their workload and treatment outcomes. Data were analyzed using thematic analysis.
RESULTS: We interviewed 18 psychiatrists; interviewees believed four key conditions needed to be in place prior to implementation of open notes including careful consideration of (1) diagnoses and symptom severity, (2) the availability of additional time for writing clinical notes and discussing them with patients, (3) available resources and system compatibility, and (4) legal and data protection aspects. As a result of introducing open notes, interviewees expected changes in documentation, treatment processes, and doctor-physician interaction. While open notes were expected to improve transparency and trust, participants anticipated negative unintended consequences including the risk of deteriorating therapeutic relationships due to note access-related misunderstandings and conflicts.
CONCLUSIONS: Psychiatrists practiced in Germany where open notes have not yet been established as part of the healthcare data infrastructure. Interviewees were supportive of open notes but had some reservations. They found open notes to be generally beneficial but anticipated effects to vary depending on patient characteristics. Clear guidelines for managing access, time constraints, usability, and privacy are crucial. Open notes were perceived to increase transparency and patient involvement but were also believed to raise issues of stigmatization and conflicts.

BACKGROUND: Extracting research of domain criteria (RDoC) from high-risk populations like those with post-traumatic stress disorder (PTSD) is crucial for positive mental health improvements and policy enhancements. The intricacies of collecting, integrating, and effectively leveraging clinical notes for this purpose introduce complexities.
METHODS: In our study, we created a natural language processing (NLP) workflow to analyze electronic medical record (EMR) data and identify and extract research of domain criteria using a pre-trained transformer-based natural language model, all-mpnet-base-v2. We subsequently built dictionaries from 100,000 clinical notes and analyzed 5.67 million clinical notes from 38,807 PTSD patients from the University of Pittsburgh Medical Center. Subsequently, we showcased the significance of our approach by extracting and visualizing RDoC information in two use cases: (i) across multiple patient populations and (ii) throughout various disease trajectories.
RESULTS: The sentence transformer model demonstrated high F1 macro scores across all RDoC domains, achieving the highest performance with a cosine similarity threshold value of 0.3. This ensured an F1 score of at least 80% across all RDoC domains. The study revealed consistent reductions in all six RDoC domains among PTSD patients after psychotherapy. We found that 60.6% of PTSD women have at least one abnormal instance of the six RDoC domains as compared to PTSD men (51.3%), with 45.1% of PTSD women with higher levels of sensorimotor disturbances compared to men (41.3%). We also found that 57.3% of PTSD patients have at least one abnormal instance of the six RDoC domains based on our records. Also, veterans had the higher abnormalities of negative and positive valence systems (60% and 51.9% of veterans respectively) compared to non-veterans (59.1% and 49.2% respectively). The domains following first diagnoses of PTSD were associated with heightened cue reactivity to trauma, suicide, alcohol, and substance consumption.
CONCLUSIONS: The findings provide initial insights into RDoC functioning in different populations and disease trajectories. Natural language processing proves valuable for capturing real-time, context dependent RDoC instances from extensive clinical notes.

BACKGROUND: Real-time surveillance of emerging infectious diseases necessitates a dynamically evolving, computable case definition, which frequently incorporates symptom-related criteria. For symptom detection, both population health monitoring platforms and research initiatives primarily depend on structured data extracted from electronic health records.
OBJECTIVE: This study sought to validate and test an artificial intelligence (AI)-based natural language processing (NLP) pipeline for detecting COVID-19 symptoms from physician notes in pediatric patients. We specifically study patients presenting to the emergency department (ED) who can be sentinel cases in an outbreak.
METHODS: Subjects in this retrospective cohort study are patients who are 21 years of age and younger, who presented to a pediatric ED at a large academic children\'s hospital between March 1, 2020, and May 31, 2022. The ED notes for all patients were processed with an NLP pipeline tuned to detect the mention of 11 COVID-19 symptoms based on Centers for Disease Control and Prevention (CDC) criteria. For a gold standard, 3 subject matter experts labeled 226 ED notes and had strong agreement (F1-score=0.986; positive predictive value [PPV]=0.972; and sensitivity=1.0). F1-score, PPV, and sensitivity were used to compare the performance of both NLP and the International Classification of Diseases, 10th Revision (ICD-10) coding to the gold standard chart review. As a formative use case, variations in symptom patterns were measured across SARS-CoV-2 variant eras.
RESULTS: There were 85,678 ED encounters during the study period, including 4% (n=3420) with patients with COVID-19. NLP was more accurate at identifying encounters with patients that had any of the COVID-19 symptoms (F1-score=0.796) than ICD-10 codes (F1-score =0.451). NLP accuracy was higher for positive symptoms (sensitivity=0.930) than ICD-10 (sensitivity=0.300). However, ICD-10 accuracy was higher for negative symptoms (specificity=0.994) than NLP (specificity=0.917). Congestion or runny nose showed the highest accuracy difference (NLP: F1-score=0.828 and ICD-10: F1-score=0.042). For encounters with patients with COVID-19, prevalence estimates of each NLP symptom differed across variant eras. Patients with COVID-19 were more likely to have each NLP symptom detected than patients without this disease. Effect sizes (odds ratios) varied across pandemic eras.
CONCLUSIONS: This study establishes the value of AI-based NLP as a highly effective tool for real-time COVID-19 symptom detection in pediatric patients, outperforming traditional ICD-10 methods. It also reveals the evolving nature of symptom prevalence across different virus variants, underscoring the need for dynamic, technology-driven approaches in infectious disease surveillance.

UNASSIGNED: Extracting research of domain criteria (RDoC) from high-risk populations like those with post-traumatic stress disorder (PTSD) is crucial for positive mental health improvements and policy enhancements. The intricacies of collecting, integrating, and effectively leveraging clinical notes for this purpose introduce complexities.
UNASSIGNED: In our study, we created an NLP workflow to analyze electronic medical record (EMR) data, and identify and extract research of domain criteria using a pre-trained transformer-based natural language model, allmpnet-base-v2. We subsequently built dictionaries from 100,000 clinical notes and analyzed 5.67 million clinical notes from 38,807 PTSD patients from the University of Pittsburgh Medical Center. Subsequently, we showcased the significance of our approach by extracting and visualizing RDoC information in two use cases: (i) across multiple patient populations and (ii) throughout various disease trajectories.
UNASSIGNED: The sentence transformer model demonstrated superior F1 macro scores across all RDoC domains, achieving the highest performance with a cosine similarity threshold value of 0.3. This ensured an F1 score of at least 80% across all RDoC domains. The study revealed consistent reductions in all six RDoC domains among PTSD patients after psychotherapy. Women had the highest abnormalities of sensorimotor systems, while veterans had the highest abnormalities of negative and positive valence systems. The domains following first diagnoses of PTSD were associated with heightened cue reactivity to trauma, suicide, alcohol, and substance consumption.
UNASSIGNED: The findings provide initial insights into RDoC functioning in different populations and disease trajectories. Natural language processing proves valuable for capturing real-time, context dependent RDoC instances from extensive clinical notes.

BACKGROUND: Over the past few years, online record access (ORA) has been established through secure patient portals in various countries, allowing patients to access their health data, including clinical notes (\"open notes\"). Previous research indicates that ORA in mental health, particularly among patients with severe mental illness (SMI), has been rarely offered. Little is known about the expectations and motivations of patients with SMI when reading what their clinicians share via ORA.
OBJECTIVE: The aim of this study is to explore the reasons why patients with SMI consider or reject ORA and whether sociodemographic characteristics may influence patient decisions.
METHODS: ORA was offered to randomly selected patients at 3 university outpatient clinics in Brandenburg, Germany, which exclusively treat patients with SMI. Within the framework of a mixed methods evaluation, qualitative interviews were conducted with patients who chose to participate in ORA and those who declined, aiming to explore the underlying reasons for their decisions. The interviews were transcribed and analyzed using thematic analysis. Sociodemographic characteristics of patients were examined using descriptive statistics to identify predictors of acceptance or rejection of ORA.
RESULTS: Out of 103 included patients, 58% (n=60) wished to read their clinical notes. The reasons varied, ranging from a desire to engage more actively in their treatment to critically monitoring it and using the accessible data for third-party purposes. Conversely, 42% (n=43) chose not to use ORA, voicing concerns about possibly harming the trustful relationship with their clinicians as well as potential personal distress or uncertainty arising from reading the notes. Practical barriers such as a lack of digital literacy or suspected difficult-to-understand medical language were also named as contributing factors. Correlation analysis revealed that the majority of patients with depressive disorder desired to read the clinical notes (P<.001), while individuals with psychotic disorders showed a higher tendency to decline ORA (P<.05). No significant group differences were observed for other patient groups or characteristics.
CONCLUSIONS: The adoption of ORA is influenced by a wide range of motivational factors, while patients also present a similar variety of reasons for declining its use. The results emphasize the urgent need for knowledge and patient education regarding factors that may hinder the decision to use ORA, including its practical usage, its application possibilities, and concerns related to data privacy. Further research is needed to explore approaches for adequately preparing individuals with SMI to transition from their inherent interest to active engagement with ORA.
BACKGROUND: German Clinical Trial Register DRKS00030188; https://drks.de/search/en/trial/DRKS00030188.

To enhance phenotype recognition in clinical notes of genetic diseases, we developed two models-PhenoBCBERT and PhenoGPT-for expanding the vocabularies of Human Phenotype Ontology (HPO) terms. While HPO offers a standardized vocabulary for phenotypes, existing tools often fail to capture the full scope of phenotypes due to limitations from traditional heuristic or rule-based approaches. Our models leverage large language models to automate the detection of phenotype terms, including those not in the current HPO. We compare these models with PhenoTagger, another HPO recognition tool, and found that our models identify a wider range of phenotype concepts, including previously uncharacterized ones. Our models also show strong performance in case studies on biomedical literature. We evaluate the strengths and weaknesses of BERT- and GPT-based models in aspects such as architecture and accuracy. Overall, our models enhance automated phenotype detection from clinical texts, improving downstream analyses on human diseases.

BACKGROUND: Information regarding opioid use disorder (OUD) status and severity is important for patient care. Clinical notes provide valuable information for detecting and characterizing problematic opioid use, necessitating development of natural language processing (NLP) tools, which in turn requires reliably labeled OUD-relevant text and understanding of documentation patterns.
OBJECTIVE: To inform automated NLP methods, we aimed to develop and evaluate an annotation schema for characterizing OUD and its severity, and to document patterns of OUD-relevant information within clinical notes of heterogeneous patient cohorts.
METHODS: We developed an annotation schema to characterize OUD severity based on criteria from the Diagnostic and Statistical Manual of Mental Disorders, 5th edition. In total, 2 annotators reviewed clinical notes from key encounters of 100 adult patients with varied evidence of OUD, including patients with and those without chronic pain, with and without medication treatment for OUD, and a control group. We completed annotations at the sentence level. We calculated severity scores based on annotation of note text with 18 classes aligned with criteria for OUD severity and determined positive predictive values for OUD severity.
RESULTS: The annotation schema contained 27 classes. We annotated 1436 sentences from 82 patients; notes of 18 patients (11 of whom were controls) contained no relevant information. Interannotator agreement was above 70% for 11 of 15 batches of reviewed notes. Severity scores for control group patients were all 0. Among noncontrol patients, the mean severity score was 5.1 (SD 3.2), indicating moderate OUD, and the positive predictive value for detecting moderate or severe OUD was 0.71. Progress notes and notes from emergency department and outpatient settings contained the most and greatest diversity of information. Substance misuse and psychiatric classes were most prevalent and highly correlated across note types with high co-occurrence across patients.
CONCLUSIONS: Implementation of the annotation schema demonstrated strong potential for inferring OUD severity based on key information in a small set of clinical notes and highlighting where such information is documented. These advancements will facilitate NLP tool development to improve OUD prevention, diagnosis, and treatment.