UNASSIGNED: Teaching medical students the skills required to acquire, interpret, apply, and communicate clinical information is an integral part of medical education. A crucial aspect of this process involves providing students with feedback regarding the quality of their free-text clinical notes.
UNASSIGNED: The goal of this study was to assess the ability of ChatGPT 3.5, a large language model, to score medical students\' free-text history and physical notes.
UNASSIGNED: This is a single-institution, retrospective study. Standardized patients learned a prespecified clinical case and, acting as the patient, interacted with medical students. Each student wrote a free-text history and physical note of their interaction. The students\' notes were scored independently by the standardized patients and ChatGPT using a prespecified scoring rubric that consisted of 85 case elements. The measure of accuracy was percent correct.
UNASSIGNED: The study population consisted of 168 first-year medical students. There was a total of 14,280 scores. The ChatGPT incorrect scoring rate was 1.0%, and the standardized patient incorrect scoring rate was 7.2%. The ChatGPT error rate was 86%, lower than the standardized patient error rate. The ChatGPT mean incorrect scoring rate of 12 (SD 11) was significantly lower than the standardized patient mean incorrect scoring rate of 85 (SD 74; P=.002).
UNASSIGNED: ChatGPT demonstrated a significantly lower error rate compared to standardized patients. This is the first study to assess the ability of a generative pretrained transformer (GPT) program to score medical students\' standardized patient-based free-text clinical notes. It is expected that, in the near future, large language models will provide real-time feedback to practicing physicians regarding their free-text notes. GPT artificial intelligence programs represent an important advance in medical education and medical practice.

OBJECTIVE: Active learning (AL) has rarely integrated diversity-based and uncertainty-based strategies into a dynamic sampling framework for clinical named entity recognition (NER). Machine-assisted annotation is becoming popular for creating gold-standard labels. This study investigated the effectiveness of dynamic AL strategies under simulated machine-assisted annotation scenarios for clinical NER.
METHODS: We proposed 3 new AL strategies: a diversity-based strategy (CLUSTER) based on Sentence-BERT and 2 dynamic strategies (CLC and CNBSE) capable of switching from diversity-based to uncertainty-based strategies. Using BioClinicalBERT as the foundational NER model, we conducted simulation experiments on 3 medication-related clinical NER datasets independently: i2b2 2009, n2c2 2018 (Track 2), and MADE 1.0. We compared the proposed strategies with uncertainty-based (LC and NBSE) and passive-learning (RANDOM) strategies. Performance was primarily measured by the number of edits made by the annotators to achieve a desired target effectiveness evaluated on independent test sets.
RESULTS: When aiming for 98% overall target effectiveness, on average, CLUSTER required the fewest edits. When aiming for 99% overall target effectiveness, CNBSE required 20.4% fewer edits than NBSE did. CLUSTER and RANDOM could not achieve such a high target under the pool-based simulation experiment. For high-difficulty entities, CNBSE required 22.5% fewer edits than NBSE to achieve 99% target effectiveness, whereas neither CLUSTER nor RANDOM achieved 93% target effectiveness.
CONCLUSIONS: When the target effectiveness was set high, the proposed dynamic strategy CNBSE exhibited both strong learning capabilities and low annotation costs in machine-assisted annotation. CLUSTER required the fewest edits when the target effectiveness was set low.

Home healthcare (HHC) enables patients to receive health services within their homes. Social determinants of health (SDOH) influence a patient\'s health and may disproportionately affect patients from racially and ethnically minoritized groups. This study describes differences in SDOH documentation in clinical notes among individuals from different racial or ethnic groups from one HHC agency in the northeastern United States. Compared to White patients, HHC episodes for patients across racially and ethnically minoritized groups had higher frequencies of SDOH documented. Further, our results suggest that race or ethnicity is significantly associated with SDOH documentation.

OBJECTIVE: Alzheimer\'s disease (AD) is the most common form of dementia in the United States. Sleep is one of the lifestyle-related factors that has been shown critical for optimal cognitive function in old age. However, there is a lack of research studying the association between sleep and AD incidence. A major bottleneck for conducting such research is that the traditional way to acquire sleep information is time-consuming, inefficient, non-scalable, and limited to patients\' subjective experience. We aim to automate the extraction of specific sleep-related patterns, such as snoring, napping, poor sleep quality, daytime sleepiness, night wakings, other sleep problems, and sleep duration, from clinical notes of AD patients. These sleep patterns are hypothesized to play a role in the incidence of AD, providing insight into the relationship between sleep and AD onset and progression.
METHODS: A gold standard dataset is created from manual annotation of 570 randomly sampled clinical note documents from the adSLEEP, a corpus of 192 000 de-identified clinical notes of 7266 AD patients retrieved from the University of Pittsburgh Medical Center (UPMC). We developed a rule-based natural language processing (NLP) algorithm, machine learning models, and large language model (LLM)-based NLP algorithms to automate the extraction of sleep-related concepts, including snoring, napping, sleep problem, bad sleep quality, daytime sleepiness, night wakings, and sleep duration, from the gold standard dataset.
RESULTS: The annotated dataset of 482 patients comprised a predominantly White (89.2%), older adult population with an average age of 84.7 years, where females represented 64.1%, and a vast majority were non-Hispanic or Latino (94.6%). Rule-based NLP algorithm achieved the best performance of F1 across all sleep-related concepts. In terms of positive predictive value (PPV), the rule-based NLP algorithm achieved the highest PPV scores for daytime sleepiness (1.00) and sleep duration (1.00), while the machine learning models had the highest PPV for napping (0.95) and bad sleep quality (0.86), and LLAMA2 with finetuning had the highest PPV for night wakings (0.93) and sleep problem (0.89).
CONCLUSIONS: Although sleep information is infrequently documented in the clinical notes, the proposed rule-based NLP algorithm and LLM-based NLP algorithms still achieved promising results. In comparison, the machine learning-based approaches did not achieve good results, which is due to the small size of sleep information in the training data.
CONCLUSIONS: The results show that the rule-based NLP algorithm consistently achieved the best performance for all sleep concepts. This study focused on the clinical notes of patients with AD but could be extended to general sleep information extraction for other diseases.

UNASSIGNED: ANCA-associated vasculitis (AAV) is a rare but serious disease. Traditional case-identification methods using claims data can be time-intensive and may miss important subgroups. We hypothesized that a deep learning model analyzing electronic health records (EHR) can more accurately identify AAV cases.
UNASSIGNED: We examined the Mass General Brigham (MGB) repository of clinical documentation from 12/1/1979 to 5/11/2021, using expert-curated keywords and ICD codes to identify a large cohort of potential AAV cases. Three labeled datasets (I, II, III) were created, each containing note sections. We trained and evaluated a range of machine learning and deep learning algorithms for note-level classification, using metrics like positive predictive value (PPV), sensitivity, F-score, area under the receiver operating characteristic curve (AUROC), and area under the precision and recall curve (AUPRC). The deep learning model was further evaluated for its ability to classify AAV cases at the patient-level, compared with rule-based algorithms in 2,000 randomly chosen samples.
UNASSIGNED: Datasets I, II, and III comprised 6,000, 3,008, and 7,500 note sections, respectively. Deep learning achieved the highest AUROC in all three datasets, with scores of 0.983, 0.991, and 0.991. The deep learning approach also had among the highest PPVs across the three datasets (0.941, 0.954, and 0.800, respectively). In a test cohort of 2,000 cases, the deep learning model achieved a PPV of 0.262 and an estimated sensitivity of 0.975. Compared to the best rule-based algorithm, the deep learning model identified six additional AAV cases, representing 13% of the total.
UNASSIGNED: The deep learning model effectively classifies clinical note sections for AAV diagnosis. Its application to EHR notes can potentially uncover additional cases missed by traditional rule-based methods.

UNASSIGNED: In many countries, health care professionals are legally obliged to share information from electronic health records with patients. However, concerns have been raised regarding the sharing of notes with adolescents in mental health care, and health care professionals have called for recommendations to guide this practice.
UNASSIGNED: The aim was to reach a consensus among authors of scientific papers on recommendations for health care professionals\' digital sharing of notes with adolescents in mental health care and to investigate whether staff at child and adolescent specialist mental health care clinics agreed with the recommendations.
UNASSIGNED: A Delphi study was conducted with authors of scientific papers to reach a consensus on recommendations. The process of making the recommendations involved three steps. First, scientific papers meeting the eligibility criteria were identified through a PubMed search where the references were screened. Second, the results from the included papers were coded and transformed into recommendations in an iterative process. Third, the authors of the included papers were asked to provide feedback and consider their agreement with each of the suggested recommendations in two rounds. After the Delphi process, a cross-sectional study was conducted among staff at specialist child and adolescent mental health care clinics to assess whether they agreed with the recommendations that reached a consensus.
UNASSIGNED: Of the 84 invited authors, 27 responded. A consensus was reached on 17 recommendations on areas related to digital sharing of notes with adolescents in mental health care. The recommendations considered how to introduce digital access to notes, write notes, and support health care professionals, and when to withhold notes. Of the 41 staff members at child and adolescent specialist mental health care clinics, 60% or more agreed with the 17 recommendations. No consensus was reached regarding the age at which adolescents should receive digital access to their notes and the timing of digitally sharing notes with parents.
UNASSIGNED: A total of 17 recommendations related to key aspects of health care professionals\' digital sharing of notes with adolescents in mental health care achieved consensus. Health care professionals can use these recommendations to guide their practice of sharing notes with adolescents in mental health care. However, the effects and experiences of following these recommendations should be tested in clinical practice.

OBJECTIVE: In a growing list of countries, patients are granted access to their clinical notes (\"open notes\") as part of their online record access. Especially in the field of mental health, open notes remain controversial with some clinicians perceiving open notes as a tool for improving therapeutic outcomes by increasing patient involvement, while others fear that patients might experience psychological distress and perceived stigmatization, particularly when reading clinicians\' notes. More research is needed to optimize the benefits and mitigate the risks.
METHODS: Using a qualitative research design, we conducted semi-structured interviews with psychiatrists practicing in Germany, to explore what conditions they believe need to be in place to ensure successful implementation of open notes in psychiatric practice as well as expected subsequent changes to their workload and treatment outcomes. Data were analyzed using thematic analysis.
RESULTS: We interviewed 18 psychiatrists; interviewees believed four key conditions needed to be in place prior to implementation of open notes including careful consideration of (1) diagnoses and symptom severity, (2) the availability of additional time for writing clinical notes and discussing them with patients, (3) available resources and system compatibility, and (4) legal and data protection aspects. As a result of introducing open notes, interviewees expected changes in documentation, treatment processes, and doctor-physician interaction. While open notes were expected to improve transparency and trust, participants anticipated negative unintended consequences including the risk of deteriorating therapeutic relationships due to note access-related misunderstandings and conflicts.
CONCLUSIONS: Psychiatrists practiced in Germany where open notes have not yet been established as part of the healthcare data infrastructure. Interviewees were supportive of open notes but had some reservations. They found open notes to be generally beneficial but anticipated effects to vary depending on patient characteristics. Clear guidelines for managing access, time constraints, usability, and privacy are crucial. Open notes were perceived to increase transparency and patient involvement but were also believed to raise issues of stigmatization and conflicts.

BACKGROUND: Extracting research of domain criteria (RDoC) from high-risk populations like those with post-traumatic stress disorder (PTSD) is crucial for positive mental health improvements and policy enhancements. The intricacies of collecting, integrating, and effectively leveraging clinical notes for this purpose introduce complexities.
METHODS: In our study, we created a natural language processing (NLP) workflow to analyze electronic medical record (EMR) data and identify and extract research of domain criteria using a pre-trained transformer-based natural language model, all-mpnet-base-v2. We subsequently built dictionaries from 100,000 clinical notes and analyzed 5.67 million clinical notes from 38,807 PTSD patients from the University of Pittsburgh Medical Center. Subsequently, we showcased the significance of our approach by extracting and visualizing RDoC information in two use cases: (i) across multiple patient populations and (ii) throughout various disease trajectories.
RESULTS: The sentence transformer model demonstrated high F1 macro scores across all RDoC domains, achieving the highest performance with a cosine similarity threshold value of 0.3. This ensured an F1 score of at least 80% across all RDoC domains. The study revealed consistent reductions in all six RDoC domains among PTSD patients after psychotherapy. We found that 60.6% of PTSD women have at least one abnormal instance of the six RDoC domains as compared to PTSD men (51.3%), with 45.1% of PTSD women with higher levels of sensorimotor disturbances compared to men (41.3%). We also found that 57.3% of PTSD patients have at least one abnormal instance of the six RDoC domains based on our records. Also, veterans had the higher abnormalities of negative and positive valence systems (60% and 51.9% of veterans respectively) compared to non-veterans (59.1% and 49.2% respectively). The domains following first diagnoses of PTSD were associated with heightened cue reactivity to trauma, suicide, alcohol, and substance consumption.
CONCLUSIONS: The findings provide initial insights into RDoC functioning in different populations and disease trajectories. Natural language processing proves valuable for capturing real-time, context dependent RDoC instances from extensive clinical notes.

BACKGROUND: Home healthcare (HHC) enables patients to receive healthcare services within their homes to manage chronic conditions and recover from illnesses. Recent research has identified disparities in HHC based on race or ethnicity. Social determinants of health (SDOH) describe the external factors influencing a patient\'s health, such as access to care and social support. Individuals from racially or ethnically minoritized communities are known to be disproportionately affected by SDOH. Existing evidence suggests that SDOH are documented in clinical notes. However, no prior study has investigated the documentation of SDOH across individuals from different racial or ethnic backgrounds in the HHC setting. This study aimed to (1) describe frequencies of SDOH documented in clinical notes by race or ethnicity and (2) determine associations between race or ethnicity and SDOH documentation.
METHODS: Retrospective data analysis.
METHODS: We conducted a cross-sectional secondary data analysis of 86,866 HHC episodes representing 65,693 unique patients from one large HHC agency in New York collected between January 1, 2015, and December 31, 2017. We reported the frequency of six SDOH (physical environment, social environment, housing and economic circumstances, food insecurity, access to care, and education and literacy) documented in clinical notes across individuals reported as Asian/Pacific Islander, Black, Hispanic, multi-racial, Native American, or White. We analyzed differences in SDOH documentation by race or ethnicity using logistic regression models.
RESULTS: Compared to patients reported as White, patients across other racial or ethnic groups had higher frequencies of SDOH documented in their clinical notes. Our results suggest that race or ethnicity is associated with SDOH documentation in HHC.
CONCLUSIONS: As the study of SDOH in HHC continues to evolve, our results provide a foundation to evaluate social information in the HHC setting and understand how it influences the quality of care provided.
CONCLUSIONS: The results of this exploratory study can help clinicians understand the differences in SDOH across individuals from different racial and ethnic groups and serve as a foundation for future research aimed at fostering more inclusive HHC documentation practices.

Building clinical registries is an important step in clinical research and improvement of patient care quality. Natural Language Processing (NLP) methods have shown promising results in extracting valuable information from unstructured clinical notes. However, the structure and nature of clinical notes are very different from regular text that state-of-the-art NLP models are trained and tested on, and they have their own set of challenges. In this study, we propose Sentence Extractor with Keywords (SE-K), an efficient and interpretable classification approach for extracting information from clinical notes and show that it outperforms more computationally expensive methods in text classification. Following the Institutional Review Board (IRB) approval, we used SE-K and two embedding based NLP approaches (Sentence Extractor with Embeddings (SE-E) and Bidirectional Encoder Representations from Transformers (BERT)) to develop comprehensive registry of anterior cruciate ligament surgeries from 20 years of unstructured clinical data at a multi-site tertiary-care regional children\'s hospital. The low-resource approach (SE-K) had better performance (average AUROC of 0.94 ± 0.04) than the embedding-based approaches (SE-E: 0.93 ± 0.04 and BERT: 0.87 ± 0.09) for out of sample validation, in addition to minimum performance drop between test and out-of-sample validation. Moreover, the SE-K approach was at least six times faster (on CPU) than SE-E (on CPU) and BERT (on GPU) and provides interpretability. Our proposed approach, SE-K, can be effectively used to extract relevant variables from clinic notes to build large-scale registries, with consistently better performance compared to the more resource-intensive approaches (e.g., BERT). Such approaches can facilitate information extraction from unstructured notes for registry building, quality improvement and adverse event monitoring.