PDF(Pubmed)
Abstract：
BACKGROUND: Medication safety in residential care facilities is a critical concern, particularly when nonmedical staff provide medication assistance. The complex nature of medication-related incidents in these settings, coupled with the psychological impact on health care providers, underscores the need for effective incident analysis and preventive strategies. A thorough understanding of the root causes, typically through incident-report analysis, is essential for mitigating medication-related incidents.
OBJECTIVE: We aimed to develop and evaluate a multilabel classifier using natural language processing to identify factors contributing to medication-related incidents using incident report descriptions from residential care facilities, with a focus on incidents involving nonmedical staff.
METHODS: We analyzed 2143 incident reports, comprising 7121 sentences, from residential care facilities in Japan between April 1, 2015, and March 31, 2016. The incident factors were annotated using sentences based on an established organizational factor model and previous research findings. The following 9 factors were defined: procedure adherence, medicine, resident, resident family, nonmedical staff, medical staff, team, environment, and organizational management. To assess the label criteria, 2 researchers with relevant medical knowledge annotated a subset of 50 reports; the interannotator agreement was measured using Cohen κ. The entire data set was subsequently annotated by 1 researcher. Multiple labels were assigned to each sentence. A multilabel classifier was developed using deep learning models, including 2 Bidirectional Encoder Representations From Transformers (BERT)-type models (Tohoku-BERT and a University of Tokyo Hospital BERT pretrained with Japanese clinical text: UTH-BERT) and an Efficiently Learning Encoder That Classifies Token Replacements Accurately (ELECTRA), pretrained on Japanese text. Both sentence- and report-level training were performed; the performance was evaluated by the F1-score and exact match accuracy through 5-fold cross-validation.
RESULTS: Among all 7121 sentences, 1167, 694, 2455, 23, 1905, 46, 195, 1104, and 195 included \"procedure adherence,\" \"medicine,\" \"resident,\" \"resident family,\" \"nonmedical staff,\" \"medical staff,\" \"team,\" \"environment,\" and \"organizational management,\" respectively. Owing to limited labels, \"resident family\" and \"medical staff\" were omitted from the model development process. The interannotator agreement values were higher than 0.6 for each label. A total of 10, 278, and 1855 reports contained no, 1, and multiple labels, respectively. The models trained using the report data outperformed those trained using sentences, with macro F1-scores of 0.744, 0.675, and 0.735 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. The report-trained models also demonstrated better exact match accuracy, with 0.411, 0.389, and 0.399 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. Notably, the accuracy was consistent even when the analysis was confined to reports containing multiple labels.
CONCLUSIONS: The multilabel classifier developed in our study demonstrated potential for identifying various factors associated with medication-related incidents using incident reports from residential care facilities. Thus, this classifier can facilitate prompt analysis of incident factors, thereby contributing to risk management and the development of preventive strategies.
OBJECTIVE: We aimed to develop and evaluate a multilabel classifier using natural language processing to identify factors contributing to medication-related incidents using incident report descriptions from residential care facilities, with a focus on incidents involving nonmedical staff.
METHODS: We analyzed 2143 incident reports, comprising 7121 sentences, from residential care facilities in Japan between April 1, 2015, and March 31, 2016. The incident factors were annotated using sentences based on an established organizational factor model and previous research findings. The following 9 factors were defined: procedure adherence, medicine, resident, resident family, nonmedical staff, medical staff, team, environment, and organizational management. To assess the label criteria, 2 researchers with relevant medical knowledge annotated a subset of 50 reports; the interannotator agreement was measured using Cohen κ. The entire data set was subsequently annotated by 1 researcher. Multiple labels were assigned to each sentence. A multilabel classifier was developed using deep learning models, including 2 Bidirectional Encoder Representations From Transformers (BERT)-type models (Tohoku-BERT and a University of Tokyo Hospital BERT pretrained with Japanese clinical text: UTH-BERT) and an Efficiently Learning Encoder That Classifies Token Replacements Accurately (ELECTRA), pretrained on Japanese text. Both sentence- and report-level training were performed; the performance was evaluated by the F1-score and exact match accuracy through 5-fold cross-validation.
RESULTS: Among all 7121 sentences, 1167, 694, 2455, 23, 1905, 46, 195, 1104, and 195 included \"procedure adherence,\" \"medicine,\" \"resident,\" \"resident family,\" \"nonmedical staff,\" \"medical staff,\" \"team,\" \"environment,\" and \"organizational management,\" respectively. Owing to limited labels, \"resident family\" and \"medical staff\" were omitted from the model development process. The interannotator agreement values were higher than 0.6 for each label. A total of 10, 278, and 1855 reports contained no, 1, and multiple labels, respectively. The models trained using the report data outperformed those trained using sentences, with macro F1-scores of 0.744, 0.675, and 0.735 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. The report-trained models also demonstrated better exact match accuracy, with 0.411, 0.389, and 0.399 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. Notably, the accuracy was consistent even when the analysis was confined to reports containing multiple labels.
CONCLUSIONS: The multilabel classifier developed in our study demonstrated potential for identifying various factors associated with medication-related incidents using incident reports from residential care facilities. Thus, this classifier can facilitate prompt analysis of incident factors, thereby contributing to risk management and the development of preventive strategies.
摘要：
背景：住宅护理设施中的药物安全是一个至关重要的问题，特别是当非医务人员提供药物援助。在这些环境中，药物相关事件的复杂性，加上对医疗保健提供者的心理影响，强调需要有效的事件分析和预防策略。深入了解根本原因,通常通过事件报告分析，对于缓解与药物相关的事件至关重要。
目的：我们旨在使用自然语言处理开发和评估多标签分类器，以使用住宅护理设施的事件报告描述来识别导致药物相关事件的因素。重点关注涉及非医务人员的事件。
方法：我们分析了2143个事件报告，包括7121个句子，2015年4月1日至2016年3月31日期间来自日本的住宅护理设施。根据已建立的组织因素模型和先前的研究结果，使用句子对事件因素进行了注释。定义了以下9个因素：程序依从性，医学,居民,居民家庭,非医务人员,医务人员,团队,环境,和组织管理。要评估标签标准，2位具有相关医学知识的研究人员注释了50份报告的子集；使用Cohenκ测量了注释者之间的一致性。整个数据集随后由1名研究人员注释。为每个句子分配了多个标签。使用深度学习模型开发了多标签分类器，包括2个来自变形金刚（BERT）型模型的双向编码器表示（Tohoku-BERT和东京大学医院BERT预先训练了日本临床文本：UTH-BERT）和一个有效的学习编码器，该编码器可以准确地对令牌替换进行分类（ELECTRA），对日语文本进行了预培训。进行了句子和报告级别的培训；通过5倍交叉验证，通过F1评分和精确匹配准确性来评估性能。
结果：在所有7121个句子中，1167、694、2455、23、1905、46、195、1104和195包括“程序遵守，\"\"药,\"\"居民,\"\"常住家庭,\"\"非医务人员，\"\"医务人员,\"\"团队,\"\"环境,“和”组织管理，\"分别。由于标签有限，模型开发过程中省略了“居民家庭”和“医务人员”。每个标签的注释间一致性值高于0.6。共有10份、278份和1855份报告没有，1，和多个标签，分别。使用报告数据训练的模型优于使用句子训练的模型，东北-BERT的宏F1分数为0.744、0.675和0.735，UTH-BERT,和ELECTRA,分别。报告训练的模型还展示了更好的精确匹配准确性，东北BERT为0.411、0.389和0.399，UTH-BERT,和ELECTRA,分别。值得注意的是,即使分析仅限于包含多个标签的报告,准确性也是一致的.
结论：在我们的研究中开发的多标签分类器证明了使用来自住宅护理机构的事件报告来识别与药物相关事件相关的各种因素的潜力。因此，该分类器可以方便快速分析事件因素，从而有助于风险管理和预防战略的制定。
目的：我们旨在使用自然语言处理开发和评估多标签分类器，以使用住宅护理设施的事件报告描述来识别导致药物相关事件的因素。重点关注涉及非医务人员的事件。
方法：我们分析了2143个事件报告，包括7121个句子，2015年4月1日至2016年3月31日期间来自日本的住宅护理设施。根据已建立的组织因素模型和先前的研究结果，使用句子对事件因素进行了注释。定义了以下9个因素：程序依从性，医学,居民,居民家庭,非医务人员,医务人员,团队,环境,和组织管理。要评估标签标准，2位具有相关医学知识的研究人员注释了50份报告的子集；使用Cohenκ测量了注释者之间的一致性。整个数据集随后由1名研究人员注释。为每个句子分配了多个标签。使用深度学习模型开发了多标签分类器，包括2个来自变形金刚（BERT）型模型的双向编码器表示（Tohoku-BERT和东京大学医院BERT预先训练了日本临床文本：UTH-BERT）和一个有效的学习编码器，该编码器可以准确地对令牌替换进行分类（ELECTRA），对日语文本进行了预培训。进行了句子和报告级别的培训；通过5倍交叉验证，通过F1评分和精确匹配准确性来评估性能。
结果：在所有7121个句子中，1167、694、2455、23、1905、46、195、1104和195包括“程序遵守，\"\"药,\"\"居民,\"\"常住家庭,\"\"非医务人员，\"\"医务人员,\"\"团队,\"\"环境,“和”组织管理，\"分别。由于标签有限，模型开发过程中省略了“居民家庭”和“医务人员”。每个标签的注释间一致性值高于0.6。共有10份、278份和1855份报告没有，1，和多个标签，分别。使用报告数据训练的模型优于使用句子训练的模型，东北-BERT的宏F1分数为0.744、0.675和0.735，UTH-BERT,和ELECTRA,分别。报告训练的模型还展示了更好的精确匹配准确性，东北BERT为0.411、0.389和0.399，UTH-BERT,和ELECTRA,分别。值得注意的是,即使分析仅限于包含多个标签的报告,准确性也是一致的.
结论：在我们的研究中开发的多标签分类器证明了使用来自住宅护理机构的事件报告来识别与药物相关事件相关的各种因素的潜力。因此，该分类器可以方便快速分析事件因素，从而有助于风险管理和预防战略的制定。
