PDF(Pubmed)
Abstract：
Assessing patient-reported outcomes (PROs) through interviews or conversations during clinical encounters provides insightful information about survivorship.
This study aims to test the validity of natural language processing (NLP) and machine learning (ML) algorithms in identifying different attributes of pain interference and fatigue symptoms experienced by child and adolescent survivors of cancer versus the judgment by PRO content experts as the gold standard to validate NLP/ML algorithms.
This cross-sectional study focused on child and adolescent survivors of cancer, aged 8 to 17 years, and caregivers, from whom 391 meaning units in the pain interference domain and 423 in the fatigue domain were generated for analyses. Data were collected from the After Completion of Therapy Clinic at St. Jude Children\'s Research Hospital. Experienced pain interference and fatigue symptoms were reported through in-depth interviews. After verbatim transcription, analyzable sentences (ie, meaning units) were semantically labeled by 2 content experts for each attribute (physical, cognitive, social, or unclassified). Two NLP/ML methods were used to extract and validate the semantic features: bidirectional encoder representations from transformers (BERT) and Word2vec plus one of the ML methods, the support vector machine or extreme gradient boosting. Receiver operating characteristic and precision-recall curves were used to evaluate the accuracy and validity of the NLP/ML methods.
Compared with Word2vec/support vector machine and Word2vec/extreme gradient boosting, BERT demonstrated higher accuracy in both symptom domains, with 0.931 (95% CI 0.905-0.957) and 0.916 (95% CI 0.887-0.941) for problems with cognitive and social attributes on pain interference, respectively, and 0.929 (95% CI 0.903-0.953) and 0.917 (95% CI 0.891-0.943) for problems with cognitive and social attributes on fatigue, respectively. In addition, BERT yielded superior areas under the receiver operating characteristic curve for cognitive attributes on pain interference and fatigue domains (0.923, 95% CI 0.879-0.997; 0.948, 95% CI 0.922-0.979) and superior areas under the precision-recall curve for cognitive attributes on pain interference and fatigue domains (0.818, 95% CI 0.735-0.917; 0.855, 95% CI 0.791-0.930).
The BERT method performed better than the other methods. As an alternative to using standard PRO surveys, collecting unstructured PROs via interviews or conversations during clinical encounters and applying NLP/ML methods can facilitate PRO assessment in child and adolescent cancer survivors.
This study aims to test the validity of natural language processing (NLP) and machine learning (ML) algorithms in identifying different attributes of pain interference and fatigue symptoms experienced by child and adolescent survivors of cancer versus the judgment by PRO content experts as the gold standard to validate NLP/ML algorithms.
This cross-sectional study focused on child and adolescent survivors of cancer, aged 8 to 17 years, and caregivers, from whom 391 meaning units in the pain interference domain and 423 in the fatigue domain were generated for analyses. Data were collected from the After Completion of Therapy Clinic at St. Jude Children\'s Research Hospital. Experienced pain interference and fatigue symptoms were reported through in-depth interviews. After verbatim transcription, analyzable sentences (ie, meaning units) were semantically labeled by 2 content experts for each attribute (physical, cognitive, social, or unclassified). Two NLP/ML methods were used to extract and validate the semantic features: bidirectional encoder representations from transformers (BERT) and Word2vec plus one of the ML methods, the support vector machine or extreme gradient boosting. Receiver operating characteristic and precision-recall curves were used to evaluate the accuracy and validity of the NLP/ML methods.
Compared with Word2vec/support vector machine and Word2vec/extreme gradient boosting, BERT demonstrated higher accuracy in both symptom domains, with 0.931 (95% CI 0.905-0.957) and 0.916 (95% CI 0.887-0.941) for problems with cognitive and social attributes on pain interference, respectively, and 0.929 (95% CI 0.903-0.953) and 0.917 (95% CI 0.891-0.943) for problems with cognitive and social attributes on fatigue, respectively. In addition, BERT yielded superior areas under the receiver operating characteristic curve for cognitive attributes on pain interference and fatigue domains (0.923, 95% CI 0.879-0.997; 0.948, 95% CI 0.922-0.979) and superior areas under the precision-recall curve for cognitive attributes on pain interference and fatigue domains (0.818, 95% CI 0.735-0.917; 0.855, 95% CI 0.791-0.930).
The BERT method performed better than the other methods. As an alternative to using standard PRO surveys, collecting unstructured PROs via interviews or conversations during clinical encounters and applying NLP/ML methods can facilitate PRO assessment in child and adolescent cancer survivors.
摘要：
在临床接触期间通过访谈或对话评估患者报告的结果（PRO）提供了有关生存的深刻信息。
这项研究旨在测试自然语言处理（NLP）和机器学习（ML）算法在识别儿童和青少年癌症幸存者所经历的疼痛干扰和疲劳症状的不同属性方面的有效性，以及PRO内容专家的判断作为验证NLP/ML算法的黄金标准。
这项横断面研究的重点是儿童和青少年癌症幸存者，8至17岁，和照顾者，从中产生疼痛干扰域中的391个含义单位和疲劳域中的423个含义单位用于分析.数据来自圣裘德儿童研究医院治疗完成后的诊所。通过深入访谈报告了经历的疼痛干扰和疲劳症状。逐字转录后，可分析的句子(即，含义单位)由2位内容专家对每个属性(物理，认知，社会,或未分类)。使用两种NLP/ML方法来提取和验证语义特征：来自变压器（BERT）和Word2vec的双向编码器表示以及ML方法之一，支持向量机或极端梯度提升。使用接收器工作特性和精确召回曲线来评估NLP/ML方法的准确性和有效性。
与Word2vec/支持向量机和Word2vec/极端梯度提升相比，BERT在两个症状领域都表现出更高的准确性，对于疼痛干扰的认知和社会属性问题，分别为0.931(95%CI0.905-0.957)和0.916(95%CI0.887-0.941)，分别,和0.929（95％CI0.903-0.953）和0.917（95％CI0.891-0.943）用于疲劳的认知和社会属性问题，分别。此外,BERT在疼痛干扰和疲劳领域的认知属性在受试者工作特征曲线下产生了优越的区域（0.923，95％CI0.879-0.997；0.948，95％CI0.922-0.979），在疼痛干扰和疲劳领域的认知属性在精确召回曲线下产生了优越的区域（0.818，95％CI0.735-0.917；0.855，95％CI0.791-0.930）。
BERT方法的性能优于其他方法。作为使用标准PRO调查的替代方法，在临床接触期间通过访谈或对话收集非结构化的PRO，并应用NLP/ML方法可以促进儿童和青少年癌症幸存者的PRO评估。
这项研究旨在测试自然语言处理（NLP）和机器学习（ML）算法在识别儿童和青少年癌症幸存者所经历的疼痛干扰和疲劳症状的不同属性方面的有效性，以及PRO内容专家的判断作为验证NLP/ML算法的黄金标准。
这项横断面研究的重点是儿童和青少年癌症幸存者，8至17岁，和照顾者，从中产生疼痛干扰域中的391个含义单位和疲劳域中的423个含义单位用于分析.数据来自圣裘德儿童研究医院治疗完成后的诊所。通过深入访谈报告了经历的疼痛干扰和疲劳症状。逐字转录后，可分析的句子(即，含义单位)由2位内容专家对每个属性(物理，认知，社会,或未分类)。使用两种NLP/ML方法来提取和验证语义特征：来自变压器（BERT）和Word2vec的双向编码器表示以及ML方法之一，支持向量机或极端梯度提升。使用接收器工作特性和精确召回曲线来评估NLP/ML方法的准确性和有效性。
与Word2vec/支持向量机和Word2vec/极端梯度提升相比，BERT在两个症状领域都表现出更高的准确性，对于疼痛干扰的认知和社会属性问题，分别为0.931(95%CI0.905-0.957)和0.916(95%CI0.887-0.941)，分别,和0.929（95％CI0.903-0.953）和0.917（95％CI0.891-0.943）用于疲劳的认知和社会属性问题，分别。此外,BERT在疼痛干扰和疲劳领域的认知属性在受试者工作特征曲线下产生了优越的区域（0.923，95％CI0.879-0.997；0.948，95％CI0.922-0.979），在疼痛干扰和疲劳领域的认知属性在精确召回曲线下产生了优越的区域（0.818，95％CI0.735-0.917；0.855，95％CI0.791-0.930）。
BERT方法的性能优于其他方法。作为使用标准PRO调查的替代方法，在临床接触期间通过访谈或对话收集非结构化的PRO，并应用NLP/ML方法可以促进儿童和青少年癌症幸存者的PRO评估。
