背景:在COVID-19大流行之后,有限的精神卫生保健资源与快速增长的患者数量之间的冲突变得更加明显。心理学家有必要借用基于人工智能(AI)的方法来分析接受精神疾病治疗的患者对药物治疗的满意度。
目的:我们的目标是通过分析精神疾病患者对药物摄入的经验和评论,构建高度准确和可转移的模型来预测他们对药物的满意度。
方法:我们从16,950个疾病类别的161,297条评论的大型公开数据集中,提取了与精神疾病相关的20种疾病类别的41,851条评论。为了发现自然语言处理模型的更优化结构,我们提出了统一的可互换模型融合来分解来自变压器(BERT)的最先进的双向编码器表示,支持向量机,和随机森林(RF)模型分为2个模块,编码器和分类器,然后重建融合的“编码器+分类器”模型,以准确评估患者的满意度。根据模型结构,融合模型分为两类,传统的基于机器学习的模型和基于神经网络的模型。针对这些基于神经网络的模型,提出了一种新的损失函数,以克服过拟合和数据不平衡的问题。最后,我们对融合模型进行了微调,并根据F1得分对其性能进行了全面评估,准确度,κ系数,和使用10倍交叉验证的训练时间。
结果:通过广泛的实验,变压器双向编码器+RF模型优于最先进的BERT,MentalBERT,和其他融合模型。它成为预测患者对药物治疗满意度的最佳模型。它的平均F1评分为0.872,准确率为0.873,κ系数为0.806。该模型适用于拥有充足计算资源的高标准用户。或者,事实证明,单词嵌入编码器RF模型显示出相对较好的性能,平均F1评分为0.801,精度为0.812,κ系数为0.695,但训练时间要少得多。它可以部署在计算资源有限的环境中。
结论:我们分析了支持向量机的性能,射频,BERT,MentalBERT,和所有融合模型,并确定了不同临床场景的最佳模型。这些发现可以作为证据,支持自然语言处理方法可以有效地帮助心理学家评估患者对药物治疗计划的满意度,并提供精确和标准化的解决方案。统一的可互换模型融合为构建心理健康的AI模型提供了不同的视角,并有可能将模型的不同组件的优势融合到单个模型中,这可能有助于AI在心理健康方面的发展。
BACKGROUND: After the COVID-19 pandemic, the conflict between limited mental health care resources and the rapidly growing number of patients has become more pronounced. It is necessary for psychologists to borrow artificial intelligence (AI)-based methods to analyze patients\' satisfaction with drug treatment for those undergoing mental illness treatment.
OBJECTIVE: Our goal was to construct highly accurate and transferable models for predicting the satisfaction of patients with mental illness with medication by analyzing their own experiences and comments related to medication intake.
METHODS: We extracted 41,851 reviews in 20 categories of disorders related to mental illnesses from a large public data set of 161,297 reviews in 16,950 illness categories. To discover a more optimal structure of the natural language processing models, we proposed the Unified Interchangeable Model Fusion to decompose the state-of-the-art Bidirectional Encoder Representations from Transformers (BERT), support vector machine, and random forest (RF) models into 2 modules, the encoder and the classifier, and then reconstruct fused \"encoder+classifer\" models to accurately evaluate patients\' satisfaction. The fused models were divided into 2 categories in terms of model structures, traditional machine learning-based models and neural network-based models. A new loss function was proposed for those neural network-based models to overcome overfitting and data imbalance. Finally, we fine-tuned the fused models and evaluated their performance comprehensively in terms of F1-score, accuracy, κ coefficient, and training time using 10-fold cross-validation.
RESULTS: Through extensive experiments, the transformer bidirectional encoder+RF model outperformed the state-of-the-art BERT, MentalBERT, and other fused models. It became the optimal model for predicting the patients\' satisfaction with drug treatment. It achieved an average graded F1-score of 0.872, an accuracy of 0.873, and a κ coefficient of 0.806. This model is suitable for high-standard users with sufficient computing resources. Alternatively, it turned out that the word-embedding encoder+RF model showed relatively good performance with an average graded F1-score of 0.801, an accuracy of 0.812, and a κ coefficient of 0.695 but with much less training time. It can be deployed in environments with limited computing resources.
CONCLUSIONS: We analyzed the performance of support vector machine, RF, BERT, MentalBERT, and all fused models and identified the optimal models for different clinical scenarios. The findings can serve as evidence to support that the natural language processing methods can effectively assist psychologists in evaluating the satisfaction of patients with drug treatment programs and provide precise and standardized solutions. The Unified Interchangeable Model Fusion provides a different perspective on building AI models in mental health and has the potential to fuse the strengths of different components of the models into a single model, which may contribute to the development of AI in mental health.