■实施对自杀的消极态度的机器学习预测可能会改善健康结果。然而,在以前的研究中,各种形式的消极态度没有得到充分考虑,和开发的模型缺乏严格的外部验证。通过分析一个大规模的社交媒体数据集(新浪微博),本文旨在全面涵盖各种形式的消极态度,并开发一个分类模型来预测整体的消极态度,然后从外部验证其在人口和个人层面上的表现。
■下载了938,866条相关关键词的微博,包括在2009年至2014年之间更新的737,849个帖子(2009-2014年数据集),2015年至2020年期间更新了201,017个帖子(2015-2020年数据集)。(1)对于模型开发,基于2009年至2014年数据集随机选择的10,000个帖子,我们进行了基于人的内容分析,以手动确定每个帖子的标签(非否定或否定态度).然后,进行了基于计算机的内容分析,以自动从相同的10,000个帖子中提取心理语言特征。最后,在选定的特征上建立了预测消极态度的分类模型。(2)对于模型验证,在人口层面上,开发的模型在2009年至2014年数据集的剩余727,849个帖子上实施,并通过比较预测结果和人工编码结果之间的消极态度比例进行了外部验证。此外,在个人层面上,对2015年至2020年数据集随机选择的300个帖子进行了类似的分析,通过比较预测结果和实际结果之间每个帖子的标签,对开发的模型进行了外部验证。
■对于模型开发,F1和ROC曲线下面积(AUC)值分别达到0.93和0.97。对于模型验证,在人口层面上,在整个样本中观察到显著差异,但效应大小非常小(χ21=32.35,p<0.001;CramerV=0.007,p<0.001),男性(χ21=9.48,p=0.002;克拉默的V=0.005,p=0.002),和女性(χ21=25.34,p<0.001;克拉默的V=0.009,p<0.001)。此外,在个人层面上,F1和AUC值分别达到0.76和0.74。
■这项研究证明了机器学习预测整体消极态度的有效性和必要性,并确认在将预测模型付诸实践之前,外部验证是必不可少的。
UNASSIGNED: Implementing machine learning prediction of negative attitudes towards suicide may improve health outcomes. However, in previous studies, varied forms of negative attitudes were not adequately considered, and developed models lacked rigorous external validation. By analyzing a large-scale social media dataset (Sina Weibo), this paper aims to fully cover varied forms of negative attitudes and develop a classification model for predicting negative attitudes as a whole, and then to externally validate its performance on population and individual levels.
UNASSIGNED: 938,866 Weibo posts with relevant keywords were downloaded, including 737,849 posts updated between 2009 and 2014 (2009-2014 dataset), and 201,017 posts updated between 2015 and 2020 (2015-2020 dataset). (1) For model development, based on 10,000 randomly selected posts from 2009 to 2014 dataset, a human-based content analysis was performed to manually determine labels of each post (non-negative or negative attitudes). Then, a computer-based content analysis was conducted to automatically extract psycholinguistic features from each of the same 10,000 posts. Finally, a classification model for predicting negative attitudes was developed on selected features. (2) For model validation, on the population level, the developed model was implemented on remaining 727,849 posts from 2009 to 2014 dataset, and was externally validated by comparing proportions of negative attitudes between predicted and human-coded results. Besides, on the individual level, similar analyses were performed on 300 randomly selected posts from 2015 to 2020 dataset, and the developed model was externally validated by comparing labels of each post between predicted and actual results.
UNASSIGNED: For model development, the F1 and area under ROC curve (AUC) values reached 0.93 and 0.97. For model validation, on the population level, significant differences but very small effect sizes were observed for the whole sample (χ 2 1 = 32.35, p < 0.001; Cramer\'s V = 0.007, p < 0.001), men (χ 2 1 = 9.48, p = 0.002; Cramer\'s V = 0.005, p = 0.002), and women (χ 2 1 = 25.34, p < 0.001; Cramer\'s V = 0.009, p < 0.001). Besides, on the individual level, the F1 and AUC values reached 0.76 and 0.74.
UNASSIGNED: This study demonstrates the efficiency and necessity of machine learning prediction of negative attitudes as a whole, and confirms that external validation is essential before implementing prediction models into practice.