背景:推荐系统有助于将大范围的项目缩小到较小的,个性化设置。NarraGive是用于心理健康恢复叙述的第一个现场混合推荐系统,根据其内容和叙述者特征(使用基于内容的过滤)以及有益地影响其他类似用户(使用协同过滤)的叙述来推荐叙述。NarraGive被整合到在线叙事体验(NEON)干预中,提供对恢复叙述的NEON集合的访问的Web应用程序。
目的:本研究旨在分析NarraGive中使用的3种推荐系统算法,以使用推荐系统为生活经验叙述提供未来的干预措施。
方法:使用最近发布的评估推荐系统的框架来构建分析,我们通过评估准确性(预测评级与真实评级的接近程度)来比较基于内容的过滤算法和协同过滤算法,精确度(相关的推荐叙述的比例),多样性(推荐的叙述有多多样化),覆盖范围(可以推荐的所有可用叙述的比例),以及跨性别和种族的不公平(算法是否对弱势参与者产生不太准确的预测)。我们使用了两个平行组的所有参与者的数据,NEON干预的候补对照临床试验(NEON试验:N=739;NEON用于其他[例如,非精神病]心理健康问题[NEON-O]试验:N=1023)。两项试验都包括有自我报告精神健康问题的人,他们有和没有使用过法定精神卫生服务。此外,NEON试验参与者在过去5年中经历过自我报告的精神病。我们的评估使用了试验参与者提供的Likert量表叙事等级数据库,以回应经过验证的叙事反馈问题。
结果:来自NEON和NEON-O试验的参与者提供了2288和1896个叙述等级,分别。每个评级的叙述都有3个评级和2个评级的中位数,分别。对于NEON的审判,基于内容的过滤算法在覆盖率方面表现更好;协同过滤算法在准确性方面表现更好,多样性,以及性别和种族的不公平;这两种算法在精度上都没有表现得更好。对于NEON-O的审判,基于内容的过滤算法在任何指标上都没有表现得更好;协同过滤算法在性别和种族的准确性和不公平性上都表现得更好;并且两种算法在精度上都没有表现得更好。多样性,或覆盖范围。
结论:临床人群可能与推荐系统性能相关。推荐系统易受各种不期望的偏差的影响。缓解这些问题的方法包括为推荐系统提供足够的初始数据(以防止过度拟合),确保项目可以在推荐系统之外访问(以防止访问项目和推荐项目之间的反馈循环),并鼓励参与者对与之互动的每个叙述提供反馈(以防止参与者仅在有强烈意见时才提供反馈)。
BACKGROUND: Recommender systems help narrow down a large range of items to a smaller, personalized set. NarraGive is a first-in-field hybrid recommender system for mental health recovery narratives, recommending narratives based on their content and narrator characteristics (using content-based filtering) and on narratives beneficially impacting other similar users (using collaborative filtering). NarraGive is integrated into the Narrative Experiences Online (NEON) intervention, a web application providing access to the NEON Collection of recovery narratives.
OBJECTIVE: This study aims to analyze the 3 recommender system algorithms used in NarraGive to inform future interventions using recommender systems for lived experience narratives.
METHODS: Using a recently published framework for evaluating recommender systems to structure the analysis, we compared the content-based filtering algorithm and collaborative filtering algorithms by evaluating the accuracy (how close the predicted ratings are to the true ratings), precision (the proportion of the recommended narratives that are relevant), diversity (how diverse the recommended narratives are), coverage (the proportion of all available narratives that can be recommended), and unfairness (whether the algorithms produce less accurate predictions for disadvantaged participants) across gender and ethnicity. We used data from all participants in 2 parallel-group, waitlist control clinical trials of the NEON intervention (NEON trial: N=739; NEON for other [eg, nonpsychosis] mental health problems [NEON-O] trial: N=1023). Both trials included people with self-reported mental health problems who had and had not used statutory mental health services. In addition, NEON trial participants had experienced self-reported psychosis in the previous 5 years. Our evaluation used a database of Likert-scale narrative ratings provided by trial participants in response to validated narrative feedback questions.
RESULTS: Participants from the NEON and NEON-O trials provided 2288 and 1896 narrative ratings, respectively. Each rated narrative had a median of 3 ratings and 2 ratings, respectively. For the NEON trial, the content-based filtering algorithm performed better for coverage; the collaborative filtering algorithms performed better for accuracy, diversity, and unfairness across both gender and ethnicity; and neither algorithm performed better for precision. For the NEON-O trial, the content-based filtering algorithm did not perform better on any metric; the collaborative filtering algorithms performed better on accuracy and unfairness across both gender and ethnicity; and neither algorithm performed better for precision, diversity, or coverage.
CONCLUSIONS: Clinical population may be associated with recommender system performance. Recommender systems are susceptible to a wide range of undesirable biases. Approaches to mitigating these include providing enough initial data for the recommender system (to prevent overfitting), ensuring that items can be accessed outside the recommender system (to prevent a feedback loop between accessed items and recommended items), and encouraging participants to provide feedback on every narrative they interact with (to prevent participants from only providing feedback when they have strong opinions).