背景:全面的会议摘要使心理健康咨询具有有效的连续性,促进知情的治疗计划。然而,手动总结提出了一个重大挑战,将专家的注意力从核心咨询过程中转移开来。利用自动总结的进步来简化总结过程解决了这个问题,因为这使心理健康专业人员能够访问冗长的治疗会议的简明摘要,从而提高其效率。然而,现有的方法往往忽略了咨询互动中固有的细微差别的复杂性。
目的:本研究通过基于方面的总结,评估了最先进的大型语言模型(LLM)在选择性总结治疗课程的各种组成部分方面的有效性,旨在衡量他们的表现。
方法:我们首先创建了心理健康咨询-成分指导对话摘要,一个基准数据集,由191个咨询会议组成,摘要集中在3个不同的咨询组成部分(也称为咨询方面)。接下来,我们评估了11种最先进的LLM在解决咨询-成分指导总结任务方面的能力.使用标准摘要指标对生成的摘要进行定量评估,并由精神卫生专业人员进行定性验证。
结果:我们的发现证明了特定任务的LLM的卓越性能,例如MentalLlama,米斯特拉尔,和MentalBART使用标准定量指标进行评估,如召回导向的激励评估(ROUGE)-1、ROUGE-2、ROUGE-L、和双向编码器表示从跨咨询组件的各个方面的变形金刚得分。此外,专家评估显示,Mistral在6个参数上取代了MentalLlama和MentalBART:情感态度,负担,伦理,连贯性,机会成本,和感知的有效性。然而,这些模型在机会成本和感知有效性指标的改进空间方面表现出共同的弱点。
结论:虽然专门针对心理健康领域数据进行微调的LLM基于自动评估评分显示出更好的性能,专家评估表明,这些模型在临床应用中还不可靠。在实际实施之前,需要进一步完善和验证。
BACKGROUND: Comprehensive session summaries enable effective continuity in mental health counseling, facilitating informed therapy planning. However, manual summarization presents a significant challenge, diverting experts\' attention from the core counseling process. Leveraging advances in automatic summarization to streamline the summarization process addresses this issue because this enables mental health professionals to access concise summaries of lengthy therapy sessions, thereby increasing their efficiency. However, existing approaches often overlook the nuanced intricacies inherent in counseling interactions.
OBJECTIVE: This
study evaluates the effectiveness of state-of-the-art large language models (LLMs) in selectively summarizing various components of therapy sessions through aspect-based summarization, aiming to benchmark their performance.
METHODS: We first created Mental Health Counseling-Component-Guided Dialogue Summaries, a
benchmarking data set that consists of 191 counseling sessions with summaries focused on 3 distinct counseling components (also known as counseling aspects). Next, we assessed the capabilities of 11 state-of-the-art LLMs in addressing the task of counseling-component-guided summarization. The generated summaries were evaluated quantitatively using standard summarization metrics and verified qualitatively by mental health professionals.
RESULTS: Our findings demonstrated the superior performance of task-specific LLMs such as MentalLlama, Mistral, and MentalBART evaluated using standard quantitative metrics such as Recall-Oriented Understudy for Gisting Evaluation (ROUGE)-1, ROUGE-2, ROUGE-L, and Bidirectional Encoder Representations from Transformers Score across all aspects of the counseling components. Furthermore, expert evaluation revealed that Mistral superseded both MentalLlama and MentalBART across 6 parameters: affective attitude, burden, ethicality, coherence, opportunity costs, and perceived effectiveness. However, these models exhibit a common weakness in terms of room for improvement in the opportunity costs and perceived effectiveness metrics.
CONCLUSIONS: While LLMs fine-tuned specifically on mental health domain data display better performance based on automatic evaluation scores, expert assessments indicate that these models are not yet reliable for clinical application. Further refinement and validation are necessary before their implementation in practice.