背景:包括GPT-4(OpenAI)在内的大型语言模型为医疗保健和定性研究开辟了新途径。传统的定性方法是耗时的,需要专业知识来捕捉细微差别。尽管与传统的自然语言处理相比,大型语言模型已经证明了增强的上下文理解和推理,与人类相比,它们在定性分析方面的表现仍有待探索。
目的:我们评估了GPT-4与人类研究人员在成人获得性埋藏阴茎(AABP)患者访谈中的有效性。
方法:定性数据来自对20例AABP患者的半结构化访谈。人类分析涉及结构化的3阶段过程-初始观察,逐行编码,和共识讨论以完善主题。相比之下,使用GPT-4的人工智能(AI)分析经历了两个阶段:(1)幼稚阶段,其中GPT-4的输出由蒙蔽的审阅者独立评估,以识别主题和次主题,以及(2)比较阶段,将人工智能生成的主题与人类识别的主题进行比较,以评估一致性。我们使用了一般的定性描述方法。
结果:研究人群(N=20)主要包括白人(17/20,85%),已婚(12/20,60%),异性恋(19/20,95%)男性,平均年龄58.8岁,BMI为41.1kg/m2。人类定性分析在95%(19/20)的访谈中确定了“泌尿系统问题”,在75%(15/20)的访谈中确定了GPT-4,在60%(12/20)和35%(7/20)的副主题“喷雾或流”中,分别。“性问题”很突出(19/20,95%的人对16/20,80%的GPT-4),尽管人类确定了更广泛的子主题,包括“性或手淫疼痛”(7/20,35%)和“性或手淫困难”(4/20,20%)。两种分析都同样强调了“心理健康问题”(11/20,55%,两者),虽然人类更频繁地编码“抑郁症”(10/20,50%人类vs4/20,20%GPT-4)。人类经常提到“使用公共厕所的问题”(12/20,60%)影响社交生活,而GPT-4强调“与浪漫关系的斗争”(9/20,45%)。“卫生问题”得到一致认可(14/20,70%的人对13/20,65%的GPT-4)。人类在所有访谈中都将“促成因素”确定为主题。人和GPT-4编码之间存在中等一致性(κ=0.401)。GPT-4分析的可靠性评估显示,包括“身体形象挣扎”在内的主题编码一致,“慢性疼痛”(10/10,100%),和“抑郁症”(9/10,90%)。其他主题如“手术动机”和“体重挑战”被可靠地编码(8/10,80%),而频率较低的主题在多次迭代中被可变地识别。
结论:包括GPT-4在内的大型语言模型可以有效地识别分析定性医疗保健数据的关键主题,与人类分析显示出适度的一致性。虽然人类分析提供了更丰富的次主题多样性,人工智能的一致性表明它是定性研究中的一种补充工具。随着AI的快速发展,未来的研究应该通过分割数据来迭代分析和规避令牌限制,进一步扩大大型语言模型驱动定性分析的广度和深度。
BACKGROUND: Large language models including GPT-4 (OpenAI) have opened new avenues in health care and qualitative research. Traditional qualitative methods are time-consuming and require expertise to capture nuance. Although large language models have demonstrated enhanced contextual understanding and inferencing compared with traditional natural language processing, their performance in qualitative analysis versus that of humans remains unexplored.
OBJECTIVE: We evaluated the effectiveness of GPT-4 versus human researchers in qualitative analysis of interviews with patients with adult-acquired buried penis (AABP).
METHODS: Qualitative data were obtained from semistructured interviews with 20 patients with AABP. Human analysis involved a structured 3-stage process-initial observations, line-by-line coding, and consensus discussions to refine themes. In contrast, artificial intelligence (AI) analysis with GPT-4 underwent two phases: (1) a naïve phase, where GPT-4 outputs were independently evaluated by a blinded reviewer to identify themes and subthemes and (2) a comparison phase, where AI-generated themes were compared with human-identified themes to assess agreement. We used a general qualitative description approach.
RESULTS: The study population (N=20) comprised predominantly White (17/20, 85%), married (12/20, 60%), heterosexual (19/20, 95%) men, with a mean age of 58.8 years and BMI of 41.1 kg/m2. Human qualitative analysis identified \"urinary issues\" in 95% (19/20) and GPT-4 in 75% (15/20) of interviews, with the subtheme \"spray or stream\" noted in 60% (12/20) and 35% (7/20), respectively. \"Sexual issues\" were prominent (19/20, 95% humans vs 16/20, 80% GPT-4), although humans identified a wider range of subthemes, including \"pain with sex or masturbation\" (7/20, 35%) and \"difficulty with sex or masturbation\" (4/20, 20%). Both analyses similarly highlighted \"mental health issues\" (11/20, 55%, both), although humans coded \"depression\" more frequently (10/20, 50% humans vs 4/20, 20% GPT-4). Humans frequently cited \"issues using public restrooms\" (12/20, 60%) as impacting social life, whereas GPT-4 emphasized \"struggles with romantic relationships\" (9/20, 45%). \"Hygiene issues\" were consistently recognized (14/20, 70% humans vs 13/20, 65% GPT-4). Humans uniquely identified \"contributing factors\" as a theme in all interviews. There was moderate agreement between human and GPT-4 coding (κ=0.401). Reliability assessments of GPT-4\'s analyses showed consistent coding for themes including \"body image struggles,\" \"chronic pain\" (10/10, 100%), and \"depression\" (9/10, 90%). Other themes like \"motivation for surgery\" and \"weight challenges\" were reliably coded (8/10, 80%), while less frequent themes were variably identified across multiple iterations.
CONCLUSIONS: Large language models including GPT-4 can effectively identify key themes in analyzing qualitative health care data, showing moderate agreement with human analysis. While human analysis provided a richer diversity of subthemes, the consistency of AI suggests its use as a complementary tool in qualitative research. With AI rapidly advancing, future studies should iterate analyses and circumvent token limitations by segmenting data, furthering the breadth and depth of large language model-driven qualitative analyses.