安全关键域通常采用遵循顺序决策设置的自主代理,由此,代理遵循策略来指示在每个步骤处的适当动作。AI从业者经常使用强化学习算法来允许代理找到最佳策略。然而,顺序系统通常缺乏明确和直接的错误行为迹象,其后果只有事后才可见,使人类难以理解系统故障。在强化学习中,这被称为信用分配问题。为了有效地与自治系统协作,特别是在安全关键的环境中,解释应该使用户能够更好地理解代理的策略并预测系统行为,以便用户认识到潜在的故障,并且可以诊断和减轻这些故障。然而,人类是多样化的,具有先天的偏见或偏好,这可能会增强或削弱顺序代理的政策解释的效用。因此,在本文中,我们设计并进行了人体实验,以确定影响感知可用性的因素,并在顺序设置中对强化学习代理进行政策解释的客观有用性。我们的研究有两个因素:向用户显示的政策解释的方式(树,文本,修改后的文本,和程序)和代理人的“第一印象”,即,用户是否在介绍性校准视频中看到代理成功或失败。我们的发现描述了偏好-性能权衡,其中参与者认为基于语言的政策解释更有用;然而,当参与者以决策树的形式提供解释时,他们能够更好地客观预测代理人的行为。我们的结果表明,用户特定的因素,如计算机科学经验(p<0.05),和情境因素,例如观看代理崩溃(p<0.05),能显著影响解释的感知和有用性。这项研究提供了关键的见解,以缓解有关不完全合规和依赖的普遍问题,在安全关键的环境中指数级地更有害,为XAI开发人员提供了未来政策解释工作的前进道路。
Safefy-critical domains often employ autonomous agents which follow a sequential decision-making setup, whereby the agent follows a policy to dictate the appropriate action at each step. AI-practitioners often employ reinforcement learning algorithms to allow an agent to find the best policy. However, sequential systems often lack clear and immediate signs of wrong actions, with consequences visible only in hindsight, making it difficult to humans to understand system failure. In reinforcement learning, this is referred to as the credit assignment problem. To effectively collaborate with an autonomous system, particularly in a safety-critical setting, explanations should enable a user to better understand the policy of the agent and predict system behavior so that users are cognizant of potential failures and these failures can be diagnosed and mitigated. However, humans are diverse and have innate biases or preferences which may enhance or impair the utility of a policy explanation of a sequential agent. Therefore, in this paper, we designed and conducted human-subjects experiment to identify the factors which influence the perceived usability with the objective usefulness of policy explanations for reinforcement learning agents in a sequential setting. Our study had two factors: the modality of policy explanation shown to the user (Tree, Text, Modified Text, and Programs) and the \"first impression\" of the agent, i.e., whether the user saw the agent succeed or fail in the introductory calibration video. Our findings characterize a preference-performance tradeoff wherein participants perceived language-based policy explanations to be significantly more useable; however, participants were better able to objectively predict the agent\'s behavior when provided an explanation in the form of a decision tree. Our results demonstrate that user-specific factors, such as computer science experience (p < 0.05), and situational factors, such as watching agent crash (p < 0.05), can significantly impact the perception and usefulness of the explanation. This research provides key insights to alleviate prevalent issues regarding innapropriate compliance and reliance, which are exponentially more detrimental in safety-critical settings, providing a path forward for XAI developers for future work on policy-explanations.