Abstract：
OBJECTIVE: To develop and validate a novel measure, action entropy, for assessing the cognitive effort associated with electronic health record (EHR)-based work activities.
METHODS: EHR-based audit logs of attending physicians and advanced practice providers (APPs) from four surgical intensive care units in 2019 were included. Neural language models (LMs) were trained and validated separately for attendings\' and APPs\' action sequences. Action entropy was calculated as the cross-entropy associated with the predicted probability of the next action, based on prior actions. To validate the measure, a matched pairs study was conducted to assess the difference in action entropy during known high cognitive effort scenarios, namely, attention switching between patients and to or from the EHR inbox.
RESULTS: Sixty-five clinicians performing 5 904 429 EHR-based audit log actions on 8956 unique patients were included. All attention switching scenarios were associated with a higher action entropy compared to non-switching scenarios (P < .001), except for the from-inbox switching scenario among APPs. The highest difference among attendings was for the from-inbox attention switching: Action entropy was 1.288 (95% CI, 1.256-1.320) standard deviations (SDs) higher for switching compared to non-switching scenarios. For APPs, the highest difference was for the to-inbox switching, where action entropy was 2.354 (95% CI, 2.311-2.397) SDs higher for switching compared to non-switching scenarios.
CONCLUSIONS: We developed a LM-based metric, action entropy, for assessing cognitive burden associated with EHR-based actions. The metric showed discriminant validity and statistical significance when evaluated against known situations of high cognitive effort (ie, attention switching). With additional validation, this metric can potentially be used as a screening tool for assessing behavioral action phenotypes that are associated with higher cognitive burden.
CONCLUSIONS: An LM-based action entropy metric-relying on sequences of EHR actions-offers opportunities for assessing cognitive effort in EHR-based workflows.
METHODS: EHR-based audit logs of attending physicians and advanced practice providers (APPs) from four surgical intensive care units in 2019 were included. Neural language models (LMs) were trained and validated separately for attendings\' and APPs\' action sequences. Action entropy was calculated as the cross-entropy associated with the predicted probability of the next action, based on prior actions. To validate the measure, a matched pairs study was conducted to assess the difference in action entropy during known high cognitive effort scenarios, namely, attention switching between patients and to or from the EHR inbox.
RESULTS: Sixty-five clinicians performing 5 904 429 EHR-based audit log actions on 8956 unique patients were included. All attention switching scenarios were associated with a higher action entropy compared to non-switching scenarios (P < .001), except for the from-inbox switching scenario among APPs. The highest difference among attendings was for the from-inbox attention switching: Action entropy was 1.288 (95% CI, 1.256-1.320) standard deviations (SDs) higher for switching compared to non-switching scenarios. For APPs, the highest difference was for the to-inbox switching, where action entropy was 2.354 (95% CI, 2.311-2.397) SDs higher for switching compared to non-switching scenarios.
CONCLUSIONS: We developed a LM-based metric, action entropy, for assessing cognitive burden associated with EHR-based actions. The metric showed discriminant validity and statistical significance when evaluated against known situations of high cognitive effort (ie, attention switching). With additional validation, this metric can potentially be used as a screening tool for assessing behavioral action phenotypes that are associated with higher cognitive burden.
CONCLUSIONS: An LM-based action entropy metric-relying on sequences of EHR actions-offers opportunities for assessing cognitive effort in EHR-based workflows.
摘要：
目的：为了开发和验证一种新的措施，动作熵，用于评估与基于电子健康记录(EHR)的工作活动相关的认知努力。
方法：包括2019年来自四个外科重症监护病房的主治医师和高级执业提供者（APP）的基于EHR的审核日志。神经语言模型(LM)分别针对出席者和APP动作序列进行了训练和验证。行动熵被计算为与下一个行动的预测概率相关的交叉熵，基于先前的行动。要验证度量，进行了一项配对研究，以评估已知高认知努力情景中动作熵的差异，即,注意患者和EHR收件箱之间的切换。
结果：纳入了65名临床医生，他们对8956名独特患者进行了基于5.904.429EHR的审计日志操作。与非切换场景相比，所有注意力切换场景都与更高的动作熵相关（P<.001），除了APP之间的从收件箱切换场景。出席者之间的最大差异是收件箱注意切换：与非切换场景相比，切换的行动熵高1.288（95％CI，1.256-1.320）标准偏差（SD）。对于APP，最大的区别是收件箱切换，与非切换场景相比，切换的动作熵高2.354(95%CI，2.311-2.397)。
结论：我们开发了一个基于LM的指标，动作熵，用于评估与基于EHR的行为相关的认知负担。当针对高认知努力的已知情况进行评估时，该指标显示出判别效度和统计意义(即，注意切换)。通过额外的验证，该指标可用作筛查工具,用于评估与较高认知负担相关的行为行为表型.
结论：基于LM的行动熵度量-依赖于EHR行动序列-为评估基于EHR的工作流程中的认知努力提供了机会。
方法：包括2019年来自四个外科重症监护病房的主治医师和高级执业提供者（APP）的基于EHR的审核日志。神经语言模型(LM)分别针对出席者和APP动作序列进行了训练和验证。行动熵被计算为与下一个行动的预测概率相关的交叉熵，基于先前的行动。要验证度量，进行了一项配对研究，以评估已知高认知努力情景中动作熵的差异，即,注意患者和EHR收件箱之间的切换。
结果：纳入了65名临床医生，他们对8956名独特患者进行了基于5.904.429EHR的审计日志操作。与非切换场景相比，所有注意力切换场景都与更高的动作熵相关（P<.001），除了APP之间的从收件箱切换场景。出席者之间的最大差异是收件箱注意切换：与非切换场景相比，切换的行动熵高1.288（95％CI，1.256-1.320）标准偏差（SD）。对于APP，最大的区别是收件箱切换，与非切换场景相比，切换的动作熵高2.354(95%CI，2.311-2.397)。
结论：我们开发了一个基于LM的指标，动作熵，用于评估与基于EHR的行为相关的认知负担。当针对高认知努力的已知情况进行评估时，该指标显示出判别效度和统计意义(即，注意切换)。通过额外的验证，该指标可用作筛查工具,用于评估与较高认知负担相关的行为行为表型.
结论：基于LM的行动熵度量-依赖于EHR行动序列-为评估基于EHR的工作流程中的认知努力提供了机会。
