背景:专家和目标用户的可用性评估是开发和评估数字解决方案过程中不可或缺的一部分。可用性评估提高了拥有更容易的数字解决方案的可能性,更安全,更有效率,和更愉快的使用。然而,尽管人们普遍认识到可用性评估的重要性,对相关概念和报告标准缺乏研究和共识。
目的:本研究的目的是就用户和专家在规划和报告健康相关数字解决方案可用性评估研究时应考虑的条款和程序达成共识,并提供一份清单,研究人员在进行可用性研究时可以轻松使用该清单。
方法:Delphi研究进行了2轮,其中包括一组在可用性评估方面有经验的国际参与者。在第一轮中,他们被要求对定义发表评论,使用9项李克特量表对预先确定的方法学程序的重要性进行评级,并建议额外的程序。在第二轮中,有经验的参与者被要求根据第1轮结果重新评估每个程序的相关性.当至少有70%或更多有经验的参与者对项目7至9进行评分,而不到15%的参与者对项目1至3进行评分时,就每个项目的相关性达成共识。
结果:共有来自11个不同国家的30名参与者(n=20名女性)参加了Delphi研究,平均年龄为37.2(SD7.7)岁。就所有与可用性评估相关的术语的定义达成了一致(可用性评估主持人,参与者,可用性评估方法,可用性评估技术,任务,可用性评估环境,可用性评估器,和域评估器)。总共确定了38个与可用性评估计划和报告有关的程序(28个与涉及用户的可用性评估有关,10个与涉及专家的可用性评估有关)。23个(82%)涉及用户的可用性评估程序和7个(70%)涉及专家的可用性评估程序的相关性达成了共识。提出了一个清单,可以指导作者设计和报告可用性研究。
结论:本研究提出了一套术语和各自的定义以及一份清单,以指导可用性评估研究的规划和报告。在可用性评估领域朝着更标准化的方法迈出了重要的一步,这可能有助于提高计划和报告可用性研究的质量。未来的研究可以通过完善定义来进一步验证这项研究工作,评估清单的实际适用性,或评估使用此清单是否会导致更高质量的数字解决方案。
Usability evaluation both by experts and target users is an integral part of the process of developing and assessing digital solutions. Usability evaluation improves the probability of having digital solutions that are easier, safer, more efficient, and more pleasant to use. However, despite the widespread recognition of the importance of usability evaluation, there is a lack of research and
consensus on related concepts and reporting standards.
The aim of the study is to generate
consensus on terms and procedures that should be considered when planning and reporting a study on a usability evaluation of health-related digital solutions both by users and experts and provide a checklist that can easily be used by researchers when conducting their usability studies.
A Delphi study with 2 rounds was conducted with a panel of international participants experienced in usability evaluation. In the first round, they were asked to comment on definitions, rate the importance of preidentified methodological procedures using a 9-item Likert scale, and suggest additional procedures. In the second round, experienced participants were asked to reappraise the relevance of each procedure informed by round 1 results.
Consensus on the relevance of each item was defined a priori when at least 70% or more experienced participants scored an item 7 to 9 and less than 15% of participants scored the same item 1 to 3.
A total of 30 participants (n=20 females) from 11 different countries entered the Delphi study with a mean age of 37.2 (SD 7.7) years. Agreement was achieved on the definitions for all usability evaluation-related terms proposed (usability assessment moderator, participant, usability evaluation method, usability evaluation technique, tasks, usability evaluation environment, usability evaluator, and domain evaluator). A total of 38 procedures related to usability evaluation planning and reporting were identified across rounds (28 were related to usability evaluation involving users and 10 related to usability evaluation involving experts).
Consensus on the relevance was achieved for 23 (82%) of the procedures related to usability evaluation involving users and for 7 (70%) of the usability evaluation procedures involving experts. A checklist was proposed that can guide authors when designing and reporting usability studies.
This study proposes a set of terms and respective definitions as well as a checklist to guide the planning and reporting of usability evaluation studies, constituting an important step toward a more standardized approach in the field of usability evaluation that may contribute to enhancing the quality of planning and reporting usability studies. Future studies can contribute to further validating this study work by refining the definitions, assessing the practical
applicability of the checklist, or assessing whether using this checklist results in higher-quality digital solutions.