PDF(Pubmed)
Abstract：
The usage of crowdsourcing to recruit numerous participants has been recognized as beneficial in the human-computer interaction (HCI) field, such as for designing user interfaces and validating user performance models. In this work, we investigate its effectiveness for evaluating an error-rate prediction model in target pointing tasks. In contrast to models for operational times, a clicking error (i.e., missing a target) occurs by chance at a certain probability, e.g., 5%. Therefore, in traditional laboratory-based experiments, a lot of repetitions are needed to measure the central tendency of error rates. We hypothesize that recruiting many workers would enable us to keep the number of repetitions per worker much smaller. We collected data from 384 workers and found that existing models on operational time and error rate showed good fits (both R 2 > 0.95). A simulation where we changed the number of participants N P and the number of repetitions N repeat showed that the time prediction model was robust against small N P and N repeat, although the error-rate model fitness was considerably degraded. These findings empirically demonstrate a new utility of crowdsourced user experiments for collecting numerous participants, which should be of great use to HCI researchers for their evaluation studies.
摘要：
在人机交互（HCI）领域，使用众包招募众多参与者已被认为是有益的，例如用于设计用户界面和验证用户性能模型。在这项工作中,我们研究了其在目标指向任务中评估错误率预测模型的有效性。与运行时间的模型相反，点击错误(即，错过目标)以一定的概率偶然发生，例如，5%。因此,在传统的实验室实验中，需要大量的重复来衡量错误率的中心趋势。我们假设招募许多工人将使我们能够将每个工人的重复次数减少得多。我们收集了384名工人的数据，发现现有的操作时间和错误率模型显示出良好的拟合（R2均>0.95）。我们改变参与者数量NP和重复次数N重复的模拟表明，时间预测模型对小NP和N重复是稳健的，尽管错误率模型适合度显著下降。这些发现从经验上证明了众包用户实验收集众多参与者的新效用，这对HCI研究人员的评估研究应该很有用。
