PDF(Pubmed)
Abstract：
UNASSIGNED: Temporal co-ordination between speech and gestures has been thoroughly studied in natural production. In most cases gesture strokes precede or coincide with the stressed syllable in words that they are semantically associated with.
UNASSIGNED: To understand whether processing of speech and gestures is attuned to such temporal coordination, we investigated the effect of delaying, preposing or eliminating individual gestures on the memory for words in an experimental study in which 83 participants watched video sequences of naturalistic 3D-animated speakers generated based on motion capture data. A target word in the sequence appeared (a) with a gesture presented in its original position synchronized with speech, (b) temporally shifted 500 ms before or (c) after the original position, or (d) with the gesture eliminated. Participants were asked to retell the videos in a free recall task. The strength of recall was operationalized as the inclusion of the target word in the free recall.
UNASSIGNED: Both eliminated and delayed gesture strokes resulted in reduced recall rates compared to synchronized strokes, whereas there was no difference between advanced (preposed) and synchronized strokes. An item-level analysis also showed that the greater the interval between the onsets of delayed strokes and stressed syllables in target words, the greater the negative effect was on recall.
UNASSIGNED: These results indicate that speech-gesture synchrony affects memory for speech, and that temporal patterns that are common in production lead to the best recall. Importantly, the study also showcases a procedure for using motion capture-based 3D-animated speakers to create an experimental paradigm for the study of speech-gesture comprehension.
UNASSIGNED: To understand whether processing of speech and gestures is attuned to such temporal coordination, we investigated the effect of delaying, preposing or eliminating individual gestures on the memory for words in an experimental study in which 83 participants watched video sequences of naturalistic 3D-animated speakers generated based on motion capture data. A target word in the sequence appeared (a) with a gesture presented in its original position synchronized with speech, (b) temporally shifted 500 ms before or (c) after the original position, or (d) with the gesture eliminated. Participants were asked to retell the videos in a free recall task. The strength of recall was operationalized as the inclusion of the target word in the free recall.
UNASSIGNED: Both eliminated and delayed gesture strokes resulted in reduced recall rates compared to synchronized strokes, whereas there was no difference between advanced (preposed) and synchronized strokes. An item-level analysis also showed that the greater the interval between the onsets of delayed strokes and stressed syllables in target words, the greater the negative effect was on recall.
UNASSIGNED: These results indicate that speech-gesture synchrony affects memory for speech, and that temporal patterns that are common in production lead to the best recall. Importantly, the study also showcases a procedure for using motion capture-based 3D-animated speakers to create an experimental paradigm for the study of speech-gesture comprehension.
摘要：
■在自然生产中已经对语音和手势之间的时间协调进行了深入研究。在大多数情况下，手势笔划在语义关联的单词中的重读音节之前或与重读音节重合。
■要了解语音和手势的处理是否与这种时间协调相协调，我们调查了延迟的影响，在一项实验研究中，83名参与者观看了基于动作捕捉数据生成的自然3D动画扬声器的视频序列，在该实验研究中，在单词的记忆中预先设置或消除单个手势。序列中的目标单词出现（a），在其原始位置呈现与语音同步的手势，（b）在原始位置之前或（c）之后在时间上偏移500ms，或(d)消除手势。参与者被要求在免费召回任务中复述视频。召回的强度被实施为将目标词包含在自由召回中。
■与同步笔划相比，消除和延迟的笔划均导致召回率降低，而高级（前置）和同步笔划之间没有区别。项目级分析还表明，目标词中延迟笔画和重读音节的发作间隔越大，对召回的负面影响越大。
■这些结果表明，语音-手势同步会影响语音记忆，以及生产中常见的时间模式导致最佳召回。重要的是,该研究还展示了使用基于动作捕捉的3D动画扬声器来创建语音手势理解研究的实验范例的过程。
■要了解语音和手势的处理是否与这种时间协调相协调，我们调查了延迟的影响，在一项实验研究中，83名参与者观看了基于动作捕捉数据生成的自然3D动画扬声器的视频序列，在该实验研究中，在单词的记忆中预先设置或消除单个手势。序列中的目标单词出现（a），在其原始位置呈现与语音同步的手势，（b）在原始位置之前或（c）之后在时间上偏移500ms，或(d)消除手势。参与者被要求在免费召回任务中复述视频。召回的强度被实施为将目标词包含在自由召回中。
■与同步笔划相比，消除和延迟的笔划均导致召回率降低，而高级（前置）和同步笔划之间没有区别。项目级分析还表明，目标词中延迟笔画和重读音节的发作间隔越大，对召回的负面影响越大。
■这些结果表明，语音-手势同步会影响语音记忆，以及生产中常见的时间模式导致最佳召回。重要的是,该研究还展示了使用基于动作捕捉的3D动画扬声器来创建语音手势理解研究的实验范例的过程。
