Multisensory object processing improves recognition memory for individual objects, but its impact on memory for neighboring visual objects and scene context remains largely unknown. It is therefore unclear how multisensory processing impacts episodic memory for information outside of the object itself. We conducted three experiments to test the prediction that the presence of audiovisual objects at encoding would improve memory for nearby visual objects, and improve memory for the environmental context in which they occurred. In Experiments 1a and 1b, participants viewed audiovisual-visual object pairs or visual-visual object pairs with a control sound during encoding and were subsequently tested on their memory for each object individually. In Experiment 2, objects were paired with semantically congruent or meaningless control sounds and appeared within four different scene environments. Memory for the environment was tested. Results from Experiments 1a and 1b showed that encoding a congruent audiovisual object did not significantly benefit memory for neighboring visual objects, but Experiment 2 showed that encoding a congruent audiovisual object did improve memory for the environments in which those objects were encoded. These findings suggest that multisensory processing can influence memory beyond the objects themselves and that it has a unique role in episodic memory formation. This is particularly important for understanding how memories and associations are formed in real-world situations, in which objects and their surroundings are often multimodal.

Tinnitus is a prevalent hearing-loss deficit manifested as a phantom (internally generated by the brain) sound that is heard as a high-frequency tone in the majority of afflicted persons. Chronic tinnitus is debilitating, leading to distress, sleep deprivation, anxiety, and even suicidal thoughts. It has been theorized that, in the majority of afflicted persons, tinnitus can be attributed to the loss of high-frequency input from the cochlea to the auditory cortex, known as deafferentation. Deafferentation due to hearing loss develops with aging, which progressively causes tonotopic regions coding for the lost high-frequency coding to synchronize, leading to a phantom high-frequency sound sensation. Approaches to tinnitus remediation that demonstrated promise include inhibitory drugs, the use of tinnitus-specific frequency notching to increase lateral inhibition to the deafferented neurons, and multisensory approaches (auditory-motor and audiovisual) that work by coupling multisensory stimulation to the deafferented neural populations. The goal of this review is to put forward a theoretical framework of a multisensory approach to remedy tinnitus. Our theoretical framework posits that due to vision\'s modulatory (inhibitory, excitatory) influence on the auditory pathway, a prolonged engagement in audiovisual activity, especially during daily discourse, as opposed to auditory-only activity/discourse, can progressively reorganize deafferented neural populations, resulting in the reduced synchrony of the deafferented neurons and a reduction in tinnitus severity over time.

UNASSIGNED: The visual system is not fully mature at birth and continues to develop throughout infancy until it reaches adult levels through late childhood and adolescence. Disruption of vision during this postnatal period and prior to visual maturation results in deficits of visual processing and in turn may affect the development of complementary senses. Studying people who have had one eye surgically removed during early postnatal development is a useful model for understanding timelines of sensory development and the role of binocularity in visual system maturation. Adaptive auditory and audiovisual plasticity following the loss of one eye early in life has been observed for both low-and high-level visual stimuli. Notably, people who have had one eye removed early in life perceive the McGurk effect much less than binocular controls.
UNASSIGNED: The current study investigates whether multisensory compensatory mechanisms are also present in people who had one eye removed late in life, after postnatal visual system maturation, by measuring whether they perceive the McGurk effect compared to binocular controls and people who have had one eye removed early in life.
UNASSIGNED: People who had one eye removed late in life perceived the McGurk effect similar to binocular viewing controls, unlike those who had one eye removed early in life.
UNASSIGNED: This suggests differences in multisensory compensatory mechanisms based on age at surgical eye removal. These results indicate that cross-modal adaptations for the loss of binocularity may be dependent on plasticity levels during cortical development.

Automatic visual word recognition requires not only well-established phonological and orthographic representations but also efficient audio-visual integration of these representations. One possibility is that in developmental dyslexia, inefficient orthographic processing might underlie poor reading. Alternatively, reading deficit could be due to inefficient phonological processing or inefficient integration of orthographic and phonological information. In this event-related potential study, participants with dyslexia (N = 25) and control readers (N = 27) were presented with pairs of words and pseudowords in an implicit same-different task. The reference-target pairs could be identical, or different in the identity or the position of the letters. To test the orthographic-phonological processing, target stimuli were presented in visual-only and audiovisual conditions. Participants with and without dyslexia processed the reference stimuli similarly; however, group differences emerged in the processing of target stimuli, especially in the audiovisual condition where control readers showed greater N1 responses for words than for pseudowords, but readers with dyslexia did not show such difference. Moreover, after 300 ms lexicality effect exhibited a more focused frontal topographic distribution in readers with dyslexia. Our results suggest that in developmental dyslexia, phonological processing and audiovisual processing deficits are more pronounced than orthographic processing deficits.

Communicative abilities in infants with Down syndrome (DS) are delayed in comparison to typically developing (TD) infants, possibly affecting language development in DS. Little is known about what abilities might underlie poor communication and language skills in DS, such as visual attention and audiovisual speech processing. This study compares DS and TD infants between 5-7 months of age in a visual orientation task, and an audiovisual speech processing task, which assessed infants\' looking pattern to communicative cues (i.e., face, eyes, mouth, and waving arm). Concurrent communicative abilities were also assessed via the CSBS-DP checklist. We observed that DS infants orient their visual attention slower than TD infants. Both groups attended more to the eyes than the mouth, and more to the face than the waving arm. However, DS infants attended less to the eyes than the background, and equally to the face and the background, suggesting their difficulty to assess linguistically relevant cues. Finally, communicative skills were related to attention to the eyes in TD, but not in DS infants. Our study showed that early attentional and audiovisual abilities are impaired in DS infants, and might underlie their communication skills, suggesting that early interventions in this population should emphasize those skills.

We examined whether children with developmental language disorder (DLD) differed from their peers with typical development (TD) in the degree to which they encode information about a talker\'s mouth shape into long-term phonemic representations. Children watched a talker\'s face and listened to rare changes from [i] to [u] or the reverse. In the neutral condition, the talker\'s face had a closed mouth throughout. In the audiovisual violation condition, the mouth shape always matched the frequent vowel, even when the rare vowel was played. We hypothesized that in the neutral condition no long-term audiovisual memory traces for speech sounds would be activated. Therefore, the neural response elicited by deviants would reflect only a violation of the observed audiovisual sequence. In contrast, we expected that in the audiovisual violation condition, a long-term memory trace for the speech sound/lip configuration typical for the frequent vowel would be activated. In this condition then, the neural response elicited by rare sound changes would reflect a violation of not only observed audiovisual patterns but also of a long-term memory representation for how a given vowel looks when articulated. Children pressed a response button whenever they saw a talker\'s face assume a silly expression. We found that in children with TD, rare auditory changes produced a significant mismatch negativity (MMN) event-related potential (ERP) component over the posterior scalp in the audiovisual violation condition but not in the neutral condition. In children with DLD, no MMN was present in either condition. Rare vowel changes elicited a significant P3 in both groups and conditions, indicating that all children noticed auditory changes. Our results suggest that children with TD, but not children with DLD, incorporate visual information into long-term phonemic representations and detect violations in audiovisual phonemic congruency even when they perform a task that is unrelated to phonemic processing.

Blindness caused by early vision loss results in complete visual deprivation and subsequent changes in the use of the remaining intact senses. We have also observed adaptive plasticity in the case of partial visual deprivation. The removal of one eye, through unilateral eye enucleation, results in partial visual deprivation and is a unique model for examining the consequences of the loss of binocularity. Partial deprivation of the visual system from the loss of one eye early in life results in behavioral and structural changes in the remaining senses, namely auditory and audiovisual systems. In the current study we use functional neuroimaging data to relate function and behavior of the audiovisual system in this rare patient group compared to controls viewing binocularly or with one eye patched. In Experiment 1, a whole brain analysis compared common regions of cortical activation between groups, for auditory, visual and audiovisual stimuli. People with one eye demonstrated a trend for increased activation for low-level audiovisual stimuli compared to patched viewing controls but did not differ from binocular viewing controls. In Experiment 2, a region of interest (ROI) analysis for auditory, visual, audiovisual and illusory McGurk stimuli revealed that people with one eye had an increased trend for left hemisphere audiovisual activation for McGurk stimuli compared to binocular viewing controls. This aligns with current behavioral analysis and previous research showing reduced McGurk Effect in people with one eye. Furthermore, there is no evidence of a correlation between behavioral performance on the McGurk Effect task and functional activation. Together with previous behavioral work, these functional data contribute to the broader understanding of cross-sensory effects of early sensory deprivation from eye enucleation. Overall, these results contribute to a better understanding of the sensory deficits experienced by people with one eye, as well as, the relationship between behavior, structure and function in order to better predict the outcome of early partial visual deafferentation.

Observing motion in one modality can influence the perceived direction of motion in a second modality (dynamic capture). For example observing a square moving in depth can influence the perception of a sound to increase in loudness. The current study investigates whether people who have lost one eye are susceptible to audiovisual dynamic capture in the depth plane similar to binocular and eye-patched viewing control participants. Partial deprivation of the visual system from the loss of one eye early in life results in changes in the remaining intact senses such as hearing. Linearly expanding or contracting discs were paired with increasing or decreasing tones and participants were asked to indicate the direction of the auditory stimulus. Magnitude of dynamic visual capture was measured in people with one eye compared to eye-patched and binocular viewing controls. People with one eye have the same susceptibility to dynamic visual capture as controls, where they perceived the direction of the auditory signal to be moving in the direction of the incongruent visual signal, despite previously showing a lack of visual dominance for audiovisual cues. This behaviour may be the result of directing attention to the visual modality, their partially deficient sense, in order to gain important information about approaching and receding stimuli which in the former case could be life-threatening. These results contribute to the growing body of research showing that people with one eye display unique accommodations with respect to audiovisual processing that are likely adaptive in each unique sensory situation.

Every day we integrate meaningful information coming from different sensory modalities, and previous work has debated whether conceptual knowledge is represented in modality-specific neural stores specialized for specific types of information, and/or in an amodal, shared system. In the current study, we investigated semantic processing through a cross-modal paradigm which asked whether auditory semantic processing could be modulated by the constraints of context built up across a meaningful visual narrative sequence. We recorded event-related brain potentials (ERPs) to auditory words and sounds associated to events in visual narratives-i.e., seeing images of someone spitting while hearing either a word (Spitting!) or a sound (the sound of spitting)-which were either semantically congruent or incongruent with the climactic visual event. Our results showed that both incongruent sounds and words evoked an N400 effect, however, the distribution of the N400 effect to words (centro-parietal) differed from that of sounds (frontal). In addition, words had an earlier latency N400 than sounds. Despite these differences, a sustained late frontal negativity followed the N400s and did not differ between modalities. These results support the idea that semantic memory balances a distributed cortical network accessible from multiple modalities, yet also engages amodal processing insensitive to specific modalities.

Integrating vision and hearing is an important way in which we process our rich sensory environment. Partial deprivation of the visual system from the loss of one eye early in life results in adaptive changes in the remaining senses (e.g., Hoover et al. in Exp Brain Res 216:565-74, 2012). The current study investigates whether losing one eye early in life impacts the temporal window in which audiovisual events are integrated and whether there is vulnerability to the sound-induced flash illusion. In Experiment 1, we measured the temporal binding window with a simultaneity judgement task where low-level auditory and visual stimuli were presented at different stimulus onset asynchronies. People with one eye did not differ in the width of their temporal binding window, but they took longer to make judgements compared to binocular viewing controls. In Experiment 2, we measured how many light flashes were perceived when a single flash was paired with multiple auditory beeps in close succession (sound induced flash illusion). Unlike controls, who perceived multiple light flashes with two, three or four beeps, people with one eye were not susceptible to the sound-induced flash illusion. In addition, they took no longer to respond compared to both binocular and monocular (eye-patched) viewing controls. Taken together, these results suggest that the lack of susceptibility to the sound-induced flash illusion in people with one eye cannot be accounted for by the width of the temporal binding window. These results provide evidence for adaptations in audiovisual integration due to the reduction of visual input from the loss of one eye early in life.