The phenomenon of mind wandering (MW), as a family of experiences related to internally directed cognition, heavily influences vigilance evolution. In particular, humans in teleoperations monitoring partially automated fleet before assuming manual control whenever necessary may see their attention drift due to internal sources; as such, it could play an important role in the emergence of out-of-the-loop (OOTL) situations and associated performance problems. To follow, quantify, and mitigate this phenomenon, electroencephalogram (EEG) systems already demonstrated robust results. As MW creates an attentional decoupling, both ERPs and brain oscillations are impacted. However, the factors influencing these markers in complex environments are still not fully understood. In this paper, we specifically addressed the possibility of gradual emergence of attentional decoupling and the differences created by the sensory modality used to convey targets. Eighteen participants were asked to (1) supervise an automated drone performing an obstacle avoidance task (visual task) and (2) respond to infrequent beeps as fast as possible (auditory task). We measured event-related potentials and alpha waves through EEG. We also added a 40-Hz amplitude modulated brown noise to evoke steady-state auditory response (ASSR). Reported MW episodes were categorized between task-related and task-unrelated episodes. We found that N1 ERP component elicited by beeps had lower amplitude during task-unrelated MW, whereas P3 component had higher amplitude during task-related MW, compared with other attentional states. Focusing on parieto-occipital regions, alpha-wave activity was higher during task-unrelated MW compared with others. These results support the decoupling hypothesis for task-unrelated MW but not task-related MW, highlighting possible variations in the \"depth\" of decoupling depending on MW episodes. Finally, we found no influence of attentional states on ASSR amplitude. We discuss possible reasons explaining why. Results underline both the ability of EEG to track and study MW in laboratory tasks mimicking ecological environments, as well as the complex influence of perceptual decoupling on operators\' behavior and, in particular, EEG measures.

Communication is the process by which one emitter conveys information to one or several receivers to induce a response (behavioral or physiological) by the receiver. Communication plays a major role in various biological functions and may involve signals and cues from different sensory modalities. Traditionally, investigations of animal communication focused on a single sensory modality, yet communication is often multimodal. As these different processes may be quite complex and therefore difficult to disentangle, one approach is to first study each sensorial modality separately. With this refined understanding of individual senses, revealing how they interact becomes possible as the characteristics and properties of each modality can be accounted for, making a multimodal approach feasible. Using this framework, researchers undertook systematic, experimental investigations on mother-pup recognition processes in a colonial pinniped species, the Australian sea lion Neophoca cinerea. The research first assessed the abilities of mothers and pups to identify each other by their voice using playback experiments. Second, they assessed whether visual cues are used by both mothers and pups to distinguish them from conspecifics, and/or whether females discriminate the odor of their filial pup from those from non-filial pups. Finally, to understand if the information transmitted by different sensory modalities is analyzed synergistically or if there is a hierarchy among the sensory modalities, experiments were performed involving different sensory cues simultaneously. These findings are discussed with regards to the active space of each sensory cue, and of the potential enhancements that may arise by assessing information from different modalities.

Little research has been conducted on dogs\' (Canis familiaris) ability to integrate information obtained through different sensory modalities during object discrimination and recognition tasks. Such a process would indicate the formation of multisensory mental representations. In Experiment 1, we tested the ability of 3 Gifted Word Learner (GWL) dogs that can rapidly learn the verbal labels of toys, and 10 Typical (T) dogs to discriminate an object recently associated with a reward, from distractor objects, under light and dark conditions. While the success rate did not differ between the two groups and conditions, a detailed behavioral analysis showed that all dogs searched for longer and sniffed more in the dark. This suggests that, when possible, dogs relied mostly on vision, and switched to using only other sensory modalities, including olfaction, when searching in the dark. In Experiment 2, we investigated whether, for the GWL dogs (N = 4), hearing the object verbal labels activates a memory of a multisensory mental representation. We did so by testing their ability to recognize objects based on their names under dark and light conditions. Their success rate did not differ between the two conditions, whereas the dogs\' search behavior did, indicating a flexible use of different sensory modalities. Little is known about the cognitive mechanisms involved in the ability of GWL dogs to recognize labeled objects. These findings supply the first evidence that for GWL dogs, verbal labels evoke a multisensory mental representation of the objects.

Social perception is a multimodal process involving vision and audition as central input sources for human social cognitive processes. However, it remains unclear how profoundly deaf people assess others in the context of mating and social interaction. The current study explored the relative importance of different sensory modalities (vision, smell, and touch) in assessments of opposite- and same-sex strangers. We focused on potential sensory compensation processes in mate selection (i.e., increased importance of the intact senses in forming impressions of an opposite-sex stranger as a potential partner). A total of 74 deaf individuals and 100 normally hearing controls were included in the study sample. We found diminished importance of vision and smell in deaf participants compared with controls for opposite- and same-sex strangers, and increased importance of touch for the assessment of same-sex strangers. The results suggested that deaf people rely less on visual and olfactory cues in mating and social assessments, highlighting a possible role of sign language in shaping interpersonal tactile experience in non-romantic relationships.

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

Coping is a survival mechanism of living organisms. It is not merely reactive, but also involves making sense of the environment by rendering sensory information into percepts that have meaning in the context of an organism\'s cognitions. Music listening, on the other hand, is a complex task that embraces sensory, physiological, behavioral, and cognitive levels of processing. Being both a dispositional process that relies on our evolutionary toolkit for coping with the world and a more elaborated skill for sense-making, it goes beyond primitive action-reaction couplings by the introduction of higher-order intermediary variables between sensory input and effector reactions. Consideration of music-listening from the perspective of coping treats music as a sound environment and listening as a process that involves exploration of this environment as well as interactions with the sounds. Several issues are considered in this regard such as the conception of music as a possible stressor, the role of adaptive listening, the relation between coping and reward, the importance of self-regulation strategies in the selection of music, and the instrumental meaning of music in the sense that it can be used to modify the internal and external environment of the listener.

In this 3-experiment study, the Weber fractions in the 300-ms and 900-ms duration ranges are obtained with 9 types of empty intervals resulting from the combinations of three types of signals for marking the beginning and end of the signals: auditory (A), visual (V), or tactile (T). There were three types of intramodal intervals (AA, TT, and VV) and 6 types of intermodal intervals (AT, AV, VA, VT, TA, and TV). The second marker is always the same during Experiments 1 (A), 2 (V), and 3 (T). With an uncertainty strategy where the first marker is 1 of 2 sensory signals being presented randomly from trial to trial, the study provides direct comparisons of the perceived length of the different marker-type intervals. The results reveal that the Weber fraction is nearly constant in the three types of intramodal intervals, but is clearly lower at 900 ms than at 300 ms in intermodal conditions. In several cases, the intramodal intervals are perceived as shorter than intermodal intervals, which is interpreted as an effect of the efficiency in detecting the second marker of an intramodal interval. There were no significant differences between the TA and VA intervals (Experiment 1) and between the AV and TV intervals (Experiment 2), but in Experiment 3, the AT intervals were perceived as longer than the VT intervals. The results are interpreted in terms of the generalized form of Weber\'s law, using the properties of the signals for explaining the additional nontemporal noise observed in the intermodal conditions.

Sensory substitution devices were developed in the context of perceptual rehabilitation and they aim at compensating one or several functions of a deficient sensory modality by converting stimuli that are normally accessed through this deficient sensory modality into stimuli accessible by another sensory modality. For instance, they can convert visual information into sounds or tactile stimuli. In this article, we review those studies that investigated the individual differences at the behavioural, neural, and phenomenological levels when using a sensory substitution device. We highlight how taking into account individual differences has consequences for the optimization and learning of sensory substitution devices. We also discuss the extent to which these studies allow a better understanding of the experience with sensory substitution devices, and in particular how the resulting experience is not akin to a single sensory modality. Rather, it should be conceived as a multisensory experience, involving both perceptual and cognitive processes, and emerging on each user\'s pre-existing sensory and cognitive capacities.

The successful promotion of vegetable consumption by children requires a deep understanding of children\'s vegetable preferences as well as the factors shaping them throughout childhood. This study analyzed children vegetable liking in four different age ranges (5-6, 7-8, 9-10 and 11-12 years old) in Chile, China and the United States. Three hundred and eighty-four children completed this study. All participants tasted and rated 14 different vegetables for liking and described the samples using Check-All-That-Apply (CATA). We found significant differences in degree of overall liking among children from the three countries (p < 0.001). Specifically, children in China gave higher overall liking scores than children in the US, and in the US higher than in Chile. Child age and gender did not influence children\'s vegetable overall liking across the three countries. Across all countries and age groups, liking of taste and texture were the best predictors of children overall liking. The penalty analysis of CATA selections by children showed that the mean impact of the attributes that children used to describe the samples on their liking varied among countries, with the descriptors having the least impact on liking for Chinese children.

Performance in temporal difference threshold and estimation tasks is markedly less accurate for visual than for auditory intervals. In addition, thresholds and estimates are likewise less accurate for empty than for filled intervals. In scalar timing theory, these differences have been explained as alterations in pacemaker rate, which is faster for auditory and filled intervals than for visual and empty intervals. We tested this explanation according to three research aims. First, we replicated the threshold and estimation tasks of Jones, Poliakoff, and Wells (Quarterly Journal of Experimental Psychology, 62, 2171-2186, 2009) and found the well-documented greater precision for auditory than visual intervals, and for filled than for empty intervals. Second, we considered inter-individual differences in these classic effects and found that up to 27% of participants exhibited opposite patterns. Finally, we examined intra-individual differences to investigate (i) whether thresholds and estimates correlate within each stimulus condition and (ii) whether the stimulus condition in which a participants\' pacemaker rate was highest was the same in both tasks. Here we found that if pacemaker rate is indeed a driving factor for thresholds and estimates, its effect may be greater for empty intervals, where the two tasks correlate, than for filled intervals, where they do not. In addition, it was more common for participants to perform best in different modalities in each task, though this was not true for ordinal intra-individual differences in the filled-duration illusion. Overall, this research presents several findings inconsistent with the pacemaker rate explanation.