Understanding the neural underpinning of human gait and balance is one of the most pertinent challenges for 21st-century translational neuroscience due to the profound impact that falls and mobility disturbances have ...Understanding the neural underpinning of human gait and balance is one of the most pertinent challenges for 21st-century translational neuroscience due to the profound impact that falls and mobility disturbances have on our aging population.Posture and gait control does not happen automatically,as previously believed,but rather requires continuous involvement of central nervous mechanisms.To effectively exert control over the body,the brain must integrate multiple streams of sensory information,including visual,vestibular,and somatosensory signals.The mechanisms which underpin the integration of these multisensory signals are the principal topic of the present work.Existing multisensory integration theories focus on how failure of cognitive processes thought to be involved in multisensory integration leads to falls in older adults.Insufficient emphasis,however,has been placed on specific contributions of individual sensory modalities to multisensory integration processes and cross-modal interactions that occur between the sensory modalities in relation to gait and balance.In the present work,we review the contributions of somatosensory,visual,and vestibular modalities,along with their multisensory intersections to gait and balance in older adults and patients with Parkinson’s disease.We also review evidence of vestibular contributions to multisensory temporal binding windows,previously shown to be highly pertinent to fall risk in older adults.Lastly,we relate multisensory vestibular mechanisms to potential neural substrates,both at the level of neurobiology(concerning positron emission tomography imaging)and at the level of electrophysiology(concerning electroencephalography).We hope that this integrative review,drawing influence across multiple subdisciplines of neuroscience,paves the way for novel research directions and therapeutic neuromodulatory approaches,to improve the lives of older adults and patients with neurodegenerative diseases.展开更多
Objective To examine the effect of multisensory exercise on balance disorders.Methods PubMed,Scopus and Web of Science were searched to identify eligible studies published before January 1,2020.Eligible studies includ...Objective To examine the effect of multisensory exercise on balance disorders.Methods PubMed,Scopus and Web of Science were searched to identify eligible studies published before January 1,2020.Eligible studies included randomized control trials(RCTs),non-randomized studies,case-control studies,and cohort studies.The methodological quality of the included studies was evaluated using JBI Critical Appraisal Checklists for RCTs and for Quasi-Experimental Studies by two researchers independently.A narrative synthesis of intervention characteristics and health-related outcomes was performed.Results A total of 11 non-randomized studies and 9 RCTs were eligible,including 667 participants.The results supported our assumption that multisensory exercise improved balance in people with balance disorders.All of the 20 studies were believed to be of high or moderate quality.Conclusion Our study confirmed that multisensory exercise was effective in improving balance in people with balance disorders.Multisensory exercises could lower the risk of fall and enhance confidence level to improve the quality of life.Further research is needed to investigate the optimal strategy of multisensory exercises and explore the underlying neural and molecular mechanisms of balance improvement brought by multisensory exercises.展开更多
Progress in understanding multisensory integration in human have suggested researchers that the integration may result into the enhancement or depression of incoming signals.It is evident based on different psychologi...Progress in understanding multisensory integration in human have suggested researchers that the integration may result into the enhancement or depression of incoming signals.It is evident based on different psychological and behavioral experiments that stimuli coming from different perceptual modalities at the same time or from the same place,the signal having more strength under the influence of emotions effects the response accordingly.Current research inmultisensory integration has not studied the effect of emotions despite its significance and natural influence in multisensory enhancement or depression.Therefore,there is a need to integrate the emotional state of the agent with incoming stimuli for signal enhancement or depression.In this study,two different neural network-based learning algorithms have been employed to learn the impact of emotions on signal enhancement or depression.It was observed that the performance of a proposed system for multisensory integration increases when emotion features were present during enhancement or depression of multisensory signals.展开更多
Energy consumption of the EU has a crucial environmental impact;several efforts are nowadays thus directed into massively reducing energy consumption by envelope improvement,system efficiency and smart control.On the ...Energy consumption of the EU has a crucial environmental impact;several efforts are nowadays thus directed into massively reducing energy consumption by envelope improvement,system efficiency and smart control.On the other hand,the indoor thermal and lighting conditions significantly influence users’wellbeing and productivity,which is especially important when dealing with educational and working facilities.Strategies to enhance system efficiency are focused on design and construction aspects.These strategies ease to promote a powerful approach which is needed when focusing on existing buildings in need of retrofit measures.When dealing with new or refurbished buildings,energy saving has a further step to achieve.In the last years,research trends moved towards an increasing inclusion of human factors in energy evaluation.This allows to account for the occupancy variability in the energy analyses,considering how to bridge the performance gap between predictive models and actual consumptions due to indoor thermal settings.In empty buildings energy consumption is huge and economic efforts are wasted,due to unconscious energy-wasting behaviors.Previous studies with Interactive Virtual Environments confirmed that indoor environments illuminated by different color lights lead to the perception of different levels of thermal comfort.The results of the present paper replicate previous evidence collected in real conditions,suggesting that Virtual Reality is a valid and reliable tool to assess thermal comfort more quickly and cheaply.This paper provides a further perspective on this topic,including also the use of different fragrances to understand how the indoor environment could be enhanced and manipulated to increase wellbeing,thermal perception and finally energy saving.展开更多
Background:The concept of stochastic facilitation suggests that the addition of precise amounts of white noise can improve the perceptibility of a stimulus of weak amplitude.We know from previous research that tactile...Background:The concept of stochastic facilitation suggests that the addition of precise amounts of white noise can improve the perceptibility of a stimulus of weak amplitude.We know from previous research that tactile and auditory noise can facilitate visual perception,respectively.Here we wanted to see if the effects of stochastic facilitation generalise to a reaction time paradigm,and if reaction times are correlated with tactile thresholds.We know that when multiple sensory systems are stimulated simultaneously,reaction times are faster than either stimulus alone,and also faster than the sum of reaction times(known as the race model).Methods:Five participants were re-tested in five blocks each of which contained a different background noise levels,randomly ordered across sessions.At each noise level,they performed a tactile threshold detection task and a tactile reaction time task.Results:Both tactile threshold and tactile reaction times were significantly affected by the background white noise.While the preferred amplitude for the white noise was different for every participant,the average lowest threshold was obtained with white noise presented binaurally at 70 db.The reaction times were analysed by fitting an ex-Gaussian,the sum of a Gaussian function and an exponential decay function.The white noise significantly affected the exponential parameter(tau)in a way that is compatible with the facilitation of thresholds.Conclusions:We therefore conclude that multisensory reaction time facilitation can,at least in part,be explained by stochastic facilitation of the neural signals.展开更多
Multisensory lab based in Peking University,has carried out basic studies in multisensory space and time processing,intersensory binding and haptic/tactile perception.We exploited a typical paradigm of multisensory il...Multisensory lab based in Peking University,has carried out basic studies in multisensory space and time processing,intersensory binding and haptic/tactile perception.We exploited a typical paradigm of multisensory illusion-temporal ventriloquist effect and applied it in a wide range of multisensory interactions(mainly focused on temporal processing).In this work,we summarized how the tactile stimuli were exploited to compose tactile cues and as tactile apparent motion to interface with other sensory stimuli(visual and auditory stimuli)to examine the underlying perceptual organization in a multisensory context.Moreover,we introduced two examples of wearable haptic/tactile perception in our lab,by using two customized tactile devices and discussed the potential applications in this field.展开更多
Accurate self-motion perception,which is critical for organisms to survive,is a process involving multiple sensory cues.The two most powerful cues are visual(optic flow)and vestibular(inertial motion).Psychophysical s...Accurate self-motion perception,which is critical for organisms to survive,is a process involving multiple sensory cues.The two most powerful cues are visual(optic flow)and vestibular(inertial motion).Psychophysical studies have indicated that humans and nonhuman primates integrate the two cues to improve the estimation of self-motion direction,often in a statistically Bayesian-optimal way.In the last decade,single-unit recordings in awake,behaving animals have provided valuable neurophysiological data with a high spatial and temporal resolution,giving insight into possible neural mechanisms underlying multisensory self-motion perception.Here,we review these findings,along with new evidence from the most recent studies focusing on the temporal dynamics of signals in different modalities.We show that,in light of new data,conventional thoughts about the cortical mechanisms underlying visuo-vestibular integration for linear self-motion are challenged.We propose that different temporal component signals may mediate different functions,a possibility that requires future studies.展开更多
A visual and tactile multisensory integrated system is essential for human walking due to the demand for real-time interactions between perception and action.Here,a piezoresistor and MoS_(2)field effect transistor are...A visual and tactile multisensory integrated system is essential for human walking due to the demand for real-time interactions between perception and action.Here,a piezoresistor and MoS_(2)field effect transistor are combined to construct an artificial integration nervous system to simulate perception and synaptic plasticity.The key characteristics of synaptic plasticity are successfully demonstrated by individual pressure signals,individual optical signals,the synergy of optical and pressure signals,which are based on the electron trapping–detrapping mechanism at the MoS_(2)/SiO_(2)interface.We demonstrate that perception under synergy is stronger than perception under optical or pressure signal alone,which is similar to a biological system.Moreover,various distinguishable motion scenarios(combination of the following conditions:external lighting environment of day or night,flat or rough road,movement state of walking or running)are simulated and verified by adjusting the amplitude and frequency of the optical and pressure signals.展开更多
Multi-sensory neuromorphic devices(MND)have broad potential in overcoming the structural bottleneck of von Neumann in the era of big data.However,the current multisensory artificial neuromorphic system is mainly based...Multi-sensory neuromorphic devices(MND)have broad potential in overcoming the structural bottleneck of von Neumann in the era of big data.However,the current multisensory artificial neuromorphic system is mainly based on unitary nonvolatile memory or volatile synaptic devices without intrinsic thermal sensitivity,which limits the range of biological multisensory perception and the flexibility and computational efficiency of the neural morphological computing system.Here,a temperature-dependent memory/synaptic hybrid artificial neuromorphic device based on floating gate phototransistors(FGT)is fabricated.The CsPbBr_(3)/TiO_(2)core–shell nanocrystals(NCs)prepared by in-situ pre-protection low-temperature solvothermal method were used as the photosensitive layer.The device exhibits remarkable multi-level visual memory with a large memory window of 59.6 V at room temperature.Surprisingly,when the temperature varies from 20 to 120℃back and forth,the device can switch between nonvolatile memory and volatile synaptic device with reconfigurable and reversible behaviors,which contributes to the efficient visual/thermal fusion perception.This work expands the sensory range of multisensory devices and promotes the development of memory and neuromorphic devices based on organic field-effect transistors(OFET).展开更多
Multisensory enhancement,as a facilitation phenomenon,is responsible for superior behavioral performance when an individual is responding to cross-modal versus modality-specific stimuli.However,the event-related poten...Multisensory enhancement,as a facilitation phenomenon,is responsible for superior behavioral performance when an individual is responding to cross-modal versus modality-specific stimuli.However,the event-related potential(ERP) counterparts of behavioral multisensory enhancement are not well known.We recorded ERPs and behavioral data from 14 healthy volunteers with three types of target stimuli(modality-specific,bimodal,and trimodal) to examine the spatio-temporal electrophysiological characteristics of multisensory enhancement by comparing behavioral data with ERPs.We found a strong correlation between P3 latency and behavioral performance in terms of reaction time(RT)(R = 0.98,P <0.001),suggesting that P3 latency constitutes a temporal measure of behavioral multisensory enhancement.In addition,a fast RT and short P3 latency were found when comparing the modality-specific visual target with the modality-specific auditory and somatosensory targets.Our results indicate that behavioral multisensory enhancement can be identified by the latency and source distribution of the P3 component.These findings may advance our understanding of the neuronal mechanisms of multisensory enhancement.展开更多
In the robotic community more and more hands are developed. Based on theexperience of HIT Hand and DLR Hand II, a smaller and easier manufactured dexterous robot hand withmultisen-sory function and high integration is...In the robotic community more and more hands are developed. Based on theexperience of HIT Hand and DLR Hand II, a smaller and easier manufactured dexterous robot hand withmultisen-sory function and high integration is jointly developed. The prototype of the hand issuccessfully built. It has 4 fingers in total 13-DOFs (degree of freedom). Each finger has 3-DOFsand 4 joints, the last 2 joints are mechanically coupled by means of four-bar linkage mechanism. Italso has an additional DOF to realize motion of the thumb relative to the palm. The fingertip forcecan reach up to 10 N. Full integration of mechanical body, actuation system, multisensory system andelectronics is a significant feature. DSP based control system is implemented in PCI busarchitecture and the serial communication between the hand and DSP needs only 2 lines.展开更多
The present study observed the effects of the non-invasive, mirror neuron system application on neurorehabilitation. In primate studies, mirror neurons have been shown to fire when the subject observes or performs a s...The present study observed the effects of the non-invasive, mirror neuron system application on neurorehabilitation. In primate studies, mirror neurons have been shown to fire when the subject observes or performs a specific action, thereby allowing for observation of motor cortex activation. This activation of the mirror neuron system could serve as a treatment for stroke patients. In the present study, the combination of a mirror neuron system-based therapy was introduced for the treatment of patients with motor-deficits, who could not perform rehabilitation exercises. The results also indicate that this therapeutic method plays a positive role in emotional regulation in the same patients.展开更多
Background:Saccades are rapid and abrupt eye movements that allow us to change the point of fixation very quickly.Saccades are generally made to visual points of interest,but we can also saccade to non-visual objects ...Background:Saccades are rapid and abrupt eye movements that allow us to change the point of fixation very quickly.Saccades are generally made to visual points of interest,but we can also saccade to non-visual objects that attract our attention.While there is a plethora of studies investigating saccadic eye movements to visual targets,there is very little evidence of how eye movement planning occurs when individuals are performing eye movements to non-visual targets across different sensory modalities.Methods:Fifteen adults with normal,or corrected to normal,vision made saccades to either visual,auditory,tactile or proprioceptive targets.In the auditory condition a speaker was positioned at one of eight locations along a circle surrounding a central fixation point.In the proprioceptive condition the participant’s finger was placed at one of the eight locations.In the tactile condition participants were touched on their right forearm in one of four eccentric location,left and right of a central point.Eye movements were made in complete darkness.Results:We compared the precision and accuracy of the eye movements to tactile,proprioceptive,and auditory targets in the dark.Overall,both precision and accuracy of movements to non-visual targets were significantly lower compared to visual targets.Conclusions:These differences emphasize the central role of the visual system in saccade planning.展开更多
Background:The ability to track objects as they move is critical for successful interaction with objects in the world.The multiple object tracking(MOT)paradigm has demonstrated that,within limits,our visual attention ...Background:The ability to track objects as they move is critical for successful interaction with objects in the world.The multiple object tracking(MOT)paradigm has demonstrated that,within limits,our visual attention capacity allows us to track multiple moving objects among distracters.Very little is known about dynamic auditory attention and the role of multisensory binding in attentional tracking.Here,we examined whether dynamic sounds congruent with visual targets could facilitate tracking in a 3D-MOT task.Methods:Participants tracked one or multiple target-spheres among identical distractor-spheres during 8 seconds of movement in a virtual cube.In the visual condition,targets were identified with a brief colour change,but were then indistinguishable from the distractors during the movement.In the audio-visual condition,the target-spheres were accompanied by a sound,which moved congruently with the change in the target’s position.Sound amplitude varied with distance from the observer and inter-aural amplitude difference varied with azimuth.Results:Results with one target showed that performance was better in the audiovisual condition,which suggests that congruent sounds can facilitate attentional visual tracking.However,with multiple targets,the sounds did not facilitate tracking.Conclusions:This suggests that audiovisual binding may not be possible when attention is divided between multiple targets.展开更多
The ability of sea turtle hatchlings to find the seashore soon after hatching is thought to be exclusively dependent upon visual information. Target-oriented movements in most vertebrates, however, relay on combining ...The ability of sea turtle hatchlings to find the seashore soon after hatching is thought to be exclusively dependent upon visual information. Target-oriented movements in most vertebrates, however, relay on combining information gathered through different sensory systems. Hence, in this work, we investigated whether olfactory and/or magnetic information might complement visual cues during hatchling’s seaward crawling. Acute olfactory deprivation and distorted magnetic sensation in visually competent hatchlings resulted in a scattering of seaward crawling routes among cardinal points, some of them being different from those strongly preferred by control hatchlings. In addition, blindfolded hatchlings also displayed a striking misrouting while crawling on the beach surface in spite of having intact olfactory and magnetic senses. Together these results support the notion that visual information is crucial for seaward crawling, but also that olfactory and magnetic information complement visual cues when turtle hatchlings display this behavior. Hence, the present observations suggest that multisensory cues are used by turtle hatchlings while crawling towards the sea. This work also has important implications on the design of species conservation measures and policies. In the near future, efforts must be made to identify and preserve the local natural sources of odors and magnetic cues, in addition to preventing the perturbing effects of artificial lighting on adult and hatchling turtle crawling behavior.展开更多
Some multimodal signals-that is,occurring in more than one sensory modality-appear to carry additional information which is not present when component signals are presented separately.To understand the function of mal...Some multimodal signals-that is,occurring in more than one sensory modality-appear to carry additional information which is not present when component signals are presented separately.To understand the function of male ring dove’s(Streptopelia risoria)multimodal courtship,we used audiovisual playback of male displays to investigate female response to stimuli differing in their audiovisual timing.From natural courtship recordings,we created a shifted stimulus where audio was shifted relative to video by afixed value and a jittered stimulus,where each call was moved randomly along the visual channel.We presented 3 groups of females with the same stimulus type,that is,control,shifted,and jittered,for 7 days.We recorded their behavior and assessed pre-and post-test blood estradiol concentration.We found that playback exposure increased estradiol levels,confirming that this technique can be efficiently used to study doves’sexual communication.Additionally,chasing behavior(indicating sexual stimulation)increased over experimental days only in the control condition,suggesting a role of multimodal timing on female response.This stresses the importance of signal configuration in multimodal communication,as additional information is likely to be contained in the temporal association between modalities.展开更多
Over the last decade virtual reality (VR) setups for rodents have been developed and utilized to in-vestigate the neural foundations of behavior. Such VR systems became very popular since theyallow the use of state-...Over the last decade virtual reality (VR) setups for rodents have been developed and utilized to in-vestigate the neural foundations of behavior. Such VR systems became very popular since theyallow the use of state-of-the-art techniques to measure neural activity in behaving rodents that can-not be easily used with classical behavior setups. Here, we provide an overview of rodent VR tech-nologies and review recent results from related research. We discuss commonalities and differ-ences as well as merits and issues of different approaches. A special focus is given to experimental(behavioral) paradigms in use. Finally we comment on possible use cases that may further exploitthe potential of VR in rodent research and hence inspire future studies.展开更多
Background:The aim of this research was to investigate the changes in the vision-related resting-state network (V-RSN) in pituitary adenoma (PA) patients after vision improvement,which was induced by operative tr...Background:The aim of this research was to investigate the changes in the vision-related resting-state network (V-RSN) in pituitary adenoma (PA) patients after vision improvement,which was induced by operative treatment.Methods:Ten PA patients with an improved visual acuity or/and visual field after transsphenoidal pituitary tumor resection were recruited and underwent a complete neuro-ophthalmologic evaluation,as well as an magnetic resonance imaging (MRI) protocol,including structural and resting-state functional MRI sequences before and after the operation.The regional homogeneity (ReHo) of the V-RSN was evaluated.Two sample t-test was performed to identify the significant differences in the V-RSN in the PA patients before and after transsphenoidal pituitary tumor resection.Results:Compared with the preoperation counterparts,the PA patients with improved vision after the operation exhibited reduced ReHo in the bilateral thalamus,globus pallidus,caudate nucleus,putamen nucleus,supplementary motor area,and left hippocampal formation,and increased ReHo in the bilateral cuneus gyrus,calcarine gyrus,right lingual gyrus,and fusiform gyrus.Conclusions:PA patients with improved vision exhibit increased neural activity within the visual cortex,but decreased neural activity in subareas of the multisensory and multimodal systems beyond the vision cortex.展开更多
Background: In the classical psychological refractory period (PRP) paradigm, two stimuli are presented in brief succession, and participants are asked to make separate speeded responses to both stimuli. Due to a ce...Background: In the classical psychological refractory period (PRP) paradigm, two stimuli are presented in brief succession, and participants are asked to make separate speeded responses to both stimuli. Due to a central cognitive bottleneck, responses to the second stimulus are delayed, especially at short stimulus-onset asynchrony (SOA) between the two stimuli. Although the mechanisms of dual-task interference in the classical PRP paradigm have been extensively investigated, specific mechanisms underlying the cross-modal PRP paradigm are not well understood. In particular, it remains unknown whether the dominance of vision over audition manifests in the cross-modal PRP tasks. The present study aimed to investigate whether the visual dominance effect manifessts in the cross-modal PRP paradigm. Methods: We adapted the classical PRP paradigm by manipulating the order of a visual and an auditory task: the visual task could either precede the auditory task or vice versa, at either short or long SOAs. Twenty-five healthy participants took part in Experiment 1, and thirty-three new participants took part in Experiment 2. Reaction time and accuracy data were calculated and further analyzed by repeated-measures analysis of variance. Results: The results showed that visual precedence in the Visual-Auditory condition caused larger impairments to the subsequent auditory processing than vice versa in the Auditory-Visual condition: a larger delay of second response was revealed in the Visual-Auditory condition ( 135 ± 10 ms) than the Auditory-Visual condition (88 ± 9 ms). This effect was found only at the short SOAs tinder the existence of the central bottleneck, but not at the long SOAs. Moreover, this effect occurred both when the single visual and the single auditory task were of equal difficulty in Experiment I and when the single auditory task was more difficult than the single visual task in Experiment 2. Conclusion: Results of the two experiments suggested that the visual dominance effect occurred under the central bottleneck of cognitive processing.展开更多
文摘Understanding the neural underpinning of human gait and balance is one of the most pertinent challenges for 21st-century translational neuroscience due to the profound impact that falls and mobility disturbances have on our aging population.Posture and gait control does not happen automatically,as previously believed,but rather requires continuous involvement of central nervous mechanisms.To effectively exert control over the body,the brain must integrate multiple streams of sensory information,including visual,vestibular,and somatosensory signals.The mechanisms which underpin the integration of these multisensory signals are the principal topic of the present work.Existing multisensory integration theories focus on how failure of cognitive processes thought to be involved in multisensory integration leads to falls in older adults.Insufficient emphasis,however,has been placed on specific contributions of individual sensory modalities to multisensory integration processes and cross-modal interactions that occur between the sensory modalities in relation to gait and balance.In the present work,we review the contributions of somatosensory,visual,and vestibular modalities,along with their multisensory intersections to gait and balance in older adults and patients with Parkinson’s disease.We also review evidence of vestibular contributions to multisensory temporal binding windows,previously shown to be highly pertinent to fall risk in older adults.Lastly,we relate multisensory vestibular mechanisms to potential neural substrates,both at the level of neurobiology(concerning positron emission tomography imaging)and at the level of electrophysiology(concerning electroencephalography).We hope that this integrative review,drawing influence across multiple subdisciplines of neuroscience,paves the way for novel research directions and therapeutic neuromodulatory approaches,to improve the lives of older adults and patients with neurodegenerative diseases.
基金supported by grants from the National Twelfth Five-Year Research Program of China(No.2012BAI12B02)the National Natural Science Foundation of China(No.81873701)the Two-hundred Talent Fund of Medical College of Shanghai Jiao Tong University(No.20191921).
文摘Objective To examine the effect of multisensory exercise on balance disorders.Methods PubMed,Scopus and Web of Science were searched to identify eligible studies published before January 1,2020.Eligible studies included randomized control trials(RCTs),non-randomized studies,case-control studies,and cohort studies.The methodological quality of the included studies was evaluated using JBI Critical Appraisal Checklists for RCTs and for Quasi-Experimental Studies by two researchers independently.A narrative synthesis of intervention characteristics and health-related outcomes was performed.Results A total of 11 non-randomized studies and 9 RCTs were eligible,including 667 participants.The results supported our assumption that multisensory exercise improved balance in people with balance disorders.All of the 20 studies were believed to be of high or moderate quality.Conclusion Our study confirmed that multisensory exercise was effective in improving balance in people with balance disorders.Multisensory exercises could lower the risk of fall and enhance confidence level to improve the quality of life.Further research is needed to investigate the optimal strategy of multisensory exercises and explore the underlying neural and molecular mechanisms of balance improvement brought by multisensory exercises.
基金This work was supported by the GRRC program of Gyeonggi province.[GRRC-Gachon2020(B04),Development of AI-based Healthcare Devices].
文摘Progress in understanding multisensory integration in human have suggested researchers that the integration may result into the enhancement or depression of incoming signals.It is evident based on different psychological and behavioral experiments that stimuli coming from different perceptual modalities at the same time or from the same place,the signal having more strength under the influence of emotions effects the response accordingly.Current research inmultisensory integration has not studied the effect of emotions despite its significance and natural influence in multisensory enhancement or depression.Therefore,there is a need to integrate the emotional state of the agent with incoming stimuli for signal enhancement or depression.In this study,two different neural network-based learning algorithms have been employed to learn the impact of emotions on signal enhancement or depression.It was observed that the performance of a proposed system for multisensory integration increases when emotion features were present during enhancement or depression of multisensory signals.
文摘Energy consumption of the EU has a crucial environmental impact;several efforts are nowadays thus directed into massively reducing energy consumption by envelope improvement,system efficiency and smart control.On the other hand,the indoor thermal and lighting conditions significantly influence users’wellbeing and productivity,which is especially important when dealing with educational and working facilities.Strategies to enhance system efficiency are focused on design and construction aspects.These strategies ease to promote a powerful approach which is needed when focusing on existing buildings in need of retrofit measures.When dealing with new or refurbished buildings,energy saving has a further step to achieve.In the last years,research trends moved towards an increasing inclusion of human factors in energy evaluation.This allows to account for the occupancy variability in the energy analyses,considering how to bridge the performance gap between predictive models and actual consumptions due to indoor thermal settings.In empty buildings energy consumption is huge and economic efforts are wasted,due to unconscious energy-wasting behaviors.Previous studies with Interactive Virtual Environments confirmed that indoor environments illuminated by different color lights lead to the perception of different levels of thermal comfort.The results of the present paper replicate previous evidence collected in real conditions,suggesting that Virtual Reality is a valid and reliable tool to assess thermal comfort more quickly and cheaply.This paper provides a further perspective on this topic,including also the use of different fragrances to understand how the indoor environment could be enhanced and manipulated to increase wellbeing,thermal perception and finally energy saving.
文摘Background:The concept of stochastic facilitation suggests that the addition of precise amounts of white noise can improve the perceptibility of a stimulus of weak amplitude.We know from previous research that tactile and auditory noise can facilitate visual perception,respectively.Here we wanted to see if the effects of stochastic facilitation generalise to a reaction time paradigm,and if reaction times are correlated with tactile thresholds.We know that when multiple sensory systems are stimulated simultaneously,reaction times are faster than either stimulus alone,and also faster than the sum of reaction times(known as the race model).Methods:Five participants were re-tested in five blocks each of which contained a different background noise levels,randomly ordered across sessions.At each noise level,they performed a tactile threshold detection task and a tactile reaction time task.Results:Both tactile threshold and tactile reaction times were significantly affected by the background white noise.While the preferred amplitude for the white noise was different for every participant,the average lowest threshold was obtained with white noise presented binaurally at 70 db.The reaction times were analysed by fitting an ex-Gaussian,the sum of a Gaussian function and an exponential decay function.The white noise significantly affected the exponential parameter(tau)in a way that is compatible with the facilitation of thresholds.Conclusions:We therefore conclude that multisensory reaction time facilitation can,at least in part,be explained by stochastic facilitation of the neural signals.
基金Supported by Natural Science Foundation of China(NSFC61527804)NSFC and the German Research Foundation(DFG)in Project Crossmodal Learning(NSFC 61621136008/DFG TRR-169)Research fund from brain lab,TAL education group,China.
文摘Multisensory lab based in Peking University,has carried out basic studies in multisensory space and time processing,intersensory binding and haptic/tactile perception.We exploited a typical paradigm of multisensory illusion-temporal ventriloquist effect and applied it in a wide range of multisensory interactions(mainly focused on temporal processing).In this work,we summarized how the tactile stimuli were exploited to compose tactile cues and as tactile apparent motion to interface with other sensory stimuli(visual and auditory stimuli)to examine the underlying perceptual organization in a multisensory context.Moreover,we introduced two examples of wearable haptic/tactile perception in our lab,by using two customized tactile devices and discussed the potential applications in this field.
基金supported by grants from the National Science and Technology Innovation 2030 Major Program(2022ZD0205000)the Strategic Priority Research Program of CAS(XDB32070000)+1 种基金the Shanghai Municipal Science and Technology Major Project(2018SHZDZX05)the Shanghai Academic Research Leader Program(21XD1404000).
文摘Accurate self-motion perception,which is critical for organisms to survive,is a process involving multiple sensory cues.The two most powerful cues are visual(optic flow)and vestibular(inertial motion).Psychophysical studies have indicated that humans and nonhuman primates integrate the two cues to improve the estimation of self-motion direction,often in a statistically Bayesian-optimal way.In the last decade,single-unit recordings in awake,behaving animals have provided valuable neurophysiological data with a high spatial and temporal resolution,giving insight into possible neural mechanisms underlying multisensory self-motion perception.Here,we review these findings,along with new evidence from the most recent studies focusing on the temporal dynamics of signals in different modalities.We show that,in light of new data,conventional thoughts about the cortical mechanisms underlying visuo-vestibular integration for linear self-motion are challenged.We propose that different temporal component signals may mediate different functions,a possibility that requires future studies.
基金the National Key Research and Development Program of China(No.2019YFB2204400)the Natural Science Basic Research Program of Shaanxi(No.2022JQ-650).
文摘A visual and tactile multisensory integrated system is essential for human walking due to the demand for real-time interactions between perception and action.Here,a piezoresistor and MoS_(2)field effect transistor are combined to construct an artificial integration nervous system to simulate perception and synaptic plasticity.The key characteristics of synaptic plasticity are successfully demonstrated by individual pressure signals,individual optical signals,the synergy of optical and pressure signals,which are based on the electron trapping–detrapping mechanism at the MoS_(2)/SiO_(2)interface.We demonstrate that perception under synergy is stronger than perception under optical or pressure signal alone,which is similar to a biological system.Moreover,various distinguishable motion scenarios(combination of the following conditions:external lighting environment of day or night,flat or rough road,movement state of walking or running)are simulated and verified by adjusting the amplitude and frequency of the optical and pressure signals.
基金the National Natural Science Foundation of China(Nos.62274035,U21A20497,61974029,and 11604051)the National Key Research and Development Program of China(Nos.2022YFB3603803 and 2022YFB3603802)+1 种基金the Natural Science Foundation of Fujian Province(Nos.2020J05104 and 2020J06012)Fujian Science&Technology Innovation Laboratory for Optoelectronic Information of China(Nos.2021ZZ129 and 2021ZZ130).
文摘Multi-sensory neuromorphic devices(MND)have broad potential in overcoming the structural bottleneck of von Neumann in the era of big data.However,the current multisensory artificial neuromorphic system is mainly based on unitary nonvolatile memory or volatile synaptic devices without intrinsic thermal sensitivity,which limits the range of biological multisensory perception and the flexibility and computational efficiency of the neural morphological computing system.Here,a temperature-dependent memory/synaptic hybrid artificial neuromorphic device based on floating gate phototransistors(FGT)is fabricated.The CsPbBr_(3)/TiO_(2)core–shell nanocrystals(NCs)prepared by in-situ pre-protection low-temperature solvothermal method were used as the photosensitive layer.The device exhibits remarkable multi-level visual memory with a large memory window of 59.6 V at room temperature.Surprisingly,when the temperature varies from 20 to 120℃back and forth,the device can switch between nonvolatile memory and volatile synaptic device with reconfigurable and reversible behaviors,which contributes to the efficient visual/thermal fusion perception.This work expands the sensory range of multisensory devices and promotes the development of memory and neuromorphic devices based on organic field-effect transistors(OFET).
基金partly supported by the National Natural Science Foundation of China(81271685)Key Project of Science and Technology Fund of Tianjin Municipality,China(10JCZDJC16100)+1 种基金the China Postdoctoral Science Foundation(2012M510754)the Independent Innovative Fund of Tianjin University,Tianjin Municipality,China(1102)
文摘Multisensory enhancement,as a facilitation phenomenon,is responsible for superior behavioral performance when an individual is responding to cross-modal versus modality-specific stimuli.However,the event-related potential(ERP) counterparts of behavioral multisensory enhancement are not well known.We recorded ERPs and behavioral data from 14 healthy volunteers with three types of target stimuli(modality-specific,bimodal,and trimodal) to examine the spatio-temporal electrophysiological characteristics of multisensory enhancement by comparing behavioral data with ERPs.We found a strong correlation between P3 latency and behavioral performance in terms of reaction time(RT)(R = 0.98,P <0.001),suggesting that P3 latency constitutes a temporal measure of behavioral multisensory enhancement.In addition,a fast RT and short P3 latency were found when comparing the modality-specific visual target with the modality-specific auditory and somatosensory targets.Our results indicate that behavioral multisensory enhancement can be identified by the latency and source distribution of the P3 component.These findings may advance our understanding of the neuronal mechanisms of multisensory enhancement.
文摘In the robotic community more and more hands are developed. Based on theexperience of HIT Hand and DLR Hand II, a smaller and easier manufactured dexterous robot hand withmultisen-sory function and high integration is jointly developed. The prototype of the hand issuccessfully built. It has 4 fingers in total 13-DOFs (degree of freedom). Each finger has 3-DOFsand 4 joints, the last 2 joints are mechanically coupled by means of four-bar linkage mechanism. Italso has an additional DOF to realize motion of the thumb relative to the palm. The fingertip forcecan reach up to 10 N. Full integration of mechanical body, actuation system, multisensory system andelectronics is a significant feature. DSP based control system is implemented in PCI busarchitecture and the serial communication between the hand and DSP needs only 2 lines.
文摘The present study observed the effects of the non-invasive, mirror neuron system application on neurorehabilitation. In primate studies, mirror neurons have been shown to fire when the subject observes or performs a specific action, thereby allowing for observation of motor cortex activation. This activation of the mirror neuron system could serve as a treatment for stroke patients. In the present study, the combination of a mirror neuron system-based therapy was introduced for the treatment of patients with motor-deficits, who could not perform rehabilitation exercises. The results also indicate that this therapeutic method plays a positive role in emotional regulation in the same patients.
文摘Background:Saccades are rapid and abrupt eye movements that allow us to change the point of fixation very quickly.Saccades are generally made to visual points of interest,but we can also saccade to non-visual objects that attract our attention.While there is a plethora of studies investigating saccadic eye movements to visual targets,there is very little evidence of how eye movement planning occurs when individuals are performing eye movements to non-visual targets across different sensory modalities.Methods:Fifteen adults with normal,or corrected to normal,vision made saccades to either visual,auditory,tactile or proprioceptive targets.In the auditory condition a speaker was positioned at one of eight locations along a circle surrounding a central fixation point.In the proprioceptive condition the participant’s finger was placed at one of the eight locations.In the tactile condition participants were touched on their right forearm in one of four eccentric location,left and right of a central point.Eye movements were made in complete darkness.Results:We compared the precision and accuracy of the eye movements to tactile,proprioceptive,and auditory targets in the dark.Overall,both precision and accuracy of movements to non-visual targets were significantly lower compared to visual targets.Conclusions:These differences emphasize the central role of the visual system in saccade planning.
文摘Background:The ability to track objects as they move is critical for successful interaction with objects in the world.The multiple object tracking(MOT)paradigm has demonstrated that,within limits,our visual attention capacity allows us to track multiple moving objects among distracters.Very little is known about dynamic auditory attention and the role of multisensory binding in attentional tracking.Here,we examined whether dynamic sounds congruent with visual targets could facilitate tracking in a 3D-MOT task.Methods:Participants tracked one or multiple target-spheres among identical distractor-spheres during 8 seconds of movement in a virtual cube.In the visual condition,targets were identified with a brief colour change,but were then indistinguishable from the distractors during the movement.In the audio-visual condition,the target-spheres were accompanied by a sound,which moved congruently with the change in the target’s position.Sound amplitude varied with distance from the observer and inter-aural amplitude difference varied with azimuth.Results:Results with one target showed that performance was better in the audiovisual condition,which suggests that congruent sounds can facilitate attentional visual tracking.However,with multiple targets,the sounds did not facilitate tracking.Conclusions:This suggests that audiovisual binding may not be possible when attention is divided between multiple targets.
基金supported by grants from the Consejo Nacional de Ciencia y Tecnologia(CONACyT No.45872M,94312 and 82879 to GGO)from the Coordinacion de la Investigacion Cientifica,Universidad Michoacana de San Nicolas de Hidalgo(No.8.37 to ALFF)Additional funding was provided by the Coordinacion de la Investigacion Cientifica,Universidad Nacional Autonoma de Mexico,and PROMEP,SEP.
文摘The ability of sea turtle hatchlings to find the seashore soon after hatching is thought to be exclusively dependent upon visual information. Target-oriented movements in most vertebrates, however, relay on combining information gathered through different sensory systems. Hence, in this work, we investigated whether olfactory and/or magnetic information might complement visual cues during hatchling’s seaward crawling. Acute olfactory deprivation and distorted magnetic sensation in visually competent hatchlings resulted in a scattering of seaward crawling routes among cardinal points, some of them being different from those strongly preferred by control hatchlings. In addition, blindfolded hatchlings also displayed a striking misrouting while crawling on the beach surface in spite of having intact olfactory and magnetic senses. Together these results support the notion that visual information is crucial for seaward crawling, but also that olfactory and magnetic information complement visual cues when turtle hatchlings display this behavior. Hence, the present observations suggest that multisensory cues are used by turtle hatchlings while crawling towards the sea. This work also has important implications on the design of species conservation measures and policies. In the near future, efforts must be made to identify and preserve the local natural sources of odors and magnetic cues, in addition to preventing the perturbing effects of artificial lighting on adult and hatchling turtle crawling behavior.
基金supported by Leonida Fusani’s Startup funding of the University of Vienna,Austria,by the Austrian Science Fund(FWF),W1262-B29by the Vienna Science and Technology Fund(WWTF)CS18-021。
文摘Some multimodal signals-that is,occurring in more than one sensory modality-appear to carry additional information which is not present when component signals are presented separately.To understand the function of male ring dove’s(Streptopelia risoria)multimodal courtship,we used audiovisual playback of male displays to investigate female response to stimuli differing in their audiovisual timing.From natural courtship recordings,we created a shifted stimulus where audio was shifted relative to video by afixed value and a jittered stimulus,where each call was moved randomly along the visual channel.We presented 3 groups of females with the same stimulus type,that is,control,shifted,and jittered,for 7 days.We recorded their behavior and assessed pre-and post-test blood estradiol concentration.We found that playback exposure increased estradiol levels,confirming that this technique can be efficiently used to study doves’sexual communication.Additionally,chasing behavior(indicating sexual stimulation)increased over experimental days only in the control condition,suggesting a role of multimodal timing on female response.This stresses the importance of signal configuration in multimodal communication,as additional information is likely to be contained in the temporal association between modalities.
文摘Over the last decade virtual reality (VR) setups for rodents have been developed and utilized to in-vestigate the neural foundations of behavior. Such VR systems became very popular since theyallow the use of state-of-the-art techniques to measure neural activity in behaving rodents that can-not be easily used with classical behavior setups. Here, we provide an overview of rodent VR tech-nologies and review recent results from related research. We discuss commonalities and differ-ences as well as merits and issues of different approaches. A special focus is given to experimental(behavioral) paradigms in use. Finally we comment on possible use cases that may further exploitthe potential of VR in rodent research and hence inspire future studies.
文摘Background:The aim of this research was to investigate the changes in the vision-related resting-state network (V-RSN) in pituitary adenoma (PA) patients after vision improvement,which was induced by operative treatment.Methods:Ten PA patients with an improved visual acuity or/and visual field after transsphenoidal pituitary tumor resection were recruited and underwent a complete neuro-ophthalmologic evaluation,as well as an magnetic resonance imaging (MRI) protocol,including structural and resting-state functional MRI sequences before and after the operation.The regional homogeneity (ReHo) of the V-RSN was evaluated.Two sample t-test was performed to identify the significant differences in the V-RSN in the PA patients before and after transsphenoidal pituitary tumor resection.Results:Compared with the preoperation counterparts,the PA patients with improved vision after the operation exhibited reduced ReHo in the bilateral thalamus,globus pallidus,caudate nucleus,putamen nucleus,supplementary motor area,and left hippocampal formation,and increased ReHo in the bilateral cuneus gyrus,calcarine gyrus,right lingual gyrus,and fusiform gyrus.Conclusions:PA patients with improved vision exhibit increased neural activity within the visual cortex,but decreased neural activity in subareas of the multisensory and multimodal systems beyond the vision cortex.
文摘Background: In the classical psychological refractory period (PRP) paradigm, two stimuli are presented in brief succession, and participants are asked to make separate speeded responses to both stimuli. Due to a central cognitive bottleneck, responses to the second stimulus are delayed, especially at short stimulus-onset asynchrony (SOA) between the two stimuli. Although the mechanisms of dual-task interference in the classical PRP paradigm have been extensively investigated, specific mechanisms underlying the cross-modal PRP paradigm are not well understood. In particular, it remains unknown whether the dominance of vision over audition manifests in the cross-modal PRP tasks. The present study aimed to investigate whether the visual dominance effect manifessts in the cross-modal PRP paradigm. Methods: We adapted the classical PRP paradigm by manipulating the order of a visual and an auditory task: the visual task could either precede the auditory task or vice versa, at either short or long SOAs. Twenty-five healthy participants took part in Experiment 1, and thirty-three new participants took part in Experiment 2. Reaction time and accuracy data were calculated and further analyzed by repeated-measures analysis of variance. Results: The results showed that visual precedence in the Visual-Auditory condition caused larger impairments to the subsequent auditory processing than vice versa in the Auditory-Visual condition: a larger delay of second response was revealed in the Visual-Auditory condition ( 135 ± 10 ms) than the Auditory-Visual condition (88 ± 9 ms). This effect was found only at the short SOAs tinder the existence of the central bottleneck, but not at the long SOAs. Moreover, this effect occurred both when the single visual and the single auditory task were of equal difficulty in Experiment I and when the single auditory task was more difficult than the single visual task in Experiment 2. Conclusion: Results of the two experiments suggested that the visual dominance effect occurred under the central bottleneck of cognitive processing.