Crossmodal Modulation as a Basis for Visual Enhancement of Auditory Performance

• Main Author: Qian, Cheng
• Other Authors: Wong, Willy
• Language: en_ca
• Published: 2009
• Subjects: crossmodal; audio-visual; multisensory; auditory detection; visual enhancement; sensory modelling; 0541; 0317; 0544
• Online Access: http://hdl.handle.net/1807/18831

The human sensory system processes many modalities simultaneously. It was believed that each modality would be processed individually first, and their combination deferred to higher-level cortical areas. Recent neurophysiological investigations indicate interconnections between early visual and auditory cortices, areas putatively considered unimodal, but the function remains unclear. The present work explores how this cross-modality might contribute to a visual enhancement of auditory performance, using a combined theoretical and experimental approach. The enhancement of sensory performance was studied through a signal detection framework. A model was constructed using principles from signal detection theory and neurophysiology, demonstrating enhancements of roughly 1.8dB both analytically and through simulation. Several experiments were conducted to observe e ects of visual cues on a 2-alternative-forced-choice detection task of an auditory tone in noise. Results of the main experiment showed an enhancement of 1.6dB. Better enhancement also tended to occur for more realistic relationships between audio to visual stimuli.