Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance

Object sounds can enhance the attentional selection and perceptual processing of semantically-related visual stimuli. However, it is currently unknown whether crossmodal semantic congruence also affects the post-perceptual stages of information processing, such as short-term memory (STM), and whethe...

Full description

Bibliographic Details
Main Authors: Erika Almadori, Serena Mastroberardino, Fabiano Botta, Riccardo Brunetti, Juan Lupiáñez, Charles Spence, Valerio Santangelo
Format: Article
Language:English
Published: MDPI AG 2021-09-01
Series:Brain Sciences
Subjects:
Online Access:https://www.mdpi.com/2076-3425/11/9/1206
id doaj-899e85f6e1d94d1a82d70987cc8ca4a1
record_format Article
spelling doaj-899e85f6e1d94d1a82d70987cc8ca4a12021-09-25T23:48:41ZengMDPI AGBrain Sciences2076-34252021-09-01111206120610.3390/brainsci11091206Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory PerformanceErika Almadori0Serena Mastroberardino1Fabiano Botta2Riccardo Brunetti3Juan Lupiáñez4Charles Spence5Valerio Santangelo6Neuroimaging Laboratory, IRCCS Santa Lucia Foundation, Via Ardeatina 306, 00179 Rome, ItalyDepartment of Psychology, School of Medicine & Psychology, Sapienza University of Rome, Via dei Marsi 78, 00185 Rome, ItalyDepartment of Experimental Psychology and Mind, Brain, and Behavior Research Center (CIMCYC), University of Granada, 18071 Granada, SpainCognitive and Clinical Psychology Laboratory, Department of Human Sciences, Università Europea di Roma, 00163 Roma, ItalyDepartment of Experimental Psychology and Mind, Brain, and Behavior Research Center (CIMCYC), University of Granada, 18071 Granada, SpainDepartment of Experimental Psychology, Oxford University, Oxford OX2 6GG, UKNeuroimaging Laboratory, IRCCS Santa Lucia Foundation, Via Ardeatina 306, 00179 Rome, ItalyObject sounds can enhance the attentional selection and perceptual processing of semantically-related visual stimuli. However, it is currently unknown whether crossmodal semantic congruence also affects the post-perceptual stages of information processing, such as short-term memory (STM), and whether this effect is modulated by the object consistency with the background visual scene. In two experiments, participants viewed everyday visual scenes for 500 ms while listening to an object sound, which could either be semantically related to the object that served as the STM target at retrieval or not. This defined crossmodal semantically cued vs. uncued targets. The target was either in- or out-of-context with respect to the background visual scene. After a maintenance period of 2000 ms, the target was presented in isolation against a neutral background, in either the same or different spatial position as in the original scene. The participants judged the same vs. different position of the object and then provided a confidence judgment concerning the certainty of their response. The results revealed greater accuracy when judging the spatial position of targets paired with a semantically congruent object sound at encoding. This crossmodal facilitatory effect was modulated by whether the target object was in- or out-of-context with respect to the background scene, with out-of-context targets reducing the facilitatory effect of object sounds. Overall, these findings suggest that the presence of the object sound at encoding facilitated the selection and processing of the semantically related visual stimuli, but this effect depends on the semantic configuration of the visual scene.https://www.mdpi.com/2076-3425/11/9/1206short-term memorysemanticsvisual scenesauditorymultisensoryspatial orienting
collection DOAJ
language English
format Article
sources DOAJ
author Erika Almadori
Serena Mastroberardino
Fabiano Botta
Riccardo Brunetti
Juan Lupiáñez
Charles Spence
Valerio Santangelo
spellingShingle Erika Almadori
Serena Mastroberardino
Fabiano Botta
Riccardo Brunetti
Juan Lupiáñez
Charles Spence
Valerio Santangelo
Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance
Brain Sciences
short-term memory
semantics
visual scenes
auditory
multisensory
spatial orienting
author_facet Erika Almadori
Serena Mastroberardino
Fabiano Botta
Riccardo Brunetti
Juan Lupiáñez
Charles Spence
Valerio Santangelo
author_sort Erika Almadori
title Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance
title_short Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance
title_full Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance
title_fullStr Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance
title_full_unstemmed Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance
title_sort crossmodal semantic congruence interacts with object contextual consistency in complex visual scenes to enhance short-term memory performance
publisher MDPI AG
series Brain Sciences
issn 2076-3425
publishDate 2021-09-01
description Object sounds can enhance the attentional selection and perceptual processing of semantically-related visual stimuli. However, it is currently unknown whether crossmodal semantic congruence also affects the post-perceptual stages of information processing, such as short-term memory (STM), and whether this effect is modulated by the object consistency with the background visual scene. In two experiments, participants viewed everyday visual scenes for 500 ms while listening to an object sound, which could either be semantically related to the object that served as the STM target at retrieval or not. This defined crossmodal semantically cued vs. uncued targets. The target was either in- or out-of-context with respect to the background visual scene. After a maintenance period of 2000 ms, the target was presented in isolation against a neutral background, in either the same or different spatial position as in the original scene. The participants judged the same vs. different position of the object and then provided a confidence judgment concerning the certainty of their response. The results revealed greater accuracy when judging the spatial position of targets paired with a semantically congruent object sound at encoding. This crossmodal facilitatory effect was modulated by whether the target object was in- or out-of-context with respect to the background scene, with out-of-context targets reducing the facilitatory effect of object sounds. Overall, these findings suggest that the presence of the object sound at encoding facilitated the selection and processing of the semantically related visual stimuli, but this effect depends on the semantic configuration of the visual scene.
topic short-term memory
semantics
visual scenes
auditory
multisensory
spatial orienting
url https://www.mdpi.com/2076-3425/11/9/1206
work_keys_str_mv AT erikaalmadori crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance
AT serenamastroberardino crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance
AT fabianobotta crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance
AT riccardobrunetti crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance
AT juanlupianez crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance
AT charlesspence crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance
AT valeriosantangelo crossmodalsemanticcongruenceinteractswithobjectcontextualconsistencyincomplexvisualscenestoenhanceshorttermmemoryperformance
_version_ 1717367859138527232