Towards a Synesthesia Laboratory: Real-time Localization and Visualization of a Sound Source for Virtual Reality Applications

• Main Authors: Ahmet Kose, Aleksei Tepljakov, Sergei Astapov, Dirk Draheim, Eduard Petlenkov, Kristina Vassiljeva
• Format: Article
• Language: English
• Published: Croatian Communications and Information Society (CCIS) 2018-03-01
• Series: Journal of Communications Software and Systems
• Subjects: virtual reality; synesthesia; acoustic localization; microphone array; therapy
• Online Access: https://jcomss.fesb.unist.hr/index.php/jcomss/article/view/410

In this paper, we present our findings related to the problem of localization and visualization of a sound source placed in the same room as the listener. The particular effect that we aim to investigate is called synesthesia—the act of experiencing one sense modality as another, e.g., a person may vividly experience flashes of colors when listening to a series of sounds. Towards that end, we apply a series of recently developed methods for detecting sound source in a three-dimensional space around the listener.We also apply a Kalman filter to smooth out the perceived motion. Further, we transform the audio signal into a series of visual shapes, such that the size of each shape is determined by theloudness of the sound source, and its color is determined by the dominant spectral component of the sound. The developed prototype is verified in real time. The prototype configuration is described and some initial experimental results are reported and discussed. Some ideas for further development are also presented.