A dual-channel language decoding from brain activity with progressive transfer training

• Main Authors: Chen, H. (Author), Cheng, K. (Author), Huang, W. (Author), Li, C. (Author), Li, J. (Author), Wang, C. (Author), Wang, Y. (Author), Yan, H. (Author), Zuo, Z. (Author)
• Format: Article
• Language: English
• Published: John Wiley and Sons Inc 2021
• Subjects: adult; Adult; Article; artificial intelligence; Artificial Intelligence; brain mapping; Brain Mapping; electroencephalogram; feature selection algorithm; female; Female; functional magnetic resonance imaging; functional neuroimaging; human; human experiment; Humans; language decoding; Magnetic Resonance Imaging; male; Male; natural language processing; nuclear magnetic resonance imaging; physiology; progressive transfer; psycholinguistics; Psycholinguistics; quantitative analysis; retina image; transfer of learning; vision; visual cortex; Visual Cortex; Visual Perception; young adult; Young Adult
• Online Access: View Fulltext in Publisher

 When we view a scene, the visual cortex extracts and processes visual information in the scene through various kinds of neural activities. Previous studies have decoded the neural activity into single/multiple semantic category tags which can caption the scene to some extent. However, these tags are isolated words with no grammatical structure, insufficiently conveying what the scene contains. It is well-known that textual language (sentences/phrases) is superior to single word in disclosing the meaning of images as well as reflecting people's real understanding of the images. Here, based on artificial intelligence technologies, we attempted to build a dual-channel language decoding model (DC-LDM) to decode the neural activities evoked by images into language (phrases or short sentences). The DC-LDM consisted of five modules, namely, Image-Extractor, Image-Encoder, Nerve-Extractor, Nerve-Encoder, and Language-Decoder. In addition, we employed a strategy of progressive transfer to train the DC-LDM for improving the performance of language decoding. The results showed that the texts decoded by DC-LDM could describe natural image stimuli accurately and vividly. We adopted six indexes to quantitatively evaluate the difference between the decoded texts and the annotated texts of corresponding visual images, and found that Word2vec-Cosine similarity (WCS) was the best indicator to reflect the similarity between the decoded and the annotated texts. In addition, among different visual cortices, we found that the text decoded by the higher visual cortex was more consistent with the description of the natural image than the lower one. Our decoding model may provide enlightenment in language-based brain-computer interface explorations. © 2021 The Authors. Human Brain Mapping published by Wiley Periodicals LLC.