A dual-channel language decoding from brain activity with progressive transfer training

When we view a scene, the visual cortex extracts and processes visual information in the scene through various kinds of neural activities. Previous studies have decoded the neural activity into single/multiple semantic category tags which can caption the scene to some extent. However, these tags are...

Full description

Bibliographic Details
Main Authors: Chen, H. (Author), Cheng, K. (Author), Huang, W. (Author), Li, C. (Author), Li, J. (Author), Wang, C. (Author), Wang, Y. (Author), Yan, H. (Author), Zuo, Z. (Author)
Format: Article
Language:English
Published: John Wiley and Sons Inc 2021
Subjects:
Online Access:View Fulltext in Publisher
LEADER 03831nam a2200697Ia 4500
001 10.1002-hbm.25603
008 220427s2021 CNT 000 0 und d
020 |a 10659471 (ISSN) 
245 1 0 |a A dual-channel language decoding from brain activity with progressive transfer training 
260 0 |b John Wiley and Sons Inc  |c 2021 
856 |z View Fulltext in Publisher  |u https://doi.org/10.1002/hbm.25603 
520 3 |a When we view a scene, the visual cortex extracts and processes visual information in the scene through various kinds of neural activities. Previous studies have decoded the neural activity into single/multiple semantic category tags which can caption the scene to some extent. However, these tags are isolated words with no grammatical structure, insufficiently conveying what the scene contains. It is well-known that textual language (sentences/phrases) is superior to single word in disclosing the meaning of images as well as reflecting people's real understanding of the images. Here, based on artificial intelligence technologies, we attempted to build a dual-channel language decoding model (DC-LDM) to decode the neural activities evoked by images into language (phrases or short sentences). The DC-LDM consisted of five modules, namely, Image-Extractor, Image-Encoder, Nerve-Extractor, Nerve-Encoder, and Language-Decoder. In addition, we employed a strategy of progressive transfer to train the DC-LDM for improving the performance of language decoding. The results showed that the texts decoded by DC-LDM could describe natural image stimuli accurately and vividly. We adopted six indexes to quantitatively evaluate the difference between the decoded texts and the annotated texts of corresponding visual images, and found that Word2vec-Cosine similarity (WCS) was the best indicator to reflect the similarity between the decoded and the annotated texts. In addition, among different visual cortices, we found that the text decoded by the higher visual cortex was more consistent with the description of the natural image than the lower one. Our decoding model may provide enlightenment in language-based brain-computer interface explorations. © 2021 The Authors. Human Brain Mapping published by Wiley Periodicals LLC. 
650 0 4 |a adult 
650 0 4 |a Adult 
650 0 4 |a Article 
650 0 4 |a artificial intelligence 
650 0 4 |a artificial intelligence 
650 0 4 |a Artificial Intelligence 
650 0 4 |a brain mapping 
650 0 4 |a Brain Mapping 
650 0 4 |a electroencephalogram 
650 0 4 |a feature selection algorithm 
650 0 4 |a female 
650 0 4 |a Female 
650 0 4 |a functional magnetic resonance imaging 
650 0 4 |a functional magnetic resonance imaging 
650 0 4 |a functional neuroimaging 
650 0 4 |a human 
650 0 4 |a human experiment 
650 0 4 |a Humans 
650 0 4 |a language decoding 
650 0 4 |a Magnetic Resonance Imaging 
650 0 4 |a male 
650 0 4 |a Male 
650 0 4 |a natural language processing 
650 0 4 |a nuclear magnetic resonance imaging 
650 0 4 |a physiology 
650 0 4 |a progressive transfer 
650 0 4 |a psycholinguistics 
650 0 4 |a Psycholinguistics 
650 0 4 |a quantitative analysis 
650 0 4 |a retina image 
650 0 4 |a transfer of learning 
650 0 4 |a vision 
650 0 4 |a visual cortex 
650 0 4 |a visual cortex 
650 0 4 |a Visual Cortex 
650 0 4 |a Visual Perception 
650 0 4 |a young adult 
650 0 4 |a Young Adult 
700 1 |a Chen, H.  |e author 
700 1 |a Cheng, K.  |e author 
700 1 |a Huang, W.  |e author 
700 1 |a Li, C.  |e author 
700 1 |a Li, C.  |e author 
700 1 |a Li, J.  |e author 
700 1 |a Wang, C.  |e author 
700 1 |a Wang, Y.  |e author 
700 1 |a Yan, H.  |e author 
700 1 |a Zuo, Z.  |e author 
773 |t Human Brain Mapping