| Summary: | Emotion emerges from the coordinated interplay between central neural activity and peripheral autonomic responses; however, many existing deep learning approaches model these modalities independently, limiting their ability to capture complementary affective information and to remain robust under small-dataset and partially missing-data conditions. To address this issue, this study presents EmoRecog-Net, a unified multimodal framework for video-induced emotion categorization and recognition using functional magnetic resonance imaging (fMRI) and physiological signals. The proposed architecture integrates convolutional neural networks to learn spatial representations, gated recurrent units to model temporal dynamics, and a multilayer perceptron to perform nonlinear cross-modal fusion. Experiments were conducted on the ICBHI 2024 multimodal benchmark, which includes synchronized fMRI, photoplethysmography (PPG), and respiration (R) signals. Model performance was evaluated using F1-score, area under the curve (AUC), accuracy, and the official Finalscore metric. The proposed method achieved the lowest Finalscore of 0.2815 among the compared methods on the benchmark, outperforming representative CNN-LSTM, attention-based, graph-based, and autoencoder-based baselines. In addition, ablation studies demonstrated that GRU-based temporal modeling and MLP-based multimodal fusion consistently improved performance under limited-data and class-imbalanced conditions. These results indicate that jointly modeling coupled brain-autonomic dynamics provides a robust and effective strategy for multimodal affective computing. Overall, EmoRecog-Net offers a compact yet effective design for emotion decoding in small-scale multimodal settings.
|
|---|
