Research on Three Key Issues of Multimodal Emotion Recognition Technology in Service Robotics Context
DOI:
https://doi.org/10.62051/jhhx1w14Keywords:
Service robotics; Multimodal emotional recognition; Zero-sample learning; Sensor-less emotion recognition technology; Macrolanguage model.Abstract
With the rapid development of artificial intelligence technology, service robots are increasingly integrating into daily life scenarios. Achieving efficient and natural human-robot interaction has become a key challenge. The ability to recognize emotions, as a core factor in enhancing the interaction experience, has attracted extensive attention from both the academic and industrial communities. This paper focuses on the emotion recognition problem of service robots, analyzes three key problems that need to be solved by the application of multimodal emotion recognition technology in service robots, lists and analyzes several advanced machine learning algorithms that can help solve the problem, evaluates and analyzes them respectively, and summarizes their common limitations as well as the possible improvement methods in the future. This research not only provides crucial theoretical support and algorithmic paths for the emotional computing of service robots, but also helps to build a more reliable human-robot collaboration relationship. It holds significant application value for promoting the real implementation of service robots in real-life scenarios.
Downloads
References
[1] Bhati V S, Tiwari N, Chawla M. A generalized zero-shot deep learning classifier for emotion recognition using facial expression images. IEEE Access, 2025, 13: 18687 - 18700. DOI: https://doi.org/10.1109/ACCESS.2025.3533580
[2] Qi F, Zhang H Z, Yang X, et al. A versatile multimodal learning framework for zero-shot emotion recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34 (7): 5728 - 5741. DOI: https://doi.org/10.1109/TCSVT.2024.3362270
[3] Gandhi A, Adhvaryu K, Poria S, et al. Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions. Information Fusion, 2023, 91: 424 - 444. DOI: https://doi.org/10.1016/j.inffus.2022.09.025
[4] Sedehi J F, Dabanloo N J, Maghooli K, et al. Multimodal insights into granger causality connectivity: Integrating physiological signals and gated eye-tracking data for emotion recognition using convolutional neural network. Heliyon, 2024, 10 (16): e36411. DOI: https://doi.org/10.1016/j.heliyon.2024.e36411
[5] Guo Z E, Yang M Q, Lin L, et al. E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition. PeerJ Computer Science, 2024, 10: e1977.
[6] Wang Z, Wang Y H. Emotion recognition based on multimodal physiological electrical signals. Frontiers in Neuroscience, 2025, 19: 1512799. DOI: https://doi.org/10.3389/fnins.2025.1512799
[7] Sun Y, Zhou Y, Zhang X. Speech emotion recognition fusing functional paralanguage proportion coefficient. Journal of Northeastern University (Natural Science), 2024, 45 (1): 40 - 48.
[8] Zhu T, Li L, Yang J, et al. Multimodal sentiment analysis with image-text interaction network. IEEE Transactions on Multimedia, 2023, 25: 3375 - 3385. DOI: https://doi.org/10.1109/TMM.2022.3160060
[9] Wen C S, Jia G L, Yang J F. DIP: dual incongruity perceiving network for sarcasm detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 2540 - 2550. DOI: https://doi.org/10.1109/CVPR52729.2023.00250
[10] Ghandeharioun A, McDuff D, Czerwinski M, et al. EMMA: an emotion-aware wellbeing chatbot. In: Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction, 2019: 1 - 7. DOI: https://doi.org/10.1109/ACII.2019.8925455
[11] Zhao S C, Feng Y F, Zhang Z C, et al. Research advancements on emotionally and intellectually integrated digital humans and robotics. Journal of Image and Graphics, 2025, 30 (6): 2139 - 2160. DOI: https://doi.org/10.11834/jig.240780
[12] Zhao S, Jia Z, Chen H, et al. PDANet: polarity-consistent deep attention network for fine-grained visual emotion regression. In: Proceedings of the 27th ACM International Conference on Multimedia, 2019: 192 - 201. DOI: https://doi.org/10.1145/3343031.3351062
[13] Zhang Y, Xie D, Luo D, et al. Modality emotion semantic correlation analysis for multimodal emotion recognition. Computers and Electrical Engineering, 2025, 126: 110467. DOI: https://doi.org/10.1016/j.compeleceng.2025.110467
[14] Guo Z, Yang M, Lin L, et al. E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition. PeerJ Computer Science, 2024, 10: e1977. DOI: https://doi.org/10.7717/peerj-cs.1977
[15] Lian Z, Sun L, Sun H, et al. GPT-4V with emotion: a zero-shot benchmark for generalized emotion recognition. Information Fusion, 2024, 108: 102367. DOI: https://doi.org/10.1016/j.inffus.2024.102367
[16] Chen Y, Wang H, Yan S, et al. Emotion Queen: a benchmark for evaluating empathy of large language models. [EB/OL], 2025. DOI: https://doi.org/10.18653/v1/2024.findings-acl.128
[17] Cheng Z, Cheng Z Q, He J Y, et al. Emotion-LLaMA: multimodal emotion recognition and reasoning with instruction tuning. 2025. DOI: https://doi.org/10.52202/079017-3518
[18] Qu Y, Mang Y, Du B. EmoLLM: multimodal emotional understanding meets large language models. 2025.
[19] Zhang Z, Peng L, Pang T, et al. Refashioning emotion recognition modeling: the advent of generalized large models. IEEE Transactions on Computational Social Systems, 2024, 11 (5): 6690 - 6704. DOI: https://doi.org/10.1109/TCSS.2024.3396345
[20] Rui T, Cui P, Zhu W. Joint user-interest and social influence emotion prediction for individuals. Neurocomputing, 2017, 230: 66 - 76. DOI: https://doi.org/10.1016/j.neucom.2016.11.054
[21] Zhao Z P, Zheng Y, Zhang Z X, et al. Exploring spatio-temporal representations by integrating attention based bidirectional-LSTM-RNNs and FCNs for speech emotion recognition. In: Proceedings of Interspeech 2018, 2018: 272 - 276. DOI: https://doi.org/10.21437/Interspeech.2018-1477
[22] Molla Hosseini A, Hasani B, Mahoor M H. AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 2019, 10 (1): 18 - 31. DOI: https://doi.org/10.1109/TAFFC.2017.2740923
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Transactions on Computer Science and Intelligent Systems Research
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
