高级检索

基于质量感知去噪融合与模态平衡的多模态情感识别

Quality-aware Denoising Fusion with Modality Balancing for Multimodal Emotion Recognition

  • 摘要: 多模态情感识别旨在通过利用语言、视觉和声学等多种模态来预测人类情感。现有方法通常在原始特征上直接进行交互或融合,缺乏对噪声信息的显式建模与过滤,容易导致冗余信息放大和模态权重分配失衡。为了解决这些问题,该文提出了一种质量感知去噪融合与模态平衡的多模态情感识别框架(Quality-aware Denoising Fusion with Modality Balancing, QDFMB),该框架首先通过语义组件解耦将各模态表征拆分为有用信息与噪声成分,基于质量感知的选择性吸收与重组机制对跨模态信息进行去噪融合。随后,引入路由机制,自适应调整融合时的模态权重。通过两者协同作用,实现了噪声抑制与各模态信息的高效利用。在CMU-MOSI、CMU-MOSEI与CH-SIMS数据集上的综合实验结果表明,QDFMB达到了与最先进方法相当或更优的性能。

     

    Abstract: Multimodal Emotion Recognition (MER) aims to predict human emotions by leveraging multiple modalities, including language, vision, acoustic. Existing methods typically perform interaction or fusion directly upon raw features without explicit modeling and filtering of noisy information This paper proposes a Quality-aware Denoising Fusion framework with Modality Balancing (QDFMB). Specifically, the framework first decomposes each modality's representation into informative and noisy components via semantic disentanglement, enabling quality-aware selective absorption and recombination for denoising-based cross-modal fusion. A routing mechanism is introduced to adaptively adjust modality weights during fusion, which contributes to the effective noise suppression and efficient utilization of each modality. The comprehensive experimental results on the CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets demonstrate that QDFMB achieves performance comparable to or better than state-of-the-art methods.

     

/

返回文章
返回