高级检索

基于LLM和多级语义分析的多模态虚假新闻检测

Multimodal Fake News Detection Based on Large Language Models and Multilevel Semantic Analysis

  • 摘要: 在信息时代,有效识别并抑制虚假新闻的传播是维护社会稳定的关键措施。现有模型在处理多模态数据时,由于未对不同层级的信息进行区分处理,限制了对核心语义信息的深入挖掘。其次,大部分研究专注于评估LLM对检测模型的影响或使用LLM生成的文本进行实验,然而,在多模态虚假新闻检测任务中,仅依赖生成的文本数据远不足以应对复杂的检测需求。为此,该文提出了一种基于LLM和多级语义分析的多模态虚假新闻检测方法,通过融合浅层语义信息、深层语义信息和外部语义信息来识别虚假新闻。首先,模型利用BERT和Swin-T模型对文本和图像数据进行编码和处理以提取浅层语义信息。其次,通过CLIP模型捕获跨模态的语义特征,并利用Transformer层进一步处理,从而获取深层语义信息。最后利用LLaVA和SDXL模型生成对应的文本和图像描述,编码处理后得到外部语义信息,并利用多级融合策略处理语义信息。在多个数据集上的实验结果进一步验证了模型的优越性。

     

    Abstract: To identify the fake news, existing models fail to capture the core semantic information owing to the challenges in differentiating different layers of information in multimodal data. This paper proposes a multimodal fake news detection model based on multilevel semantic analysis, aiming to identify fake news by integrating shallow semantics, deep semantics and external semantic information. First, the model utilizes BERT and Swin-T models to encode the text and image data to extract shallow semantic information. Secondly, the cross-modal semantic features are captured by CLIP model and further processed using Transformer layer to obtain deep semantic information. Finally, the LLaVA and SDXL models are utilized to generate the corresponding text and image descriptions, which are encoded and processed to obtain the external semantic information. Experimental results on multiple datasets demonstrate the validness of our model.

     

/

返回文章
返回