Multimodal Fake News Detection Based on Large Language Models and Multilevel Semantic Analysis

To identify the fake news, existing models fail to capture the core semantic information owing to the challenges in differentiating different layers of information in multimodal data. This paper proposes a multimodal fake news detection model based on multilevel semantic analysis, aiming to identify fake news by integrating shallow semantics, deep semantics and external semantic information. First, the model utilizes BERT and Swin-T models to encode the text and image data to extract shallow semantic information. Secondly, the cross-modal semantic features are captured by CLIP model and further processed using Transformer layer to obtain deep semantic information. Finally, the LLaVA and SDXL models are utilized to generate the corresponding text and image descriptions, which are encoded and processed to obtain the external semantic information. Experimental results on multiple datasets demonstrate the validness of our model.