Exploring Semantic-aware Discrete Diffusion Model via Self-prompting Multimodal LLMs for Image Captioning

In recent years, non-autoregressive image captioning has gained significant attention due to its capabilities in bidirectional propagation and parallel word generation. Meanwhile, considerable progress has been made in research on discrete diffusion-based approaches. However, during the processes of discrete noise addition and denoising, existing methods still face critical challenges such as weak image-text relevance, object omission, inaccurate descriptions, and word repetition. To address these issues, this paper proposes a semantic-aware discrete diffusion model. This model incorporates a learnable query mechanism to construct a semantic perception module, which captures latent correlations with object-level semantic features in images. Building upon this foundational model, we further introduce a self-prompting optimization framework that leverages large language models to generate richer descriptions that better align with image details. Comprehensive experiments on the COCO dataset demonstrate that our method achieves notable improvements in image captioning tasks and outperforms existing approaches.