Abstract: Speech Complex Emotion Detection (SCED) aims to recognize all emotion categories and their intensities in speech. Compared with traditional Speech Emotion Recognition (SER), which only recognizes the main emotion categories, SCED can capture the speaker’s emotional state comprehensively. A challenge for this task is the roughness of manual labels, such as commonly used one-hot labels that ignore non-primary emotion categories and emotion intensity information. However, annotating complex emotion labels requires extensive human effort. In order to automatically mine emotion components from rough labels, this paper proposes a Cross-modal Emotion Alignment Transformer (EAT) model as a label enhancement component of SER models. EAT generates a more precise complex emotion distribution by cross-modal alignment of the semantic correlation between the label’s textual emotion features and the speech’s acoustic emotion features. This distribution replaces one-hot labels, thus guiding the SER model to capture multiple emotion categories and their intensities. We verify the effectiveness of the proposed method on the Chinese dataset CNSCED and demonstrate its generalization in the traditional SER task on two English datasets, IEMOCAP and RAVDESS.