基于自监督表征蒸馏的Whisper低资源语音识别优化方法

Optimizing Whisper for Low-resource Speech Recognition via Self-supervised Representation Distillation

摘要: Whisper是一种强大的多语言语音识别模型，在英语等高资源语言上表现优异，但在缅甸语等部分低资源语言上性能仍受限于预训练数据的不足。为此，该文提出了一种基于自监督表征蒸馏的Whisper低资源语音识别优化方法。通过跨模型表征蒸馏机制，实现自监督模型表征向Whisper编码器的知识迁移，提升对缅甸语等语言的表征建模能力。实验结果表明，在缅甸语、柬埔寨语、乌兹别克语和旁遮普语四种语言上进行的实验表明所提方法有效提升了识别效果，分别实现了4.8%、6.5%、6.0%和6.3%的字符错误率降低，验证了所提方法的有效性。

Abstract: Whisper is a powerful multilingual automatic speech recognition model and performs well on high-resource languages such as English. Its accuracy on some low-resource languages, including Burmese, remained limited because of insufficient pretraining data. This study proposes a low-resource ASR optimization method based on self-supervised representation distillation. A cross-model representation distillation mechanism is used to transfer knowledge from a self-supervised speech model to the Whisper encoder. Experiments on Burmese, Khmer, Uzbek, and Punjabi achieves the character error rate reduction of 4.8%, 6.5%, 6.0%, and 6.3%, respectively.