Abstract:
This study proposes an efficient method to evaluate the instruction-following and multiturn dialogue capabilities of Chinese large language model (LLM). It introduces the Chinese Multiturn Instruction Following Benchmark (CMIF). A dataset of atomic instructions tailored to Chinese is constructed, covering language structures, pinyin, and tones. Rules and LLM are employed to verify the validity of multiturn questions, ensuring accurate evaluation results. Fourteen open-source and closed-source models, including GPT4o and Qwen2.5-72B-Instruct, are evaluated. Results show that while most models perform well in single-turn scenarios, their performance in multiturn settings declines significantly. Claude-3.5-Sonnet achieves the highest single-turn instruction-level accuracy, but in multiturn scenarios, its accuracy drops from 73.8% to 40.0%. Additionally, the models demonstrate notable performance degradation when processing Chinese-specific atomic instructions. The highest overall accuracy for Chinese tasks, achieved by GPT4o, is only 51.0%, far below the 79.0% average across other task categories.