高级检索

大语言模型的中文多轮指令遵循能力评测研究

Evaluation on Chinese Multiturn Instruction Following Capability of Large Language Models

  • 摘要: 该文提出了一种高效评测中文大语言模型(LLM)指令遵循能力和多轮对话能力的方法,并构建了中文多轮指令遵循基准(Chinese Multiturn Instruction Following Benchmark,CMIF)。该文研究设计了专门针对中文的原子指令数据集,涵盖语言结构、拼音、音调等特性,并结合规则与LLM对多轮问题的合法性进行复查,确保评测结果的准确性。在实验中,选取了包括GPT4o和Qwen2.5-72B-Instruct在内的14个开源及闭源模型进行评估。结果显示,主流模型在单轮对话场景中具有较好的指令遵循能力,但多轮对话表现仍有较大提升空间。其中,单轮指令级准确率最高的Claude-3.5-Sonnet在多轮场景下准确率从73.8%下降至40.0%。此外,这些模型在处理中文原子指令时表现出明显的性能下降,中文任务的综合准确率最高仅为51.0%,显著低于其他四类任务平均79.0%的综合准确率。

     

    Abstract: This study proposes an efficient method to evaluate the instruction-following and multiturn dialogue capabilities of Chinese large language model (LLM). It introduces the Chinese Multiturn Instruction Following Benchmark (CMIF). A dataset of atomic instructions tailored to Chinese is constructed, covering language structures, pinyin, and tones. Rules and LLM are employed to verify the validity of multiturn questions, ensuring accurate evaluation results. Fourteen open-source and closed-source models, including GPT4o and Qwen2.5-72B-Instruct, are evaluated. Results show that while most models perform well in single-turn scenarios, their performance in multiturn settings declines significantly. Claude-3.5-Sonnet achieves the highest single-turn instruction-level accuracy, but in multiturn scenarios, its accuracy drops from 73.8% to 40.0%. Additionally, the models demonstrate notable performance degradation when processing Chinese-specific atomic instructions. The highest overall accuracy for Chinese tasks, achieved by GPT4o, is only 51.0%, far below the 79.0% average across other task categories.

     

/

返回文章
返回