基于解题思维机制的大语言模型多项选择阅读理解增强框架
Facilitating Large Language Model-based Multiple-choice Reading Comprehension with Question-solving Thought
-
摘要: 多项选择阅读理解任务目标是要求模型基于给定的文章和问题从候选选项中选择正确答案。已有工作表明,尽管大语言模型在该项任务上已经达到最好性能,但其缺乏激活选项相关的思维方式,难以正确感知到给定文章、问题和选项之间的深层联系并充分利用候选选项信息,故未能达到预期的性能。为此,该文提出一种基于解题思维机制的大语言模型多项选择阅读理解增强框架。具体地,该文考虑两种解题思维: 按步分析和选项评估,分别从按步骤求解以获取正确选项和深入验证选项正误两方面提升大语言模型对于多项选择阅读理解题型的解题能力。为了合理利用这两种解题思维,该文进一步提出了自我激活和教师蒸馏两种提升策略,进一步提升大模型的解题效果。实验结果表明,该文所提方法相对于已有基准在六个多项选择阅读理解数据集上的平均准确率和平均严格准确率分别提升了22.61%和28.95%。Abstract: The goal of the Multiple-Choice Reading Comprehension (MCRC) task is to require a model to select the correct answer from a set of candidate options based on the given paragraph and question. Related work shows that although large language models (LLM) have achieved state-of-the-art performance on this task, the failure to activate option-related thought prevents them from correctly perceiving the deep connections between the passage, question, and options. To this end, we propose a LLM enhancement framework for MCRC based on Question-Solving Thought (QST). Specifically, two types of mechanism are considered in our work: step-by-step and option-by-option. Two methods are further proposed to better utilize these QSTs: Self Activate (SA) and Distill from Teacher (DT), which further boost the question-solving of the model. Experimental results demonstrate that our proposed approaches increase average accuracy and average accuracy-hard by 22.61% and 28.95% respectively, compared to existing baselines across six multiple-choice reading comprehension datasets.
下载: