Abstract: Natural language understanding models are prone to biases in datasets, which usually leads to poor generalizability. Previous methods believed that filtering biased parts from the text representations could eliminate bias. This paper argues that representation-calibration methods need to introduce a priori knowledge of bias in the feature space, which may not necessarily describe the bias reasonably. Therefore, this paper proposes to mitigate data bias from the perspective of the classifier, and presents a memory-guided multi-expert network that uses multiple classifiers to focus on different subspace features. In addition, the model dynamically allocates training steps for samples based on the classification results of experts at different times, improving the model's memorization of difficult samples. This method significantly improves Out-of-Distribution (OOD) performance at the cost of a small part of the In-Distribution (ID) performance on multiple benchmark datasets for natural language understanding tasks (MNLI, FEVER, and QQP).