在自然语言处理领域各项任务中,模型广泛存在性别偏见。然而,当前尚无中文性别偏见评估和消偏的相关数据集,因此无法对中文自然语言处理模型中的性别偏见进行评估。首先,该文根据16对性别称谓词,从一个平面媒体语料库中筛选出性别无偏的句子,构建了一个含有20 000条语句的中文句子级性别无偏数据集SlguSet(Sentence-Level Gender Unbiased Dataset)。随后,该文提出了一个可衡量预训练语言模型性别偏见程度的指标,并对5种流行的预训练语言模型中的性别偏见进行评估。结果表明,中文预训练语言模型中存在不同程度的性别偏见,该文所构建数据集能够很好地对中文预训练语言模型中的性别偏见进行评估。
Abstract
In various tasks in the field of natural language processing, models are widely gender biased. However, there is no relevant dataset for Chinese gender bias assessment and debiasing. According to 16 pairs of gender appellations, this paper screened out gender-unbiased sentences from a print media corpus, and constructed a Chinese sentence-level gender-unbiased data set SlguSet(sentence-level gender unbiased dataset) containing 20,000 sentences. This paper further proposes an index that can measure the degree of gender bias in pre-trained language models, and evaluates the gender bias in five popular pre-trained language models. The results show that there are different degrees of gender bias in the Chinese pre-training language model, and the dataset constructed in this article can effectively evaluate the gender bias in the Chinese pre-training language model.
关键词
性别偏见 /
数据集 /
预训练语言模型
{{custom_keyword}} /
Key words
gender bias /
dataset /
pre-training language model
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] SUN T,GAUT A, TANG S, et al. Mitigating gender bias in natural language processing: Literature review[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,2019: 1630-1640.
[2] 朱述承, 苏祺, 刘鹏远. 基于语料库的我国职业性别无意识偏见共时历时研究[J]. 中文信息学报, 2021, 35(5): 130-140.
[3] BOLUKBASI T, CHANG K W, ZOU J Y, et al. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings[J]. Advances in Neural Information Processing Systems, 2016, 29: 4349-4357.
[4] KURITA K, VYAS N, PAREEK A, et al. Measuring bias in contextualized word representations[C]//Proceedings of the 1st Workshop on Gender Bias in Natural Language Processing, 2019: 166-172.
[5] FONT J E, COSTA-JUSS M R. Equalizing gender bias in neural machine translation with word embeddings techniques[C]//Proceedings of the 1st Workshop on Gender Bias in Natural Language Processing, 2019: 147-154.
[6] WEBSTER K,RECASENS M, AXELROD V, et al. Mind the gap: A balanced corpus of gendered ambiguous pronouns[J]. Transactions of the Association for Computational Linguistics, 2018, 6: 605-617.
[7] ZHAO J, WANG T,YATSKAR M, et al. Gender bias in coreference resolution: Evaluation and debiasing methods[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018: 15-20.
[8] COSTA-JUSS M R, LIN P L, ESPAA-BONET C. GeBioToolkit: Automatic extraction of gender-balanced multilingual corpus of Wikipedia biographies[C]//Proceedings of the 12th Language Resources and Evaluation Conference, 2020: 4081-4088.
[9] MAY C, WANG A, BORDIA S, et al. On measuring social biases in sentence encoders[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 622-628.
[10] GARG N,SCHIEBINGER L, JURAFSKY D, et al. Word embeddings quantify 100 years of gender and ethnic stereotypes[J]. Proceedings of the National Academy of Sciences, 2018, 115(16): E3635-E3644.
[11] ZHAO J, WANG T,YATSKAR M, et al. Men also like shopping: Reducing gender bias amplification using corpus-level constraints[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2017: 2979-2989.
[12] HITTI Y, JANG E, MORENO I, et al. Proposed taxonomy for gender bias in text: A filtering methodology for the gender generalization subtype[C]//Proceedings of the 15t Workshop on Gender Bias in Natural Language Processing, 2019: 8-17.
[13] SUN T,GAUT A, TANG S, et al. Mitigating gender bias in natural language processing: Literature review[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019: 1630-1640.
[14] CALISKAN A, BRYSON J J, NARAYANAN A. Semantics derived automatically from language corpora contain human-like biases[J]. Science, 2017, 356(6334): 183-186.
[15] NADEEM M,BETHKE A, REDDY S. Stereoset: Measuring stereotypical bias in pretrained language models[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 5356-5371.
[16] PARK J H, SHIN J, FUNG P. Reducing gender bias in abusive language detection[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2018: 2799-2804.
[17] ZHANG B H, LEMOINE B, MITCHELL M. Mitigating unwanted biases with adversarial learning[C]//Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2018: 335-340.
[18] KIRITCHENKO S, MOHAMMAD S. Examining gender and race bias in two hundred sentiment analysis systems[C]//Proceedings of the 7th Joint Conference on Lexical and Computational Semantics, 2018: 43-53.
[19] GONEN H, GOLDBERG Y. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 609-614.
[20] KIRITCHENKO S, MOHAMMAD S. Examining gender and race bias in two hundred sentiment analysis systems[C]//Proceedings of the 7th Joint Conference on Lexical and Computational Semantics, 2018: 43-53.
[21] ZHU S, LIU P. Great males and stubborn females: a diachronic study of corpus-based gendered skewness in chinese adjectives[C]//Proceedings of the 19th Chinese National Conference on Computational Linguistics, 2020: 31-42.
[22] DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 4171-4186.
[23] CUI Y, CHE W, LIU T, et al. Pre-training with whole word masking for Chinese bert[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3504-3514.
[24] CUI Y, CHE W, LIU T, et al. Revisiting pre-trained models for Chinese natural language processing[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing: Findings, 2020: 657-668.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
北京市自然科学基金(4192057)
{{custom_fund}}
PDF(4211 KB)
