Pre-training technology has stepped into the center stage of natural language processing, especially with the emergence of ELMo, GTP, BERT, XLNet, T5, and GTP-3 in the last two years. In this paper, we analyze and classify the existing pre-training technologies from four aspects: language model, feature extractor, contextual representation, and word representation. We discuss the main issues and development trends of pre-training technologies in current natural language processing.

Over the past decade, there has been a steady momentum of innovation and breakthroughs that convincingly push the limits of modeling single modality, e.g., vision, speech and language. Going beyond such research progresses made in single modality, the rise of multimodal social network, short video applications, video conferencing, live video streaming and digital human highly demands the development of multimodal intelligence and offers a fertile ground for multimodal analysis. This paper reviews recent multimodal applications that have attracted intensive attention in the field of natural language processing, and summarizes the mainstream multimodal fusion approaches from the perspectives of single modal representation, multimodal fusion stage, fusion network, fusion of unaligned modalities, and fusion of missing modalities. In addition, this paper elaborate the latest progresses of the vision-language pre-training.

Introducing knowledge into data-driven artificial intelligence models is an important way to realize human-machine hybrid intelligence. The current pre-trained language models represented by BERT have achieved remarkable success in the field of natural language processing. However, the pre-trained language models are trained on large scale unstructured corpus data, and it is necessary to introduce external knowledge to alleviate its defects in determinacy and interpretability to some extent. In this paper, the characteristics and limitations of two kinds of pre-trained language models, pre-trained word embeddings and pre-trained context encoders, are analyzed. The related concepts of knowledge enhancement are explained. Four types of knowledge enhancement methods of pre-trained word embeddings are summarized and analyzed, which are pre-trained word embeddings retrofitting, hierarchizing the process of encoding and decoding, attention mechanism optimization and knowledge memory introduction. The knowledge enhancement methods of pre-training context encoders are described from two perspectives: 1) task-specific and task-agnostic; 2) explicit knowledge and implicit knowledge. Through the summary and analysis of the knowledge enhancement methods of the pre-trained language model, the basic pattern and algorithm are provided for the human-machine hybrid artificial intelligence.

Named entity recognition (NER), as one of the basic tasks in natural language processing, aims to identify the required entities and their types in unstructured text. In recent years, various named entity recognition methods based on deep learning have achieved much better performance than that of traditional methods based on manual features. This paper summarizes recent named entity recognition methods from the following three aspects: 1) A general framework is introduced, which consists of an input layer, an encoding layer and a decoding layer. 2) After analyzing the characteristics of Chinese named entity recognition, this paper introduces Chinese NER models which incorporate both character-level and word-level information. 3) The methods for low-resource named entity recognition are described, including cross-lingual transfer methods, cross-domain transfer methods, cross-task transfer methods, and methods incorporating automatically labeled data. Finally, the conclusions and possible research directions are given.

Adversarial attack and defense is a popular research issue in recent years. Attackers use small modifications to generate adversarial examples to cause prediction errors from the deep neural network. The generated adversarial examples can reveal the vulnerability of the neural network, which can be repaired to improve the security and robustness of the model. This paper gives a more detailed and comprehensive introduction to the current mainstream adversarial text example attack and defense methods, the data set together with the target neural network of the mainstream attack. We also compare the differences between different attack methods in this paper. Finally, the challenges of the adversarial text examples and the prospect of future research are summarized.

The named entity recognition of traditional Chinese medicine books is a less addressed topic. Considering the difficulty and cost in annotating such professional text in classical Chinese, this paper proposes a method for identifying traditional Chinese medicine entities based on a combination of semi-supervised learning and rules. Under the framework of the conditional random fields model, supervised features such as lexical features and dictionary features are introduced together with the unsupervised semantic features derived from word vectors. The optimal semi-supervised learning model is gained by examining the performance of different feature combinations. Finally, the recognition results of the model are analyzed and a rule based post-processing is established with the linguistic characteristics of ancient books. Experiments results reveals 83.18% F-score, which proves the validity of this method.

Document AI, or Document Intelligence, is a relatively new research topic that refers to the techniques to automatically read, understand and analyze business documents. It is an important interdisciplinary study involving natural language processing and computer vision. In recent years, the popularity of deep learning technology has greatly advanced the development of Document AI tasks, such as document layout analysis, document information extraction, document visual question answering, and document image classification etc. This paper briefly introduces the early-stage heuristic rule-based document analysis, statistical machine learning based algorithms, as well as the deep learning-based approaches especially the pre-training approaches. Finally, we also look into the future direction of Document AI.

In the field of biomedical text mining, biomedical named entity recognition and relations extraction are of great significance. This paper builds a Chinese biomedical entity relation extraction system based on deep learning technology. Firstly, Chinese biomedical entity relation corpus is construction from the publicly available English biomedical annotated corpora via translation and manual annotation. Then this paper applies the ELMo (Embedding from Language Model) trained in Chinese biomedical text to the Bi-directional LSTM (BiLSTM) combined conditional random fields (CRF) model for Chinese entity recognition. Finally, the relation between entities is extracted using BiLSTM combined with the Attention mechanism. The experimental results show that the system can accurately extract biomedical entities and inter-entity relation from Chinese text.

Gender bias is a hot topic in sociology. In recent years, machine learning algorithms have learnt bias from data, which have arouse much more attention on this topic. Based on the markedness theory, this paper examines the unconscious gender bias of 63 occupations in BCC and DCC corpora from both synchronic and diachronic perspectives. Firstly, the gender preference of 63 occupations among different age and gender groups is investigated via questionnaires. There is a significant positive correlation between the questionnaire and the occupation gender bias word frequency indicators in the BCC corpus. Then, from the perspective of synchronic study, most of the occupations are found with a growing gender bias against women from the corpus of different fields in the BCC corpus, and the newspaper texts of the 31 provincial administrative units in the DCC corpus in 2018, There also are differences in occupational gender bias in different regions. Finally, from a diachronic perspective, it is found that the occupational gender unconscious bias phenomenon shows an overall weakening trend form the DCC corpus from 2005 to 2018 newspaper texts for statistical analysis.

Extracting relational triples is a basic task for large-scale knowledge graph construction. In order to improve the ability of extracting overlapped relation triples and multi-slot relation triples, this paper proposes BSLRel, an end-to-end relation triple extraction model based on neural network. Specifically, BSLRel model converts the relation triplet extraction task into a cascade binary sequence labeling task,which consists of a new multiple information fusion structure “Conditional Layer Normalization” to integrate information. With BSLRel, we participate in the “Relation Extraction” task organized by “the 2020 Language and Intelligence Challenge” and achieve Top 5 among all competitive models.

Natural Language Generation in a task-oriented dialogue system (ToDNLG) aims to generate natural language responses given the corresponding dialogue acts, which has attracted increasing research interest. With the development of deep neural networks and pre-trained language models, great success has been witnessed in the research of ToDNLG field. We present a comprehensive survey of the research field, including: (1) a systematical review on the development of NLG in the past decade, covering the traditional methods and deep learning-based methods; (2) new frontiers in emerging areas of complex ToDNLG as well as the corresponding challenges; (3) rich open-source resources, including the related papers, baseline codes and the leaderboards on a public website. We hope the survey can promote future research in ToDNLG.

Most of the existing recommendation methods based on deep reinforcement learning use recurrent neural network (RNN) to learn users' short-term preference, while ignoring their long-term preference. This paper proposes a deep reinforcement learning recommendation method combining both long-term and short-term user preference (LSRL). LSRL uses collaborative filtering to learn users' long-term preference representation and applies the gated recurrent unit (GRU) to learn user's short-term preference representation. The redesigned Q-network framework combines two types of representation and Deep Q-Network is used to predict users' feedback on items. Experimental results on MovieLens datasets show that the proposed method has a significant improvement according to NDCG and Hit Ratio compared to other baseline methods.

Question generation (QG) aims to automatically generate fluent and semantically related questions for a given text. QG can be applied to generate questions for reading comprehension tests in the education field, and to enhance question answering and dialog systems. This paper presents a comprehensive survey of related researches on QG. We first describe the significance of QG and its applications, especially in the education field. Then we outline the traditional rule-based methods on QG, and make a detailed description on the neural network based models from different views. We also introduce the evaluation metrics of generated questions. Finally, we discuss the limitations of previous studies and suggest future works.

Recently, the artificial intelligence-based dialogue system has been widely applied in, human-computer interaction, intelligent assistant, smart customer service, Q&A consulting, and so on. This paper proposes a definition about the task-oriented dialogue system, which is to satisfy the user’s requirements and certain tasks with the least response turns in dialogue between human and machine. Furthermore, three critical technical problems and challenges are summarized: the user’s intent detection in the complex context, the limitation annotated data, and the personalized response under the multi-modal situation. The research progress in these three challenges are discussed in the paper. Finally, we outline the research directions in the future and the key issues in the next generation of task-oriented dialogue system.

Sentiment analysis is a popular research issue in the field of natural language processing, and multimodal sentiment analysis is the current challenge in this task. Existing studies are defected in capturing context information and combining information streams of different models. This paper proposes a novel multi-LSTMs Fusion Model Network (MLFN), which performs deep fusion between the three modalities of text, voice and image via the internal feature extraction layer for single-modal, and the inter-modal fusion layer for dual-modal and tri-modal. This hierarchical LSTM framework takes into account the information features inside the modal while capturing the interaction between the modals. Experimental results show that the proposed method can better integrate multi-modal information, and significantly improve the accuracy of multi-modal emotion recognition.

Evaluation in natural language processing drives and promotes research on models and methods. In recent years, new evaluation data sets and evaluation tasks have been continuously proposed. At the same time, a series of problems exposed by such evaluations seems to restrict the progress of natural language processing technology. Starting from the concept, composition, development and significance of natural language Processing evaluation, this article classifies and summarizes the tasks and characteristics of mainstream natural language Processing evaluation, and then reveals the problems and their possible causes. In parallel to the human language ability evaluation standard, this paper puts forward the concept of human-like machine language ability evaluation, and proposes a series of basic principles and implementation ideas for human-like machine language ability evaluation from three aspects: reliability, difficulty and validity.

Although the pre-trained language model has achieved high performance on a large number of natural language processing tasks, the knowledge contained in some pre-trained language models is difficult to support more efficient textual inference. Focused on using a wealth of knowledge to enhance the pre-trained language model for textual inference, we propose a framework for textual inference to integrate the knowledge of graphs and graph structures into the pre-trained language model. Experiments on two subtasks of textual inference indicate our framework outperforms a series of baseline methods.

Text sentiment analysis is an important branch in the field of natural language processing. This paper proposes a text sentiment analysis capsule model that combines convolutional neural networks and bidirectional GRU networks. Firstly, the multi-head attention is used to learn the dependency between words and capture the emotional words in the text. Then, the convolutional neural network and bidirectional GRU network are used to extract emotional features of different granularities in the text. After the feature fusion, the global average pooling is used to get the instance feature representation of the text, and the attention mechanism is combined to generate feature vectors for each emotion category to construct an emotion capsule. Finally, the emotion category of the text is judged by the capsule attributes. Tested on the MR, IMDB, SST-5 and Tan Songbo hotel review datasets, the proposed model achieves better classification effect than other baseline models.

Graph neural networks(GNN) recently appears to be an effective method to model the global context representation of samples, but defected in over-smoothing when faced with the noisy few-shot text classification scenario. We propose a dual channel graph neural network to model the full context features while making full use of the label propagation mechanism. A multi-task parameter sharing mechanism is used in the dual channels to effectively constrain the graph iteration process. Compared with the baseline graph neural network, our method achieves an average improvement of 1.51% on the FewRel dataset and 11.1% improvement on the ARSC dataset.

Legal named entity recognition(LNER)is a fundamentaltask for the field of smart judiciary.This paper presents a new definition of LNER and a corpus of letters of proposal for prosecution named LegalCorpus. This paper proposes novel BERT based NER model for legal texts, named BERT-ON-LSTM-CRF (Bidirectional Encoder Representations from Transformers-Ordered Neuron-Long Short Term Memory Networks-Conditional Random Field). The proposed model utilizes BERT model to dynamically obtain the semantic vectors according to the context of words. Then the ONLSTM is adopted to extract the text features by modeling the input sequence and hierarchy. Finally, the text features are decoded by CRF to obtain the optimal tag sequence. Experiments show that the proposed model can achieve a F₁-value of 86.09%, with 7.8% increased than the best baseline Lattice-LSTM.

Event extraction (EE) refers to the technology of extracting events from natural language texts and identifying event types and event elements. This paper proposes an end-to-end multi-label pointer network for event extraction, in which the event detection task is integrated into the event element recognition task to extract event elements and event types at the same time. This method avoids the problem of wrong cascade and task separation in traditional pipeline methods, and alleviates the problem of role overlapping and element overlapping in event extraction. The proposed method achieves 85.9% F₁ score on the test set in 2020 Language and Intelligence Challenge Event Extraction task.

Implicit discourse relation recognition is a challenging task in that it is difficult to obtain semantic informative and interaction-informative argument representations. The paper proposes an implicit discourse relation recognition method based on the Graph Convolutional Network (GCN). With the arguments encoded by fine-tuned BERT, the GCN is designed by concatenating the argument representations as feature matrix, and concatenating the attention score matrixes as adjacent matrix. It is hoped that the argument representations can be optimized by the self-attention and inter-attention information to improve implicit discourse relation recognition. Experimental results on the Penn Discourse Treebank (PDTB) show that the proposed method outperforms BERT in recognizing the four of implicit discourse relations, and it outperforms the state-of-the-art methods on Contingency and Expansion with 60.70% and 74.49% on F₁ score, respectively.

Retweeting, a simple and powerful function, is the key mechanism of information diffusion on major microblog platforms. Retweeting behavior can help us understand the characteristics of information diffusion and better explore users' behaviors and interests, which is of significance in various applications such as information recommendation, emergency prevention and public opinion monitoring. This paper discusses various research work related to predicting retweeting behavior and retweet count, and summarizes the current challenges and future research direction. This paper reviews the research in the field of retweeting so as to provide a reference for future researchers.

There are a large amount of logical knowledge that portray the logical evolutionary relationships between things in the open texts. Logical knowledge bases are an important foundation to advancing the knowledge reasoning, the development of large-scale logical reasoning knowledge bases can help support conduction-driven decision-making tasks for entities or events. This paper presents an overview of the logical knowledge base, including categories and basic compositions. It also proposes a method for entity description and event causal logical knowledge extraction from the large-scale open text. Finally, a reliable interpretable path reasoning algorithm and financial entity influence generation system based on the logical reasoning knowledge base is explored for the financial domain. The algorithm model and system have achieved good results.

Humor recognition is a challenge in the field of natural language processing. According to humor theories and cognitive linguistics, five types of distinctive aspects of humor are systematically analyzed, and a variety of humor features are derived. The experiment results show that the proposed features can better represent the latent semantic information of humor. Furthermore, the deep learning can benefit from these features for humor recognition.

Path selection, as a key step in the Knowledge Base Question Answering (KBQA) task, relies on the the semantic similarity between a question and candidate paths. To deal with massive unseen relations in the test set, a method based on dynamic sampling of negative examples is proposed to enrich the relations in the training set. In the prediction phase, two path pruning methods, i.e., the classification method and the beam search method, are compared to tackle the explosion of candidate paths. On the CCKS 2019-CKBQA evaluation data set containing simple and complex problems, the proposed method achieves an average F₁ value of 0.694 for the single-model system, and 0.731 for the ensemble system.

In recent years, legal artificial intelligence has attracted increasing attention for its efficiency and convenience. Among others, legal text is the most common manifestation in legal practice, thus, using natural language understanding method to automatically process legal text is an important direction for both academia and industry. In this paper, we provide a gentle survey to summarize recent advances on natural language understanding for legal texts. We first introduce the popular task setups, including legal information extraction, legal case retrieval, legal question answering, legal text summarization, and legal judgement prediction. We further discuss the main challenges from three perspectives: understanding the difference of languages between legal domain and open domain, understanding the rich argumentative texts in legal documents, and incorporating legal knowledge into existing natural language processing models.

To deal with model reconstruction process and the lack of training data for various domains in the task of named entity recognition, a domain adaptive named entity recognition method is proposed based on attention mechanism. Firstly, a bidirectional long-short term memory conditional random field named entity recognition model based on the BERT (BERT-BiLSTM-CRF)is constructed on the general dataset. Then, such-bulit model is fine-tuned using the ancient Chinese corpus, with an adaptive neural network layer based on the attention mechanism inserted. The comparison experiment is set with the model in the target domain and the existing transfer learning method. The experimental results show that the proposed model improves the F₁ value by 4.31% compared with the generic domain BERT-BiLSTM-CRF model, by 2.46% compared with the same model trained only on the ancient Chinese domain corpus.

Domain specific corpora such as Tibetan animal husbandry corpus are rich in direct transliteration or synthesis of unknown words. To improve the word segmentation for such corpora, this paper proposes a Chinese word segmentation model via Multi-Head Attention. To capture the dependence relationship and syncopation point information between key character vectors, the Multi-Head Attention mechanism is applied to calculate the correlation between important character vectors and other character vectors in parallel regardless the distance between them. Then the conditional random fields is employed to model lexeme labels for the optimal word segmentation sequence. Finally, a domain dictionary is constructed to further improve the effect of word segmentation. Experiments on the corpus of animal husbandry in Tibet show that, compared with classical models such as Bi-LSTM-CRF, the accuracy, recall rate and F₁ value of the proposed model are increased by 3.93%, 5.3% and 3.63%, respectively.

Text correction, an important research field in Natural Language Processing (NLP), is of great application value in fields such as news, publication, and text input . This paper provides a systematic overview of automatic error correction technology for Chinese texts. Errors in Chinese texts are divided into spelling errors, grammatic errors and semantic errors, and the methods of error correction for these three types are reviewed. Moreover, datasets and evaluation methods of automatic error correction for Chinese texts are summarized. In the end, prospects for the automatic error correction for Chinese texts are raised.

In aspect-based sentiment classification, the attention mechanism is often combined in recurrent neural network or convolutional neural network to obtain the importance of different words. However, such kind of methods fail to capture long-range syntactic relations that are obscure from the surface form, which would be beneficial to identify sentiment features directly related to the aspect target. In this paper, we propose a novel model named MemGCN to explicitly utilize the dependency relationship among words. Firstly, we employ the memory network to obtain the context-aware memory representation. After that, we apply graph convolutional network over the dependency tree to propagate sentiment features directly from the syntactic context of an aspect target. Finally, the attention mechanism is used to fuse memory and syntactic information. Experiment results on SemEval 2014 and Twitter datasets demonstrate our model outperforms baseline methods.

Chinese named entity recognition (CNER) is one of the basic tasks of natural language processing applications such as question answering systems, machine translation and information extraction. Although traditional CNER system has achieved satisfactory experiment results with the help of manually designed domain-specific features and grammatical rules, it is still defected in aspects such as weak generalization ability, poor robustness and difficult maintenance. In recent years, deep learning techniques have been adopted to deal with the above shortcomings by automatically extracting text features in an end-to-end learning manner. This article surveys the recent advances of deep learning-based CNER. It first introduces the concepts, difficulties and applications of CNER, and introduces the common datasets and evaluation metrics. Recent neural network models for the CNER task are then grouped according to their network architectures, and representative models in each group are detailed. Finally, future research directions are discussed.

Clinical entity normalization is an indispensable part of medical statistics. In practice, a standard clinical term entity has several kinds of colloquialisms and non-standardized mentions, and for some applications such as the a clinical knowledge base construction, how to normalize these mentions is an issue that has to address. This paper is focused on the Chinese clinical entity normalization, i.e., linking non-standard Chinese clinical entity to the standard words which are in the given clinical terminology base. Specifically, we treat the clinical entity normalization task as a translation task, and employ a deep learning model to generate the core semantics of the clinical mentions and obtain the candidate set of the standard entity. The final standard words were obtained by re-ranking the candidate set by using a BERT-based semantic similarity model. Experiments on the data of the 5th China Conference on Health Information Processing (CHIP2019) achieve good results.

The grammatical structure of natural language text consists of words, phrases, sentences, clause complexes and texts. This paper re-examines the definition of sentences in linguistics and the segmentation of sentences in natural language processing, and puts forward the task of Chinese sentence segmentation. Based on the theory of clause complex, the sentence is defined as the smallest topic self-sufficient punctuation sequence, and a sentence boundary recognition model based on BERT is designed and implemented. The experimental results show that the accuracy and F₁ value of the model are 88.37% and 83.73%, respectively, much better than that of mechanical segmentation according to punctuation marks.

Whether and how human brains can master new grammar rules has been hotly debated in linguistic research. There is a lack of consensus regarding what the most important factors are in grammar rule learning (e.g., age of acquisition and amount of input) and their influences. This question yielded the current study, which utilized Artificial Grammar Learning (AGL) paradigm and Event Related Potentials (ERPs) to examine longitudinal changes in the neural mechanism underlying processing artificial grammar among adult Mandarin native speakers. We manipulated the amount of the input, and created three artificial grammars, each featuring a different level of similarity to the Chinese Mandarin grammar. The results showed that (a) within the framework of small data learning, adults can use unsupervised learning to master new grammar rules; (b) different grammar rules can be acquired with a relatively small amount of input and processed to a native-like level; and (c) grammar rules are acquired through competitive interactions between brain mechanisms. These findings contribute to learning theories using AGL paradigm and inform future research on Natural Language Processing.

The 6th China conference on Health Information Processing (CHIP 2020) organized six shared tasks in Chinese medical information processing. The second task was entity and relation extraction that automatically extracts the triples consist of entities and relations from Chinese medical texts. A total of 174 teams signed up for the task, and eventually 17 teams submitted 42 system runs. According to micro-average F₁ which was the key evaluation criterion in the task, the top performance of the submitted results reaches 0.648 6.

As a branch of the dialogue system, open domain dialogue has a good prospect in application. Different from task-based dialogue, it has strong randomness and uncertainty. This paper reviews the researches on open domain dialogue from the perspective of reply method, focusing on the application and improvement of sequence-to-sequence model in dialogue generation scenarios. The researches exhibit a clear clue from single-round dialogue to multi-round dialogue, and we further reveal that the sequence to sequence generation model has some problems that the characteristics of the model implementation and the application scenarios do not exactly match in the multi-round dialogue generation. Finally, we explore the possible improvements for the generation of multi-round dialogues from introducing external knowledge, introducing rewriting mechanism and introducing agent mechanism.

As a cross-domain research task, depression detection using multimodal information has recently received considerable attention from researchers in several communities, such as natural language processing, computer vision, and mental health analysis. These studies mainly utilize the user-generated contents on social media to perform depression detection. However, existing approaches have difficulty in modeling long-range dependencies(global information). Therefore, how to obtain global user information has become an urgent problem. In addition, considering that social media contains not only textual but also visual information, how to fuse global information in different modalities has become another urgent problem. To overcome the above challenges, we propose a multimodal hierarchical dynamic routing approach for depression detection. We obtain global user information from hierarchical structure and use dynamic routing policy to fuse text and image modalities which can adjust and refine message to detect depression. Empirical results demonstrate the impressive effectiveness of the proposed approach in capturing the global user information and fusing multimodal information to improve the performance of depression detection.

Knowledge representation learning, which aims to encode entities and relations into a dense, real-valued and low-dimensional semantic space, has drawn massive attention in natural language processing tasks, such as relation extraction and question answering. To better capture the neighbor information, we propose a model named TransE-NA (Neighborhood Aggregation on TransE) based on TransE, which determines the number of neighbors according to sparse degrees of entities and then aggregates the most relevant attributes of neighbors according to the corresponding relations. Experimental results on link prediction and triplet classification show that our approach outperforms baselines, alleviating the data sparsity issue and improving the performance effectively.

The combination of artificial intelligence with law has become a hot research issue. Focused on the machine reading comprehension task of China AI Law Challenge 2020 (CAIL2020), this paper proposes a multi-task joint training of four sub-modules: word embedding module, answer extraction module, answer classification module and supporting facts discrimination module. This paper proposes a data augmentation method based on TF-IDF ‘question-context’ similarity matching, which re-labels the training set of CAIL2019 for data augmentation. After performing CAIL2020 machine reading comprehension task, the F₁ value of this model achieves 74.49 as the first place in this task.