2021 Volume 35 Issue 3
Published: 16 April 2021
-
Select all|
-
Survey2021, 35(3): 1-23.In recent years, neural networks have gradually overtaken classical machine learning models and become the de facto paradigm for natural language processing tasks. Most typical neural networks are capable of dealing with data in Euclidean space. Due to the linguistic nature, however, the language information such as discourse and syntactic information is of graph structures. Therefore, there has been an increasing number of researches that use graph neural networks to explore structures in natural languages. This paper systematically introduces applications of graph neural networks in natural language processing areas. It first discusses the fundamental concepts and introduces three main categories of graph neural networks, namely graph recurrent neural network, graph convolutional network, and graph attention network. Then this paper introduces methods to construct proper graph structures according to different tasks, and to apply graph neural networks to embed those structures. This paper suggests that compared with focusing on novel structures, exploring how to use the key information in specific tasks to create corresponding graphs is more universal and is of more academic value, which can be a promising future research direction.
-
Survey2021, 35(3): 24-42.In recent years, the data-driven end-to-end chatbots have developed rapidly and attracted widespread attentions from both industrial and academic circle. This paper reviews existing automatic evaluation methods for generative model based chatbots. Firstly, the research background and the state-of-the-art of the automatic evaluation methods are introduced. Then the automatic evaluation methods for the basic ability of chatbots to generate reasonable responses are presented, revealing the advantages and disadvantages of each type of methods. The automatic evaluation methods for the expansion ability of chatbots are also introduced, involving the generation of various responses, the dialogue with specific personality or emotion, and the conversation topic in depth and breadth. Besides, the evaluation methods to evaluate the comprehensive ability of chatbots are also addressed together the development direction. After reviewing the method to evaluate automatic chatbots evaluation metrics, this paper finally discusses the challenges and future development trends to develop automatic evaluation methods.
-
Machine TranslationChinese-Vietnamese Unsupervised Neural Machine Translation Based on EMD Minimal Bilingual Dictionary2021, 35(3): 43-50.Neural machine translation (NMT) has achieved good results in tasks with sufficient parallel corpora, but often has poor results in translation tasks with scarce resources. To address NMT between Chinese and Vietnamese without large-scale parallel corpus, we explore the use of easily available Chinese and Vietnamese monolingual corpora by mining cross-language information at the word level. A Chinese-Vietnamese unsupervised neural machine translation method that incorporates Earth Mover's Distance(EMD) to minimize bilingual dictionaries is proposed. First, monolingual word embeddings for Chinese and Vietnamese are trained independently, and a Chinese-Vietnamese bilingual dictionary is obtained by minimizing their EMD. The dictionary is then used as a seed dictionary to train the Chinese-Vietnamese bilingual word embeddings. Finally, the shared encoder unsupervised machine translation model is applied to construct a Chinese-Vietnamese unsupervised neural machine translation. Experiments show that this method can effectively improve the performance of Chinese-Vietnamese unsupervised neural machine translation.
-
Machine Translation2021, 35(3): 51-59.Traditionally, neural machine translation relies on large-scale bilingual parallel corpora. In contrast, unsupervised neural machine translation avoids the dependence on bilingual corpora by generating pseudo-parallel data, whose quality plays a decisive role in the model training. To ensure the final quality of machine translation, we propose an unsupervised neural machine translation model using quality estimation to control the quality of pseudo-parallel data generated. Specifically, in the process of back-translation, we use quality estimation to score the generated pseudo-parallel data, and then select parallel data with higher score (HTER) to train the neural network. Compared with the baseline system, the BLEU scores are increased by 0.79 and 0.55, respectively, on WMT 2019 German-English and Czech-English monolingual news corpora.
-
Machine Translation2021, 35(3): 60-68,77.Neural machine translation (NMT) achieves the state-of-the-art performance and becomes the mainstream of machine translation. Yet it remains an open issue how much semantic information the NMT encoders could learn from source sentences. To address this problem, we propose to capture and measure source semantics in both word-level and sentence-level via the semantic formalism of Abstract Meaning Representation (AMR). Then we attempt to improve NMT performance by encoding the captured source semantics. Experimental results indicate that NMT encoders can learn certain amount of both word-level and sentence-level semantics, as far as enhancing source-side semantics in NMT encoder can boost NMT translation performance for small training dataset.
-
Information Extraction and Text Mining2021, 35(3): 69-77.Relation classification is an important semantic processing task in the field of natural language processing. With the development of machine learning technology, the pre-trained model BERT has achieved excellent results in many natural language processing tasks. This paper proposes a relation classification method (EC_BERT) based on entity and entity context information enhanced BERT. This method uses BERT to obtain the sentence feature representation vector, and then combines two target entities and entity context statement information. In addition, the article also carried out experiments on RoBERTa model and DistiBERT model which are the improved model of BERT. The results on the SemEval-2010 task 8 dataset and the KBP-37 dataset show that the Bert based method performs best, achieving 89.69% and 65.92% of the Macro-F1, respectively.
-
Information Extraction and Text Mining2021, 35(3): 78-87.The current method of abstractive summarization generally adopt machine learning models based on encoder-decoder architecture, with recurrent neural network as the encoder often. To capture the structure information of the text which is believed to play an important role in judging the important content, this paper proposes a text structure information encoder via graph convolutional neural network. This paper designs a normalization and fusion layer, which aims to enable the model to model both the linear and the structure information in the text. In addition, a multi-headed attention decoder is adopted to improve the quality of the generated summary. The experimental results show that the proposed method significantly improves the system performance according to ROUGE evaluation.
-
Information Extraction and Text Mining2021, 35(3): 88-93.Clinical term normalization is an indispensable task in clinical text information extraction. There are often various ways of writing about the same clinical term like diagnosis, operation, medicine, examination, laboratory test, symptom, etc., and term normalization is to find the corresponding standard name for different clinical terms. Based on the candidate answers generated by information retrieval tools, this paper proposes a method of reordering candidates based on BERT (Bidirectional Encoder Representation from Transformers). The experimental results show that the accuracy of single model and fusion model achieves 89.1% and 92.8%, respectively.
-
Information Extraction and Text Mining2021, 35(3): 94-99.The 5th China Conference on Health Information Processing held a shared task including three tracks on Chinese clinical medical information processing. The first track is normalization of Chinese clinical terminology that assigns standard terminologies to surgical entities extracted from Chinese electronic medical records. All surgical entities in the Track1 dataset were collected from real medical data and annotated with standard surgical terminologies of "ICD9-2017 Clinical Edition". A total of 56 teams signed up for the track, and eventually 20 teams submitted 47 system runs. Accuracy is used to measure the performances of all systems, and the highest accuracy of all submitted system runs reached 0.9483.
-
Information Extraction and Text Mining2021, 35(3): 100-106.Medical question appeal classification can be dealt with as a text classification issue. This paper presents a reinforcement learning method for medical question appeal classification. Firstly, the keywords in medical problems are automatically identified by reinforcement learning, and the keywords and non-keywords in medical problems are assigned different values to constitute a vector. Secondly, the vector is set as the weight vector of the attention mechanism, and the weighted sum of the hidden layer generated by the Bi-LSTM model constitute question representation. At last, the question representation is classified by softmax classifier. Experimental results show that the accuracy of this method outperforms Bi-LSTM model by 1.49%.
-
Information Extraction and Text Mining2021, 35(3): 107-114.To avoid the issue of gradient disappearance or gradient explosion associated with the deeper layers and better capture the word semantic information, this paper proposed a fusion network for Chinese news text classification. Firstly, this paper applies the densely connected bi-GRU to learn the deep semantic representation. Secondly, it applies max-pooling layer to reduce the key vector dimension. Thirdly, it adopted the self-attention mechanism to capture more important features. Finally, the learning representations are concatenated as the input of the classifier. The experimental results on NLPCC2014 dataset show that the proposed fusion model is better than the latest model AC-BiLSTM.
-
Sentiment Analysis and Social Computing2021, 35(3): 115-124.A hybrid model based on ELMo (Embeddings from Language Models) and Transformer is proposed for sentimental analysis. Firstly, the ELMo model based on bilateral LSTM model is applied to generate word vectors that combine the contexts features and word features, with different vectors for different meanings of a polysemous word. Then, the ELMo vector is input into a Transformer with the encoder and decoder modified for sentiment classification. The hybrid model of ELMo and Transformer with two different network structures can extract the semantic features of sentences from different aspects. The experimental results show that, compared with state-of-the-arts methods, the proposed model improves the accuracy by 3.52% on NLPCC2014 Task2 datasets, by 0.7%, 2%, 1.98% and 1.36% on 4 sub-datasets of hotel reviews respectively.
-
Sentiment Analysis and Social Computing2021, 35(3): 125-133.It is of great significance to capture the intention of customer's call. Besides the existing manual intention detection, so far, there is no public report on the intention detection of customer's call via machine learning or deep learning models. This paper proposes three intention detection methods based on classical machine learning model, on single/multiple deep learning model, and on the combination of BERT and deep learning model, respectively. The experiments on mobile customer service corpus show that the top F1-value reaches 86.30%.
-
Sentiment Analysis and Social Computing2021, 35(3): 134-142.In review sentiment analysis, the user and product information related to the review are of great help to improve the accuracy. This paper proposes a graph network-based model to capture the relationship between products and user information and reviews via a graph. It learns the impact of product and user information on reviews based on the graph convolutional network model. Experiments on the Yelp2013 dataset show that the model can effectively improve the accuracy of emotion classification for user comments.
