汉英MT源语分析首先遇到的问题是词的识别。汉语中的“词”没有明确的定义,语素和词、词和词组、词组和句子,相互之间也没有清楚的界限。按照先分词、再句法分析的办法,会在分词时遇到构词问题和句法问题相互交错的困难。作者认为,可以把字作为源语句法分析的起始点,使词和词组的识别与句法分析同时进行。本文叙述了这种观点及其实现过程,并且以处理离合词为例,说明了识别的基本方法。

The first problem we have met in source language analysis in a Chinese-English MT system is Chinese sentence tokenization , as in written Chinese there is no explicit word delimiter. Finding token boundaries for a character string will be often interlaced with syntactic parsing , or even with semantic relations. This paper presents an approach of combination of sentence tokenization and syntactic-semantic analysis. Instead of getting tokenized word string before sentence parsing , the tokenizing component is built into the parser , i. e. syntactic and semantic information could be used for recognizing words when necessary during parsing which is supported by a dictionary with descriptions for individual usage and a set of common rules.