Construction of an Optimal Tibetan Character Set for Large Language Models

Character sets bridge the gap between raw text and model inputs in language models and serve as fundamental components of large language models and other intelligent information processing systems. However, no public Tibetan character set is currently available. This study aims to identify an optimal Tibetan character set for large-scale data modeling and develop a simple and efficient construction method that captures the fine-grained linguistic features of the Tibetan writing system. Based on classical Tibetan grammar theory, we analyze Tibetan characters, stacks, syllables, blocks, and words and establish a formal definition of the Tibetan character set. A Tibetan character set dataset is then compiled according to this definition. Considering the characteristics of tokenization algorithms used in large language models, we propose a novel construction method and compare it with traditional approaches. Experimental results show that the Bod-WL character set reduces vocabulary size by 28.67% compared with conventional character-level and syllable-level processing methods. Relative to the baseline, the Bod-BPE, Bod-WordPiece, and Bod-Unigram tokenization algorithms improve language modeling performance by 2.01, 2.87, and 2.96 percentage points, respectively. These results demonstrate the effectiveness of the proposed method for Tibetan language modeling. The constructed Tibetan character set and related resources are publicly available at https://github.com/AI-Bod/.