PangolinTokenizer: a byte-level BPE tokenizer for Traditional Chinese and Taiwan

1. Introduction

Tokenizers in large language models are often treated as a preprocessing tool. Yet the tokenizer determines how many tokens a model needs to represent the same span of text. When a model’s context window, training token budget, and inference cost are all counted in tokens, the tokenizer directly shapes the efficiency of training and deployment.

For English, whitespace roughly supplies word boundaries. Chinese writing does not mark word boundaries with whitespace. Whether a Chinese tokenizer’s merge rules capture common words therefore directly affects token counts. For the Taiwan context, the problem is harder still. Taiwan text often mixes Traditional Chinese, English, numerals, Bopomofo, Taigi romanization, Hakka romanization, URLs, JSON, emoji, and speech-transcript structure all at once. General multilingual tokenizers can usually represent such text losslessly, but not necessarily at low token cost.

PangolinTokenizer is designed as a byte-level BPE tokenizer for the Taiwan context. It does not aim to be best on average across all languages; it aims for a better vocabulary allocation on Taiwan Mandarin, Taiwan named entities, Taiwan institutional terms, mixed writing systems, and speech-transcript formats.

The main contributions of this report are as follows.

1. We present PangolinTokenizer, a byte-level BPE tokenizer trained for the Taiwan context.
2. PangolinTokenizer’s vocabulary is roughly half the size of mainstream multilingual tokenizers, yet it matches or exceeds them in token efficiency on the Taiwan context.
3. PangolinTokenizer natively supports speech-transcript structural markers, without spending vocabulary space on large blocks of timestamp tokens.
4. We present PangolinBench, a tokenizer evaluation suite focused on the Taiwan context. PangolinBench covers 10 test subsets with explicit pass thresholds, used to check a tokenizer’s correctness, compression, and format stability.
5. PangolinBench results show that PangolinTokenizer reaches the lowest overall tokens/character with the smallest vocabulary, and passes every threshold.

2. Why Taiwan needs its own tokenizer

2.1 Chinese has no whitespace segmentation

Chinese writing does not use whitespace as a stable word boundary. If a tokenizer’s merge rules for Traditional Chinese are insufficient, common words are split into overly short fragments. This increases token counts, reduces the amount of usable text a context window can hold, and raises training and inference cost.

For example, if words such as “戶政事務所” (household registration office), “勞保年金” (labor-insurance pension), or “晶圓代工” (wafer foundry) are split into too many tokens, the model needs more tokens to express the same concept. Such fragmentation affects not only compression but also how the model learns semantic units.

2.2 Taiwan has many local institutional terms and named entities

Taiwan text contains many highly localized institutional terms, place names, agency names, and industry terms. For example: National Health Insurance, labor-insurance pension, household registration office, village chief, MRT, science park, wafer foundry, semiconductor, Taipei, Tainan.

These terms are very frequent in the Taiwan context. If the tokenizer has not learned these fragments in its merge rules, the model represents the same content at higher token cost. That makes Taiwan text carry a higher cost in both training and inference.

2.3 Taiwan text frequently mixes writing systems

The Taiwan context is not a single-writing-system problem. Real text often mixes Traditional Chinese, English, numerals, Bopomofo, Taigi Han characters, Taigi romanization (e.g. Tâi-gí, tshit-á), Hakka romanization (e.g. Hak-kâ), emoji, URLs, email, JSON, code, API documentation, and speech-transcript formats — all at once.

Taigi romanization and Hakka romanization are not simply English. They often contain Latin letters with diacritics, hyphen-separated syllable structures, and different accents or spelling systems. If a tokenizer treats this content as ordinary English, it easily produces unstable segmentation.

2.4 Speech models need structured transcripts

Speech recognition (ASR) and speech-model training need more than text. Transcripts usually also contain speaker identity, start time, end time, non-speech events, JSON fields, segment boundaries, and content fields.

Some tokenizers use a dense block of timestamp tokens (for example, a contiguous run from <|ts_000000|> to <|ts_999999|>). Such a design wastes a large amount of vocabulary space and ties the tokenizer to a fixed time resolution and a fixed audio length. PangolinTokenizer instead represents timestamps as ordinary text numbers (for example 0.00, 3.42, 3575.50), reserving no dedicated token block for timestamps. This keeps the tokenizer’s design the same for audio transcripts of any length.

3. Design principles

PangolinTokenizer follows four design principles.

3.1 Byte-level BPE for lossless UTF-8 round-trips

PangolinTokenizer uses the 256 byte values as its base representation, then learns high-frequency fragments through BPE merge rules. This design means the tokenizer does not need to enumerate every Unicode character in advance, and it avoids the <unk> problem that character-level tokenizers face on unknown characters.

A “lossless round-trip” means that for any UTF-8 text, encoding and then decoding must return exactly the original text, with no character lost. Across PangolinBench’s 30 test samples — covering Traditional Chinese, Bopomofo, Taigi romanization, Hakka romanization, emoji, JSON, URLs, and speech transcripts — every encode/decode result is identical to its input, for a lossless round-trip rate of 100%.

3.2 Train merge rules on the Taiwan context

A tokenizer’s vocabulary reflects the distribution of its training corpus. If a tokenizer is learned mainly from general multilingual corpora, it may sacrifice token efficiency on low-resource or specific local contexts. Rust et al. (2021) show that, under fair controlled comparison, a dedicated monolingual tokenizer can improve downstream performance across most languages and tasks.

PangolinTokenizer’s goal is not to be the “best on average across all languages.” Its goal is better token efficiency on the Taiwan context. The training corpus is stratified by language, writing system, domain, and format, covering the following categories.

3.3 Preserve speech-transcript structural markers

PangolinTokenizer includes 10 built-in speech-transcript structural markers. Each marker is encoded as a single ID rather than split into multiple tokens.

In addition, PangolinTokenizer supports OCR, vision, retrieval, and audio-related structural markers (for example <|ocr_start|>, <|image_start|>, <|audio_start|>).

These special markers serve only general control formats and structured I/O. PangolinTokenizer explicitly excludes two kinds of tokens: dense timestamp token blocks (such as <|ts_0|> through <|ts_N|>), and discrete audio-codec token blocks.

3.4 Evaluate with a benchmark; let the numbers speak

Most tokenizer write-ups rely on subjective example sentences. PangolinTokenizer instead uses PangolinBench for systematic evaluation. PangolinBench is a tokenizer benchmark designed for the Taiwan context. It does not evaluate general NLP tasks; it focuses on a few key properties of the tokenizer itself.

First, correctness: whether any text is identical to the original after encode and decode. Second, compression: how many tokens are needed to represent the same text. Third, fragmentation of Taiwan vocabulary: whether common Taiwan words are split into too many tokens. Fourth, writing-system coverage: how well it handles Traditional Chinese, Bopomofo, Latin letters, emoji, and other scripts. Fifth, speech-transcript stability: whether a JSON-format transcript still parses correctly after encode/decode.

These metrics let a tokenizer’s strengths and limitations be quantified concretely.

4. Training method

PangolinTokenizer is trained with UbiTok. UbiTok is an in-house streaming byte-level BPE training toolkit. The overall pipeline starts from the 256 byte values, then learns merge rules from pair frequencies in the corpus.

4.1 Streaming architecture

UbiTok does not need to load the full corpus into memory. Large files are split into chunks aligned to line boundaries. Multiple workers compute fragment frequencies in parallel. The main process then merges the statistics and runs merge-rule learning.

This design lets tokenizer training scale to large corpora while keeping memory usage controllable.

4.2 Deterministic training

The entire training pipeline is deterministic. The same corpus, the same parameters, and the same settings produce an identical tokenizer.

UbiTok ensures determinism through the following design:

• shard assignment uses a blake2b hash;
• merge ordering uses a fixed tie-breaking rule;
• the tie-breaking rule is -frequency, len(merged_bytes), merged_bytes, pair_first_seen_order;
• the fragment-count cap policy produces fixed output under the same input.

This property matters for model training. It lets the tokenizer be reproduced and audited, and it supports long-term version management.

After training, UbiTok produces Hugging Face–compatible tokenizer files. Users can load them directly with AutoTokenizer.from_pretrained, with no need for trust_remote_code=True.

5. Evaluation method: PangolinBench

PangolinBench is a tokenizer benchmark designed specifically for the Taiwan context. It covers 10 test subsets, each corresponding to a different facet of Taiwan text.

5.1 Test subsets

5.2 Evaluation metrics

6. PangolinBench results

This section compares PangolinTokenizer with 8 baseline tokenizers, for 9 tokenizers in total.

6.1 Baseline tokenizers

PangolinTokenizer has the smallest vocabulary of all tokenizers tested. It is about 90% of LLaMA 3.1, 76% of Qwen 3, 57% of GPT-4o, 46% of Qwen 3.6, 44% of Gemma 4, and 36% of TAIDE.

Gemma 3 and Gemma 4 produce identical results on every subset and are therefore shown merged in the tables. Qwen 2.5 and Qwen 3 behave the same way. Qwen 3.6 uses a different and larger tokenizer.

6.2 Token compression results

The table below shows each tokenizer’s token compression on PangolinBench. Unless otherwise noted, values are tokens/character. Lower values mean a more token-efficient tokenizer.

PangolinTokenizer’s overall tokens/character is 0.485, the lowest of all tokenizers tested. TAIDE’s overall tokens/character is 0.486, very close. However, TAIDE’s vocabulary is 318,080 — about 2.77× that of PangolinTokenizer.

This shows PangolinTokenizer’s clear advantage in vocabulary efficiency. It reaches overall compression close to, or slightly better than, much larger tokenizers, with a much smaller vocabulary.

6.3 Core Taiwan Mandarin text

On formal Traditional Chinese, colloquial Taiwan Mandarin, and Taiwan named entities, PangolinTokenizer is not the absolute best on every subset. TAIDE still has the lowest tokens/character on these three. However, PangolinTokenizer’s vocabulary is only 36% of TAIDE’s. Given that, PangolinTokenizer still reaches performance very close to the large tokenizers and clearly beats several mainstream multilingual tokenizers.

The token savings of PangolinTokenizer relative to the baselines are below. Negative values mean PangolinTokenizer uses fewer tokens.

This shows an important trade-off. TAIDE trades a much larger vocabulary for stronger Chinese and Taiwan-vocabulary compression. PangolinTokenizer keeps comparable overall performance with a much smaller vocabulary, and wins on speech-transcript JSON and vocabulary efficiency.

6.4 Speech transcript JSON

PangolinTokenizer’s tokens/character on the speech-transcript JSON subset is 0.400, the lowest of all tokenizers tested. This matches the design goal. PangolinTokenizer natively includes speech-transcript structural markers, so it can represent ASR training formats more efficiently.

6.5 Trade-offs on Taigi, Hakka, Bopomofo, and code-switching

PangolinTokenizer still has room to improve on the Taigi Han-roman mix, Hakka romanization, Bopomofo, and code-switching subsets.

On these subsets, TAIDE and Gemma 3/4 usually have better compression. This reflects the influence of vocabulary size and corpus distribution. A 115K vocabulary cannot allocate enough merge rules to Traditional Chinese, Taigi, Hakka, Bopomofo, English technical terms, JSON, URLs, and multimodal structural markers all at the same time.

Even so, PangolinTokenizer has several noteworthy results. On the Bopomofo subset, PangolinTokenizer’s 1.068 beats GPT-4o, LLaMA 3.1, Qwen 3.6, and Qwen 2.5/3. On the Taigi Han-roman mix subset, PangolinTokenizer’s 0.714 is close to Qwen 2.5/3’s 0.692. The lossless round-trip rate is 100% on every subset. Even when token counts are higher, PangolinTokenizer never loses a character.

7. Taiwan-vocabulary metrics

PangolinBench builds a Taiwan-specific vocabulary set and computes token counts and over-fragmentation rate for each word.

7.1 Per-word token counts

High-frequency Taiwan terms such as “健保”, “捷運”, “台北”, “臺南”, and “半導體” can each be encoded as a single token. This shows that PangolinTokenizer’s merge rules did learn some high-frequency Taiwan-context fragments.

Longer or more compound words are still split into multiple tokens. For example, “晶圓代工” is 4 tokens. This means future versions can still tune merge-rule allocation for industry terms, institutional terms, and long compound nouns.

7.2 Weighted vocabulary token cost

TAIDE is best at Taiwan-vocabulary compression, consistent with its 318K vocabulary. PangolinTokenizer’s weighted cost is 2.48 — the best once TAIDE is set aside. Compared with the similarly sized LLaMA 3.1, PangolinTokenizer’s weighted Taiwan-vocabulary cost is about 29% lower.

This supports PangolinTokenizer’s core design hypothesis: under a limited vocabulary, training merge rules on the Taiwan context can reduce the fragmentation of Taiwan vocabulary.

8. Speech-transcript stability

PangolinBench tests speech-transcript encode/decode stability with the following JSON structure.

[
  {"Start":0.00,"End":3.42,"Speaker":0,"Content":"大家好，歡迎回來。"},
  {"Start":3.50,"End":7.10,"Speaker":1,"Content":"今天我們要討論台灣本土語音模型。"},
  {"Start":10.25,"End":12.00,"Speaker":0,"Content":"注音 ㄅㄆㄇ 與台語 tshit-á 混合測試。"},
  {"Start":12.10,"End":14.00,"Speaker":null,"Content":"[Silence]","Type":"non_speech"},
  {"Start":3575.50,"End":3578.25,"Speaker":2,"Content":"長會議時間戳測試。"}
]

Every tokenizer tested passes lossless round-trip, JSON parsing, and timestamp-precision checks. However, only PangolinTokenizer natively includes the full set of speech-transcript structural markers.

The point here is not that other tokenizers cannot handle JSON. Every byte-level or Unicode-compatible tokenizer can round-trip JSON losslessly. The difference is that PangolinTokenizer designs the speech-transcript structural markers as native single-ID tokens. It can therefore support speech-model training formats directly, without extending the vocabulary after the fact.

9. Vocabulary efficiency analysis

A key observation about PangolinTokenizer is that it reaches the lowest overall tokens/character with the smallest vocabulary.

Vocabulary efficiency is defined as follows:

PangolinTokenizer’s vocabulary efficiency is 1.795, the highest of all tokenizers tested. That is, it buys the most characters/token per 100K of vocabulary space.

This has several practical implications. First, a smaller vocabulary reduces the parameter count of the embedding matrix and the output projection layer. Second, lower tokens/character lets the same context window hold more Taiwan-context text. Third, when the training token budget is fixed, lower token cost lets the model see more raw text content. Fourth, at inference time, shorter token sequences also reduce KV-cache pressure.

Note that vocabulary size by itself does not directly reduce the KV cache. The KV cache is driven mainly by sequence length. PangolinTokenizer’s KV-cache advantage comes from lower token counts, not from the smaller vocabulary itself.

10. Usage

PangolinTokenizer can be loaded directly through Hugging Face Transformers.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "OpenFormosa/PangolinTokenizer",
    trust_remote_code=False,
)

text = "<|system|>台灣健保與注音ㄅㄆㄇ，Tailo: Tâi-uân"
ids = tokenizer.encode(text)
decoded = tokenizer.decode(ids)

assert decoded == text

The model’s maximum sequence length is 131,072 tokens.

11. Known limitations

11.1 Token efficiency on Taigi, Hakka, and Bopomofo has room to improve

On the Taigi Han-roman mix, Hakka romanization, and Bopomofo subsets, PangolinTokenizer underperforms the larger-vocabulary TAIDE, Gemma 3/4, and some baseline tokenizers. This reflects the natural limit of a 115K vocabulary. Improving compression on Taigi, Hakka, Bopomofo, and English technical terms at the same time may require a larger vocabulary or a more careful merge-rule allocation strategy.

11.2 Code-switching and English technical terms can still be strengthened

On the code-switching subset, PangolinTokenizer’s tokens/character is 0.478, slightly above TAIDE, Gemma 3/4, and Qwen 3.6. This shows that a smaller vocabulary still has limited merge-rule coverage for English technical terms, API names, and code snippets.

Future versions can raise the sampling weight for Taiwan technology text, open-source code, API documentation, hardware specifications, and AI-engineering corpora.

11.3 PangolinBench is still a synthetic test set

For now, PangolinBench uses a built-in test set of 30 curated samples. These samples cover the main linguistic phenomena, but they are still not enough to represent the full distribution of Taiwan text.

The results in this report should therefore be read as a targeted, tokenizer-level benchmark, not a full real-world distribution evaluation. Stronger conclusions would require adding large unseen corpora — for example government open data, Taiwan news corpora, education and healthcare text, ASR transcripts, customer-service and meeting data, Taigi and Hakka corpora, and technical documents and code.

PangolinBench already supports mounting private unseen data through an external config file. Future evaluation should use larger-scale, deduplicated, stratified unseen corpora.

11.4 Tokenizer compression is not language-model capability

This report evaluates tokenizer-level compression, correctness, and format stability. These metrics matter, but they do not directly imply that “a language model using PangolinTokenizer must be better.”

Supporting model-level claims would require controlled language-model ablations. For example: fix the model architecture, fix the training data, fix the training token budget or byte budget, and compare bits-per-byte, bits-per-character, downstream Taiwan-context tasks, and training stability on speech-transcript formats.

Until such experiments are done, claims about PangolinTokenizer should stay at the tokenizer level.

12. Conclusion

The core contribution of PangolinTokenizer is not “being able to tokenize Traditional Chinese.” Byte-level tokenizers can already represent arbitrary UTF-8. The point of PangolinTokenizer is this: with a smaller vocabulary, it offers lower token cost, lower over-fragmentation, and more stable speech-transcript structural support on the Taiwan context.

On PangolinBench, we compared PangolinTokenizer with 8 baseline tokenizers, with the following results.

• The vocabulary size is 114,822, the smallest of all tokenizers tested.
• The overall tokens/character is 0.485, the lowest of all tokenizers tested.
• On formal Traditional Chinese, token counts are 25% lower than GPT-4o and LLaMA 3.1, 8% lower than Qwen 2.5/3, and on par with Qwen 3.6.
• On Taiwan named entities, token counts are about 28% lower than GPT-4o, about 21% lower than Qwen 2.5/3, and about 18% lower than Gemma 3/4.
• On colloquial Taiwan Mandarin, tokens/character is 0.708, better than GPT-4o, LLaMA 3.1, Qwen 2.5/3, and Gemma 3/4.
• On speech-transcript JSON, tokens/character is 0.400, the lowest of all tokenizers tested.
• The weighted Taiwan-vocabulary cost is 2.48, the best once the 318K-vocabulary TAIDE is set aside.
• The vocabulary efficiency is 1.795, the highest of all tokenizers tested.
• The lossless round-trip rate is 100%.
• PangolinTokenizer passes every quality-check threshold.
• PangolinTokenizer is the only tokenizer tested with native support for the full set of speech-transcript structural markers.

A tokenizer suited to the Taiwan context should meet three conditions at once. First, it must round-trip arbitrary UTF-8 losslessly. Second, it must have lower token cost on Taiwan Mandarin, Taiwan named entities, institutional terms, and mixed-writing-system text. Third, it must support modern speech, multimodal, and structured training formats.

Through byte-level BPE, Taiwan-context merge rules, general structural special tokens, and the PangolinBench evaluation pipeline, PangolinTokenizer provides a concrete implementation of all three.

13. Open resources

• PangolinTokenizer model: huggingface.co/OpenFormosa/PangolinTokenizer
• UbiTok training toolkit: github.com/OpenFormosa/ubi_tokenizer
• PangolinBench evaluation suite: github.com/OpenFormosa/PangolinBench

References

• Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. Proceedings of ACL.
• Rust, Phillip, Ivan Pfeiffer, Jonas Pfeiffer, Elad Ben-Zaken, Sebastian Ruder, and Iryna Gurevych. 2021. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. Proceedings of ACL-IJCNLP.
• Hugging Face. Tokenizers Documentation. See the byte-level BPE and pre-tokenizer documentation.