TL;DR. Instead of one English-dominant tokenizer for everyone, train a small dedicated BPE tokenizer per language and merge them into one shared ID space. No architecture changes, English is preserved, and 21 languages across six scripts improve, most of all morphologically rich and non-Latin-script ones.
Abstract
Large language models rely on a single tokenizer that is chosen at training time and fixed thereafter. Most tokenizers are trained on English-dominant corpora, making them under-serve morphologically rich and non-Latin-script languages, where they can produce fragmented representations, longer sequence lengths, and reduced information density during training.
We propose Mixture of Tokenizers (MoTo), a modular tokenization framework that trains dedicated per-language BPE tokenizers and composes them into a unified superset vocabulary, where tokens shared across languages are deduplicated into a common ID space. Each tokenizer maintains its own normalization policy, pre-tokenization rules, and vocabulary budget. Despite allocating significantly fewer entries to English, MoTo retains English performance while delivering consistent gains across 21 typologically diverse languages spanning six scripts, with improvements most pronounced on morphologically complex and non-Latin-script languages. Our results suggest that modular subword tokenization is a practical and extensible alternative to monolithic tokenization for multilingual parity.
Why one tokenizer isn’t enough
Pre-tokenization rules and merges tuned for English can block useful units in other languages. Here, the elision in French l’école is split at the apostrophe by Llama and English tokenizers, but a French tokenizer compresses it to a single token.
Method
Language-specific tokenizers are trained on their own corpora, then united into a superset vocabulary where shared tokens receive a single global index. At train and inference time, a language ID routes each input to its tokenizer; the transformer sees superset indices and the output is decoded with the same tokenizer.
The model. The architecture is unchanged. The input is tokenized by the tokenizer for its language ID, embedded in the superset vocabulary, passed through a standard Transformer (LLaMA 3.2), unembedded over the same superset, and decoded with the same tokenizer.
Contributions
- Modular superset tokenization. Small per-language BPE tokenizers across 21 languages, composed with no model architecture changes.
- Controlled benchmark. A synthetic multilingual addition task showing a jointly trained tokenizer cannot serve all languages equitably.
- MoTo models. 1B non-embedding-parameter decoder-only LMs pre-trained on FineWeb2: better compression and parity, consistent gains, English preserved.
- Ablations and analysis. Vocabulary size, routing strategy (oracle, regularized, random), and per-language vs. per-script granularity, plus a 14-tokenizer superset.
Results
Multilingual gains over the Llama tokenizer baseline
The default view is mean accuracy per language over all benchmarks (as in the paper figure); pick a single benchmark to drill down, and switch between the difference to the Llama-tokenizer model and absolute accuracy. Hover or tap a bar for exact numbers; toggle models with the chips.
English is retained
Controlled multilingual addition
A synthetic task with the same problem in 21 languages, so differences come only from the tokenizer. Here the baseline is the ByT5 model; bars show per-language accuracy relative to it.
Ablations: vocabulary size and routing
BibTeX
@inproceedings{altintas2026moto, title = {{MoTo}: Mixture of Tokenizers Towards Fair Multilingual Language Modeling}, author = {Alt{\i}nta\c{s}, G{\"u}l Sena and Raffel, Colin}, booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems}, year = {2026}}