Somali tokenizer v2: 1.34 tokens per word on the 1.0.0 corpus

Tokenizers decide how much of a model's context, compute and money a Somali sentence costs. A general multilingual tokenizer breaks Somali words into many pieces. Ours keeps most words in one or two. This post reports how well it does on the final 1.0.0 corpus, where it still falls short, and where to download it.

Results

Against XLM-RoBERTa, the model most people reach for on Somali, ours uses 26% fewer tokens for the same text. Against BERT-base it uses about half. Bytes per token tells the same story without depending on how you define a word: 4.84 for v2, 3.69 for XLM-RoBERTa, 2.53 for BERT.

How we measured

• The test set is 63,542 documents, 0.8% of the corpus. A document is held out when a hash of its id falls in a fixed bucket, so the split is reproducible from the ids alone.
• The headline number is tokens per word for each document, averaged over documents. A word is a whitespace-separated string. The corpus-weighted ratio, which favours long documents, is 1.3702 for v2.
• Every document is encoded, decoded, and compared with the original. This checks that the tokenizer is lossless.
• BERT and XLM-RoBERTa use different normalizers and pre-tokenizers, so the comparison measures tokenizer efficiency. It does not say anything about downstream accuracy.
• If anything v1's numbers flatter it, because v1 was trained on an earlier version of some of these documents.

The tokenizer has not seen these documents. It was trained before the corpus was finalised, which also means five of the 14 sources in the test set were not in its training data. We come back to that below.

What it means in practice

Here are four sentences, tokenized by each model. The token counts are real output, not estimates.

Scale that to the corpus. Encoded with v2, the whole 1.0.0 corpus is 1,129,084,663 tokens. At the ratios we measured on the test set, XLM-RoBERTa would need roughly 1.5 billion tokens for the same text and BERT roughly 2.2 billion. Fewer tokens means shorter sequences, so more text fits in the same context window, each training step covers more text, and every request to a served model costs less. These two corpus totals are estimates from the test-set ratio. The v2 total is measured on every document.

You can try the tokenizer on the models page, where the outputs are the real thing.

Why there is a v2

The honest summary of v1 against v2 is that v2 is a bit smaller in tokens and much better in correctness. The compression gain is 2.8%, from 1.3830 to 1.3440. The reason we replaced v1 is that its decoder could not reconstruct its own input. On the 1.0.0 test set it fails on every one of the 63,542 documents, because it was configured to strip an end-of-word marker that its trainer never wrote, and it produces an unknown token in 8 documents. That is invisible if you only classify or retrieve, and disqualifying if you generate text.

v2 is byte-level. It starts from all 256 byte values, so any input can be encoded and an unknown token is unreachable, and its decoder is the exact inverse of its encoder. You can read the full story of the bug in the v2 announcement.

Source by source

Tokens per word varies by what kind of text it is. The table is sorted from hardest to easiest for v2. A star marks the five sources that were added after v2's vocabulary was trained.

• v2 beats v1 on 12 of 14 sources. It loses on FinePDFs (1.4905 against 1.4554) and ties on Wikipedia.
• v2 beats XLM-RoBERTa on every source, by 0.35 to 0.58 tokens per word.
• The five sources v2 never saw behave well. Three of them sit below the overall mean of 1.344. Somali Alpaca is 4% above it. FinePDFs is 11% above it.
• The hardest text is encyclopedic and PDF text, full of names, numbers and borrowed words. The easiest is news, general web text and short parallel sentences.

Why 48,000 tokens

Before shipping v2 we trained four vocabulary sizes on the same data and scored them on the same test set. This sweep predates the 1.0.0 corpus, so its test set was a different, 49,424-document split and the numbers cannot be compared with the table above. Only the shape matters.

Each step buys less. Going from 48,000 to 65,536 saves another 2.5% of tokens for 36% more embedding parameters, so we stopped at 48,000. If your budget cares more about context length than model size, the 65,536 version is the better trade.

Limits

• The vocabulary was trained on an earlier build with eleven sources and has not been retrained for the corpus we published. Glot500, Somali Web Corpus, FinePDFs, FineWeb-2 and Somali Alpaca came after it. FinePDFs, with its 11% penalty, is the clearest sign that a retrain would help.
• That earlier training text included the QuranEnc and Tanzil translations, which we removed from the corpus release because their redistribution terms are not confirmed. The tokenizer holds vocabulary, not text, and we publish it under Apache-2.0, but a retrain on the 14 released sources would remove the question for good.
• Most of the training text is web text, so the merges reflect web register. Spoken, administrative and dialectal Somali are under-represented.
• Tokens per word measures efficiency. It does not measure whether a model trained with this tokenizer is more accurate, and we have not yet run that comparison.
• The corpus is Latin-script Somali. Arabic-script text will encode without loss, but inefficiently.

Use it

The tokenizer is on the Hub as somast/somali-bpe-v2, under Apache-2.0. The benchmark, the training scripts and the research note are in the tokenizer directory of the corpus repository.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("somast/somali-bpe-v2")

text = "Caafimaadka carruurtu waa mudnaanteenna koowaad."
ids = tok.encode(text)

print(len(text.split()), len(ids))   # 5 words, 7 tokens
print(tok.decode(ids) == text)       # True

Next we will retrain on the 14 released sources, re-run this benchmark, and compare downstream accuracy with a model trained from each tokenizer. The corpus is described in the 1.0.0 announcement.