Why Somali needs its own tokenizer

When you run a standard multilingual tokenizer on Somali text, something goes wrong. Words a Somali speaker reads as single units get shattered into fragments, inflating sequence length and wasting context and compute. We measured the damage across the full SomNLP-Corpus and trained a native BPE tokenizer that cuts Somali's token-to-word ratio from 2.69 (BERT-base) down to 1.53.

The problem

BPE tokenizers learn their subword vocabulary from statistical distributions in a training corpus. Because Somali is massively underrepresented in multilingual data, off-the-shelf tokenizers have almost no Somali-specific merges, so Somali words get decomposed with patterns learned from European languages, at character-level granularity that ignores where meaning sits.

The clean way to quantify this is the token-to-word ratio: how many subword tokens a tokenizer emits per whitespace-separated word. Benchmarked over all 1.67M documents of the corpus, BERT-base-uncased averages 2.69 tokens per word and XLM-RoBERTa-base 1.94, while our native Somali BPE averages 1.53. On median the gap is even wider: 1.33 native versus 2.63 for BERT-base, meaning a typical Somali document compresses better than the mean suggests.

Why Somali fragments

Two properties of Somali drive the over-segmentation. First, it is agglutinative, it builds words by stacking morphemes onto roots. Caafimaad (health) appears as caafimaadka (the health), caafimaadkiisa (his health), or caafimaadkooda (their health). Second, its Latin-script orthography and loanword conventions differ enough from English that foreign merges rarely line up with real word boundaries.

Our approach

Rather than bolt heuristics onto a foreign vocabulary, we trained a BPE tokenizer from scratch on Somali-only text, the final split of SomNLP-Corpus v0.2 (data/final/final_so.jsonl): 1,668,080 documents, 528,853,952 words, ~4.0 GB. The pipeline is three scripts (corpus preparation, BPE training with the Hugging Face tokenizers library, and benchmarking against baselines) with this configuration:

• 32,000-token target vocabulary, trained on Somali-only data
• Minimum merge frequency of 2, a merge must occur at least twice to enter the vocabulary
• Whitespace pre-tokenization with NFC Unicode normalization
• Five standard control tokens: [UNK], [CLS], [SEP], [PAD], [MASK]
• Evaluated with stratified sampling (seed 42) across the entire corpus, not a small held-out slice

Results

At corpus scale the compounding is dramatic. Encoding the full release, the native tokenizer emits roughly 810M subword tokens, versus about 1.03B under XLM-RoBERTa and 1.42B under BERT-base. That's ~610M fewer tokens than BERT for the same text, fewer positions to attend over, longer effective context windows, and cheaper training and inference on every downstream Somali model.

The tokenizer

from tokenizers import Tokenizer
from transformers import AutoTokenizer

# Native Somali BPE: 32K vocab, trained on SomNLP-Corpus v0.2
som = Tokenizer.from_file("somali-bpe-tokenizer.json")
xlmr = AutoTokenizer.from_pretrained("xlm-roberta-base")

text = "Caafimaadka carruurtu waa mudnaanteenna koowaad."  # 5 words

n_som = len(som.encode(text).tokens)
n_xlmr = len(xlmr.tokenize(text))

print(n_som / 5, n_xlmr / 5)
# native BPE stays near ~1.5 tokens/word; XLM-R runs higher

# The trained tokenizer, benchmarks, and training scripts live in
# github.com/goobolabs/SomNLP-Corpus/tree/main/tokenizer