SomNLP-Corpus is the largest open Somali text corpus we have released: 1,668,080 clean documents, 528,853,952 words, and approximately 793 million subword tokens (~4.0 GB JSONL). It merges six public Somali-heavy distributions through a reproducible cleaning, deep-cleaning, deduplication, and language-ID pipeline.
Corpus results
Starting from 2.63M raw rows across HPLT, CC100, mC4, OPUS, MADLAD, and MT560, the pipeline merges, cleans, LID-verifies, deep-cleans, and near-dedupes down to 1.67M documents. Final word counts are measured on data/final/final_so.jsonl; token estimates use a ×1.5 rule-of-thumb for Somali BPE/SPM relative to whitespace-separated words.
793M+
Subword tokens
529M
Words
1.67M
Clean documents
4.0 GB
Release size
- Downloaded (raw): 2,633,281 documents
- Merged: 2,329,800 documents
- Cleaned: 2,225,791 documents
- LID verified: 2,035,287 documents
- Deep cleaned: 2,003,228 documents
- Final: 1,668,080 documents · 528,853,952 words
Six-stage pipeline
Each stage removes noise, duplicates, or misidentified language. We drop byte-exact duplicates after merge, apply text normalization and quality filters during cleaning, run language identification with a Somali-specific threshold, deep-clean for boilerplate and segment-level LID, then near-dedup before the final export.
We prefer a smaller, verified corpus over a larger noisy dump. The largest removals happen at merge (303K rows) and near-dedup (335K rows), where cross-source overlap and near-duplicate web documents are filtered out.
Sources
The corpus draws from HPLT, CC100, mC4, OPUS, MADLAD, and MT560, the same upstream families used in multilingual pretraining research, but filtered and deduplicated specifically for Somali. License terms follow each upstream; the release bundle is CC-BY-4.0 where compatible.
How to use it
from datasets import load_dataset
ds = load_dataset("goobolabs/somnlp-corpus", split="train")
print(ds[0])
# Stream without a full download
ds_stream = load_dataset(
"goobolabs/somnlp-corpus", split="train", streaming=True
)
for row in ds_stream.take(10):
print(row["text"])What's next
Public Hugging Face hosting and a full dataset card are in progress alongside model checkpoints. v3 will expand informal and conversational text: SMS, social media, and community forums.
- Hugging Face release with reproducibility manifest
- Dialect and domain metadata on a subset of documents
- Streaming API for large slices without full download
Writing about Somali language technology, open data, and AI from the lab in Mogadishu.