somast/somali-nlp-corpus
The largest open Somali text corpus - 1.1B+ tokens from 14 upstream sources (HPLT · CC-100 · mC4 · OPUS · MADLAD-400 · MT560 · Wikipedia · XL-Sum · NLLB · Glot500 · Somali Web Corpus · FinePDFs · FineWeb-2 · Somali Alpaca), cleaned through a five-stage pipeline (merge and exact dedup, clean, language ID, deep clean, near-duplicate removal). Released as a gated Hugging Face dataset in two license tiers, open and non-commercial.
View on Hugging FaceModality
Text
Size
1.1B+ tokens · 2.5 GB
License
Per-record (open / non-commercial tiers)
Language
Somali (af)
Format
Parquet (zstd)
Version
v1.0.0
Tasks
LM, NER, POS, Classification
Domains
HPLT · CC-100 · mC4 · OPUS · MADLAD-400 · MT560 · Wikipedia · XL-Sum · NLLB · Glot500 · Somali Web Corpus · FinePDFs · FineWeb-2 · Somali Alpaca
| Split | Rows | Size |
|---|---|---|
| open | 6,955,948 | 2.3 GB |
| non-commercial | 957,060 | 187 MB |
from datasets import load_dataset
# Gated: accept the terms on the Hub, then run `huggingface-cli login`.
# Configs: "open" and "non-commercial"; every record carries its own license.
ds = load_dataset("somast/somali-nlp-corpus", "open", split="train", revision="v1.0.0")
print(ds[0])@misc{somali_nlp_corpus,
title = {Somali NLP Corpus: a cleaned Somali text corpus},
author = {SOMAST},
year = {2026},
note = {Each record carries its upstream license; there is no single corpus-wide license.},
url = {https://huggingface.co/datasets/somast/somali-nlp-corpus}
}