Releasing Somali NLP Corpus: 1.13 billion tokens

Somali NLP Corpus 1.0.0 is out. It holds 7,913,008 clean documents, 826,973,689 words and 1,129,084,663 subword tokens from 14 public sources, published as a gated dataset on Hugging Face. Every record keeps the license of the dataset it came from, and the files are split into two tiers so you can tell at a glance what you may use commercially.

What is in it

Token counts come from encoding every document with our Somali ByteLevel BPE v2 tokenizer, not from a sample, and work out to 1.3653 tokens per word. The release is 2.5 GB of zstd Parquet on the Hub. The same text as JSONL is 9.07 GB.

Each record carries its text, source, sub-source, license, the language the detector assigned, the detector's score, a content hash and any quality flags. Earlier builds had records whose license was just "Other". That was not good enough for people who want to build on the corpus, and this release fixes it.

From 18.1 million rows to 7.9 million

We downloaded 18,149,257 rows and kept 44% of them. The pipeline is the same five stages as before: merge with exact deduplication, clean, language identification, deep clean, and near-duplicate removal.

Deep clean is where most of the judgment calls live. It removed 53,189 exact duplicates that only appeared after cleaning, 47,151 boilerplate records, 11,081 that were not Somali on a second look, and about 2,900 that were too long, too short after cleaning, or mostly numbers or symbols. The whole pipeline ran on one laptop with an 8-thread Intel i7 and 31 GB of memory, with no GPU. Language identification was the slow part at 1 hour 13 minutes.

The 14 sources

Two sources dominate by record count and a different three dominate by text. NLLB and Glot500 are made of short sentences, so together they are 70% of the documents but only 15% of the tokens. mC4, HPLT and FineWeb-2 are web pages and carry 68% of the tokens.

Full URLs, access methods and licensing notes for every source are in docs/SOURCES.md, and the release ships an ATTRIBUTION.md that credits each upstream dataset.

Licenses and access

There is no single license for this corpus, and we did not invent one. Each record has the license of the dataset it came from. Glot500 bundles many datasets, so we kept the name of each one and took the license from it. Where we could not establish a license, the rows are not in the release.

The tiers are a grouping, not a license. Records in open still carry conditions such as attribution and share-alike, and you are responsible for following the license named on each record. The export stops if it meets a license it cannot classify, so nothing slips into the open tier by default.

Some content was left out for licensing reasons: Somali TinyStories (no license on its card), the QuranEnc and Tanzil translations (redistribution terms not confirmed), and 303,550 Glot500 rows from sub-sources without a redistributable license (MTData, Earthlings, Bible_crawled, Bible_pbc and rows with an unknown sub-source).

What changed since the 17-source build

The September build had 17 sources and 7,981,982 documents. The new one is smaller on purpose, and cleaner.

• Three sources removed (TinyStories, QuranEnc, Tanzil) and 303,550 unlicensable Glot500 rows dropped.
• HTML tags no longer glue neighbouring words together, and malformed tags are removed.
• The length floor is applied again after deep clean, so no document is below 25 words and no sentence below 5.
• Quality flags are recomputed from the final text, which took the records marked for review from 8,678 in the September audit to 40.
• Exact duplicates created by cleaning are dropped. The final file has no duplicate ids and no duplicate content hashes.
• Phone numbers are masked next to URLs and emails: international numbers, Somali local mobile numbers, and local numbers that follow a phone keyword.

Known limits

A corpus this size has problems that numbers do not hide. These are the ones we know about.

• Language tags are detector guesses. 107,326 records carry a tag other than Somali, almost all of them NLLB sentences. They are kept as metadata and not used to reject sentences.
• 28 records still contain a 12 to 14 digit number after a phone keyword. Names and other personal details in news text are not masked.
• Quality was measured automatically. Nobody has reviewed dialect or register by hand, and the text is mostly web text, so it leans formal and urban.
• Share-alike terms (CC-100, Wikipedia, FLORES-200 inside Glot500) have not had a legal review.

How to use it

from datasets import load_dataset

# Gated: accept the terms on the Hub, then log in with a token.
ds = load_dataset(
    "somast/somali-nlp-corpus", "open",
    split="train", revision="v1.0.0",
)
print(ds[0]["text"], ds[0]["license"])

Use the non-commercial configuration for research that will not ship in a product. Pin revision to a tag so your results stay reproducible. Releases follow semantic versioning: a patch for takedowns and corrections, a minor for new sources, a major for schema or license changes. To cite the corpus, use the CITATION.cff in the repository or the BibTeX on the dataset page.

What is next

• A native-speaker review of a sample from each source type, to put real numbers on dialect and register.
• A legal review of the share-alike sources.
• Targeted collection of Somali news, government and university sites, with provenance on every page.
• A retrained tokenizer on the full 14 sources. The tokenizer results are in a companion post.

If you find a record that should not be in the corpus, write to [email protected] with the record id. If you find a bug in the pipeline, open an issue on GitHub.