Documentation
Quick-start guides
Load datasets, transcribe speech, and run the Somali tokenizer.
All datasets and models are hosted on Hugging Face under the somast organisation. Install with: pip install datasets transformers.
Load a dataset
from datasets import load_dataset
# Gated dataset: accept the terms on the Hub, then log in with a token.
# Configs: "open" and "non-commercial". Pin the release with revision.
ds = load_dataset("somast/somali-nlp-corpus", "open", split="train", revision="v1.0.0")
print(ds[0]["text"], ds[0]["license"])Transcribe Somali speech (ASR)
import whisper
model = whisper.load_model("somast/whisper-som-small")
result = model.transcribe("audio.wav", language="so")
print(result["text"])
# 'Beeraha Soomaaliya waxay u baahan yihiin...'Tokenize Somali text
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("somast/somali-bpe-v2")
ids = tok.encode("Caafimaadka carruurtu waa mudnaanteenna koowaad.")
print(tok.convert_ids_to_tokens(ids))
# ['Caafimaadka', 'Ġcarruurtu', 'Ġwaa', 'Ġmudnaan', 'teenna', 'Ġkoowaad', '.']
print(tok.decode(ids)) # the input, exactlyNewsletter
Stay in the loop
Dataset releases, new bootcamp cohorts, research papers, and lab updates, delivered straight to your inbox. No spam, unsubscribe anytime.