open-ogma-small

open-ogma-small

An 8.96M-parameter English sentence-embedding model with a fully permissive license chain — MIT-licensed teachers, ODC-BY corpora, MIT weights. The open-ogma family is the permissive successor to the axiotic/ogma-* family (which is CC-BY-NC-4.0 due to its training-data mix).

• Architecture: 6-layer transformer trunk (d_model=256, d_output=256, mean pooling, task tokens [QRY]/[DOC]/[SYM]) + two projection heads: proj_small 256→384 and proj_large 256→1024.
• Default encode path: proj_small (384d) — the head used for the headline MTEB row. High-fidelity mode: proj_large (1024d) scores +0.011 mean / +0.028 retrieval over the default — opt in when output width is not a constraint.
• Matryoshka: both heads and the trunk were trained with nested-slice losses at [32, 64, 128, 256] — truncate any output to its leading dims and re-normalise for smaller embeddings.

MTEB(eng, v2) — full 41-task benchmark

Reference points measured with the identical harness (subprocess-per-task, mteb 2.12/2.18; harness validated by reproducing minishlab/potion-base-8M at 0.5328 vs its official 53.33): teacher bge-small-en-v1.5 (33M) scores 0.6368 and bge-large-en-v1.5 (335M) 0.6528 — this model retains ~90-92% of the small teacher's mean at 27% of its parameters.

In-training small-mteb (20-task subset) best: 0.5857 at 88B tokens.

Recipe

Distilled for 100B tokens (bs=128 × seq 1024) from an ensemble of two MIT teachers — BAAI/bge-small-en-v1.5 (384d) and BAAI/bge-large-en-v1.5 (1024d) — each supervised through its own projection head with equal loss weight (0.5/0.5), on a 10.6M-document permissive blend of C4 and FineWeb-Edu (contamination-flagged against MTEB test sets, 0.11% hits marked). Loss: matryoshka-weighted distillation (α) + in-batch contrastive (β) on a 1.0→0.7/0.3 schedule; task-token mix {QRY 0.25, DOC 0.25, SYM 0.5}; AdamW lr 5e-4 cosine, wd 0.01.

License chain (verified)

ODC-BY requires attribution for the source corpora, which this card provides. No non-commercial or share-alike terms anywhere in the chain.

Usage

# clone this repo, then:
from ogma_libre import OgmaLibre
model = OgmaLibre.from_repo("path/to/open-ogma-small")

embs = model.encode(["A quick brown fox."])                    # (N, 384) default
embs = model.encode(["A quick brown fox."], head="large1024")  # (N, 1024) best quality
embs = model.encode(["A quick brown fox."], head="base")       # (N, 256) trunk

# Retrieval convention: encode queries with task="qry", documents with task="doc".
q = model.encode(["what is a fox?"], task="qry")
d = model.encode(["The fox is a small canid."], task="doc")

Files: model.safetensors (weights), config.json (architecture + heads), tokenizer/ogma_sp.model (sentencepiece), ogma/ (vendored model code), ogma_libre.py (loader).

Sibling

axiotic/open-ogma-micro — the 2.3M-parameter sibling (same recipe, micro trunk).