kenga-embed-prophet3

kenga-embed-prophet3

44M-parameter Russian/English sentence encoder from the Kenga project. Bidirectional Z-factored transformer, SentencePiece-16k tokenizer, mean pooling, 768-d L2-normalised output, 512-token context. Prefix protocol is the FRIDA / BERTA one, so it drops into any pipeline that already uses them.

kenga-embed-prophet2 starts from kenga-embed-z2 and adds a supervised stage: InfoNCE with mined hard negatives on 80k retrieval pairs (RuBQ / MIRACL-style QA, wiki title-passage), CoSENT on 20k STS pairs, plus an anchor loss that keeps the embedding close to the teacher so the distilled knowledge is not overwritten. This is the Kenga Prophet stage: learn from verified pairs, penalise confident mistakes (hard negatives).

Official MTEB(rus, v1.1) numbers

Run with mteb==2.20.5, all splits and subsets as defined by the benchmark, no task was skipped or re-weighted. Reference columns are the models' own leaderboard submissions (embeddings-benchmark/results). 23/23 tasks done.

Leaderboard-style mean (average of task-type means): 51.2.

What this is and is not: a 44M model, roughly 10x smaller than Giga-Embeddings-instruct-480M, meant to be compared with the 30-40M Russian encoders (USER2-small, rubert-tiny-turbo). On the 23 finished tasks (plain mean) it scores 53.4 vs 74.2 for Giga-Embeddings-instruct-480M (+20.7 gap) and 58.5 for USER2-small-34M (-5.0). It does not beat Giga. The numbers above are the whole story; the raw result files are in mteb_results/ of the training tree.

Usage

import sys; sys.path.insert(0, "<this folder>")     # or trust_remote_code-style import after download
from modeling_kenga_embed_v2 import KengaEmbedV2HF

m = KengaEmbedV2HF.from_pretrained("<this folder>", device="cuda")   # cpu works too

q = m.encode(["??? ??????? ????? ? ????"], prefix="search_query")
d = m.encode(["????? ???? ???????? ...", "?????? ?????"], prefix="search_document")
print(q @ d.T)                       # cosine, embeddings are L2-normalised

a = m.encode(["??? ???? ?? ??????."], prefix="paraphrase")
b = m.encode(["?? ?????? ???? ?????."], prefix="paraphrase")

Prefixes ("<prefix>: <text>" is prepended for you):

Download with huggingface_hub.snapshot_download("GermannM/kenga-embed-prophet3"); the folder contains pytorch_model.bin, config.json, kenga_spm.model and the self-contained modeling_kenga_embed_v2.py (torch + sentencepiece only).

Architecture

d=768, layers=8, heads=12, dff=3072, factorised token embedding (16385 x 128 -> 768), Z-factored attention/FF projections with rank 192/512, learned positions up to 512. 44.2M parameters, fp32 checkpoint 177 MB. Checkpoint step 1000.

Training code

PyTorch trainers live in the z-system lab tree (embed_v2/: build_segments.py, teacher.py, distill.py, build_ft_data.py, mine_hard.py, finetune_prophet.py, run_mteb.py), not in the public kenga-lang repo; the recipe and the Prophet contract are documented in docs/PROPHETS.md. Trained on one GTX 1660 (6 GB).

License

MIT.