kenga-embed-prophet-instruct

kenga-embed-prophet-instruct

42M bidirectional UTF-8 byte encoder (seq 160). Instruct prefix matches Giga-Embeddings: query only for retrieval, both sides for STS.

This is not Giga-480M and not a chat model. It does not write replies.

How to encode (same protocol as Giga instruct)

import torch
from modeling_kenga_embed import KengaEmbed

m = KengaEmbed.from_pretrained("GermannM/kenga-embed-prophet-instruct")

# retrieval
q = m.encode_queries(["Where is the capital of Russia?"])
d = m.encode_documents(["Moscow is the capital of Russia."])
print(float((q * d).sum(-1)))

# STS / paraphrase
a = m.encode_sts(["Кот сидит на коврике."])
b = m.encode_sts(["Кошка лежит на ковре."])

Prefixes baked into the loader:

• retrieval query: Instruct: Given a query, retrieve relevant passages\nQuery:
• documents: raw text
• STS: Instruct: Retrieve semantically similar text.\nQuery: on both sentences

Do not skip the prefix. Scoring z_embed.pt with this template is wrong.

Numbers we measured (2026-09-03)

Handmade holdout, 12 EN + 12 RU triples in embed_holdout.json. Not ruMTEB.

Prophet misses on that set: Anna Karenina (Leo Strauss trap), H2O vs H2O2.

RuSTSBenchmarkSTS test (1264 pairs, cosine Spearman), CPU, this folder:

0.4416 (was 0.3583 before instruct+STS train). Still far from Giga.

Giga on the same split: 0.8033. A 42M byte encoder is not expected to win STS. Full MTEB(rus, v1) was not run here (pytrec_eval missing in the train env). Please run the official suite yourselves.

Architecture

D=768, L=8, heads=12, dff=3072, V=256 bytes, rank 192/512. Mean-pool + linear. ~42M params. Checkpoint acc_en=0.833 acc_ru=1.0, step 10400.

License

MIT. Train code lives in the kenga-lang / z-system tree, not in this repo.