memra-bench

Benchmark artifacts for the tiyuvta serving engine

Reproduction artifacts for the performance claims of the tiyuvta serving engine (OpenAI-compatible serving), tuned for RTX PRO 6000 Blackwell (sm_120a) and RTX 5090, with a compile-gated H100 (sm_90a) lane. If a claim depends on a specific artifact, the artifact is public here.

Refreshed 2026-08-21 against serving engine commit 73d79a3cc8 (release v0.99.0 at that refresh). The engine and its env prefix were both renamed since this card's first revision: every BW24_* incantation in older revisions of this card is dead.

Board update — v0.101.0 headline cell (added 2026-08-22)

Serving engine release v0.101.0 (tag 23d5ae3ffb) lands the DFlash2 drafter and the round-cost engine together; its headline cell, quoted from the release's own measured receipts (the "DFlash2 drafter and the round-cost engine" release entry):

This is an engine bench claim on a named card under stated conditions — not a hosted-service throughput figure.

Drafts (drafts/<model>/) — the standard regime, one file per model

Every spec board row runs one trimmed draft file built by the documented draft regime: FR-Spec ranks derived from the model's own generations (never transferred between models — foreign ranks measured −12 acceptance pts on an identical tokenizer), MTP block extracted byte-verbatim from the published model GGUF, head requantized NVFP4 after trimming (measured zero acceptance cost), block Q4_K_M (measured faster AND higher acceptance than Q8_0). The serving engine attaches the draft file next to its trunk with no other flags (winners are defaults). Proof of attach is a log line, not the absence of an error: [mtp-draft] loading external MTP draft: <path> or [worker] <name>: regime draft attached (<path>).

Rows are the tracked RTX 5090 Laptop spec cells at serving engine commit 73d79a3cc8 (medians of N=5 same-session interleaved reps, measured 2026-08-02, no flags). The long-agentic column is SAMPLED (temp 0.7): rejection-sampling spec decode is distribution-exact; short/medium are greedy.

Each directory carries the draft (draft-owntrim-nvfp4head-q4blk.gguf) and the rank file it was built from (owngen-ranks-32768.txt, one token id per line, rank order — derived 2026-07-17/18, 218-prompt mixed corpus, ~110k own-generated tokens per model).

Use ours for these exact models. For any other model, requant, or finetune, build your own (a finetune's distribution moved, so its draft must too): frspec-owngen derives 32,768 ranks from the model's OWN generations, and make-trimmed-draft.sh extracts, trims and quantizes the head into the draft file. Validate before trusting: frspec-owngen --validate A/Bs baseline-vs-trimmed spec e2e and prints a GOOD/WASH/BAD verdict.

drafts/kf4/ — archived experiment, no released config consumes these

These directories were built for an experimental NVFP4 KV-cache format arm (kv-fp4 lane, 2026-07-20). That arm was never merged: at serving engine commit 73d79a3cc8 the KV-cache format accepts K = q8_0 | fp8 and V = q5_1 | q4_0 | fp8 only (defaults q8_0/q5_1). The files are kept as evidence of regime law 1 — a new numeric config re-derives each model's draft from its own generations under that config. That lane's 2026-07-20 verdicts, for the record: 9B wash, 27B kf4-draft win (+2.5% e2e), 35B old-draft stays.

Gemma ranks (drafts/gemma4-*/)

Gemma drafters are separate assistant GGUFs, not NextN/MTP heads — the serving engine attaches them through its assistant-drafter seam (the seam trap: the qwen-family MTP attach refuses this format; with a drafter attached, the gemma spec route arms at K=5 by default). The FR-Spec trim applies at load from the ranks file; the serve-time adaptive trim (on by default with ranks — 512 spare head slots; a static trim is the alternative) learns coverage escapes live from prompt + verify tokens and persists them to <ranks>.learned. Own-gen rank files per model (regime law 1):

Legacy trims (top level, Qwen3.6-27B)

The pre-regime artifacts the earlier boards used — superseded by drafts/ but kept because published claims referenced them:

Prompts + configs

prompts/ holds the exact board prompts — p1-code-short.txt, p2-code-medium.txt, p3-agentic-long.txt plus the well-formed p3-agentic-long-v2.txt the sampled protocol uses — and the long-context depth documents p4-16k.txt / p5-32k.txt / p6-64k.txt. CONFIGS.md carries the per-cell engine configs. Protocol: interleaved same-session reps (N≥2–3 medians, both orders), power state pinned per window.

Related

• Flagship (Qwen3.8-27B) ranks + drafts: tiyuvta/Qwen3.8-27B-NVFP4-MTP-GGUF — three ranks flavours; the safetensors path trims at load from the ranks file, no separate draft file.
• Avifenesh/Hy3-REAP-Layer103p5-bw24 — the ~100 GB Hy3 expert overlay the serving engine serves on a 24 GB card via VRAM→RAM→NVMe spill.
• No card? A hosted instance runs at inference.tiyuvta.ai.