mistral-nemo-instruct-2407-kv-sidecars

fraQtl/mistral-nemo-instruct-2407-kv-sidecars

mistral-nemo-instruct-2407-kv-sidecars is a machine learning model from fraQtl. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for gguf. The card lists the license as apache-2.0.

32K per user on one A100-80GB: 32 users before OOM (32/32 needles, 233.0 tok/s aggregate) vs 24 for q80 KV (128.4 tok/s). Single-sequence decode 65.7 vs 56.4 tok/s at 32K, 73.7 vs 71.5 at 8K.

Downloads · 30 days

Access

Public

Updated Oct 25, 2026

Repo size

21 MB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.bin21 MB · 99%

At a glance

Library: gguf
License: apache-2.0
Access: Public
Created: Aug 25, 2026
Updated: Oct 25, 2026
SHA: 23f7893b

Base models

mistralai/Mistral-Nemo-Instruct-2407

Model card

From the Hugging Face model README

fraQtl Membrane Sidecar: Mistral-Nemo-Instruct-2407

32K per user on one A100-80GB: 32 users before OOM (32/32 needles, 233.0 tok/s aggregate) vs 24 for q8_0 KV (128.4 tok/s). Single-sequence decode 65.7 vs 56.4 tok/s at 32K, 73.7 vs 71.5 at 8K.

fraQtl calibration sidecars for Mistral-Nemo-Instruct-2407 (12B, 40 layers), built by the fraQtl sidecar factory with zero kernel changes on the same membrane runtime.

Receipts (llama.cpp membrane runtime, 1×A100-80GB, Q4_K_M weights)

Decode tok/s, single sequence:

Context	membrane-exclusive	q8_0
8K	73.7	71.5
32K	65.7	56.4

Concurrent capacity ("users" = N parallel sequences via -np N, each with an independent KV stream and a full 32,832-token context allocation; exact command line in every receipt JSON):

32K tokens/user	membrane-exclusive	q8_0
Max users before OOM	32 (32/32 needles, 233.0 tok/s)	24 (hard OOM at 28)
Aggregate tok/s at max	233.0	128.4

Scope

Receipts stop at 32K. At 128K, needle retrieval fails on this model in every configuration tested, including llama.cpp q8_0 KV and stock upstream llama.cpp with uncompressed fp16 KV (both 0/1): the advertised 128K window exceeds the model's effective retrieval range. The membrane matched fp16 retrieval behavior at every context.

Single-needle misses at high load: q8_0 dropped 1/16 at 16 users; membrane dropped 1/28 at 28 users and passed 32/32 at 32 users. Per-cell counts are in the receipts.

How to run

Same prebuilt runtime and flags as the Qwen3-4B sidecar card, with this model's sidecar files (this repo). Runtime binaries (Linux x86_64, CUDA, SM80+SM90, manifest-sha'd): fraQtl/fraqtl-membrane-llamacpp-runtime