Downloads · 30 days
23
33% of all-time downloads
fraQtl/qwen3-4b-instruct-2507-kv-sidecars
qwen3-4b-instruct-2507-kv-sidecars is a machine learning model from fraQtl. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Nine concurrent ≈128K-context users on one A100 at 134.1 tok/s aggregate decode, with nine distinct passkeys planted at placements spanning 0–100% of the context and 9/9 recovered exactly. fp16 holds 2 users without p…
Downloads · 30 days
23
33% of all-time downloads
All-time downloads
70
Public
Repo size
19.3 MB
Likes
1
Public
Click a slice to open those files.
.bin18.9 MB · 97%
From the Hugging Face model README
Nine concurrent ≈128K-context users on one A100 at 134.1 tok/s aggregate decode, with nine distinct passkeys planted at placements spanning 0–100% of the context and 9/9 recovered exactly. fp16 holds 2 users without preemption (66.6 tok/s); FP8 holds 5 (117.6 tok/s peak, at 4 users): 4.5× fp16's users and 1.8× FP8's (9 vs 5), at 2× fp16's total decode throughput and 1.14× FP8's peak. The receipt for every number on this page is in this repository.
The sidecars are calibrated per-layer, per-KV-head eigenbasis artifacts consumed by the fraqtl runtime (a prebuilt binary vLLM attention backend). Base model weights are not modified or redistributed.


Sidecars and receipts: Apache 2.0. Base model weights download from Qwen/Qwen3-4B-Instruct-2507 under its own license. The fraqtl runtime wheel is proprietary, free to install and run for verification and evaluation (license).
Global.
Teams serving LLMs with vLLM who need more concurrent long-context users per GPU, longer contexts within a fixed memory budget, or lower KV-cache cost per token: chatbots, RAG systems, and agentic workloads with large contexts.
Hugging Face 07/2026 via this repository. Companion model: Mistral-7B-Instruct-v0.3 kit.
receipts/repro_2026_07_06/Artifact type: Calibrated KV-cache eigenbasis sidecars (K + V) consumed by a compressed-page attention runtime for vLLM.
Compression architecture: fraQtl inserts a compression membrane between the model and vLLM's paged KV cache. On write, K and V are stored in a compressed page format: a calibrated protected subspace kept at high precision (rank 16 of 128 for K, rank 32 of 128 for V, per layer per KV head, the sidecars in this repository) plus low-bit tails on the remaining dimensions; V is compressed more aggressively than K, whose error softmax amplifies. Logical rank stays 128; nothing is truncated. On read, the attention kernel consumes compressed pages directly at tensor-core speed, with no decompress-then-attend step, so capacity gains do not cost decode bandwidth. The exact tail format is part of the unpublished method.
How the protected subspaces are calibrated is not published. The sidecars are the calibrated artifacts, sufficient to run and verify every number on this page.
Supported runtime engine: vLLM 0.20.2 (torch 2.11.0, the receipt-validated stack)
Supported hardware microarchitecture: NVIDIA Ampere (SM80 / A100). Hopper/Blackwell not yet supported.
Operating system: Linux
Integration uses vLLM's standard out-of-tree plugin mechanism (vllm.general_plugins entry point): pip install the wheel and the backend registers in every engine worker process.
Reproduce every batch-1 number on this page with one command (≈$15 of rented A100 time):
pip install modal && modal setup
modal secret create huggingface HF_TOKEN=hf_...
modal run fraqtl_repro_receipts.py --model qwen3-4b-instruct-2507 --contexts 8k,32k,128k
Prints the three-arm table with per-arm needle-in-a-haystack results. Docker fallback: --local. Script: fraqtl_repro_receipts.py.

| Arm | Users @≈128K | Aggregate decode | Per user |
|---|---|---|---|
| fp16 | 2 (holds 2 without preemption) | 66.6 tok/s | ~33 |
| FP8 | 5 (holds 5 without preemption) | 117.6 tok/s (its peak, at 4 users) | ~29 at peak |
| fraQtl D2 | 9 | 134.1 tok/s | 14.9 |
Same GPU (A100-80GB), same model, vLLM custom backend, CUDA graphs on, prefix caching off. KV pool: 1,255,376 tokens. Recipe as executed: K 16 protected dims, V 32 protected dims, low-bit tails, logical rank 128 (KV pool fingerprint: 1,255,376 tokens).
The retrieval check ran 131,040-token prompts per user (+32 generated); the timing pass ran 125,795-token prompts per user with 1,024 decode tokens each (9,216 total); both are ≈128K workloads. Baseline arms were measured at 130,048-token prompts (a 0.76% difference). Needle grid: nine distinct passkeys, placements spanning 0–100% of context (depths 0.0 / 0.1 / 0.25 / 0.25 / 0.4 / 0.5 / 0.75 / 0.9 / 1.0): 9/9 found. Zero kernel fallbacks, zero CUDA errors, byte-exact roundtrip witness on the storage format.
Per-user decode at 9 users is 14.9 tok/s; fp16 gives ~33 per user at 2 users and ~9.3 at 7 users under preemption. All three arms passed 100% of retrieval checks in the baseline depth grids (189/189).
Run IDs: needles ap-pBrqbRbG2UJqb5HvQDbMg4 · timing ap-U44I0iXVh2vrYX9ALCqIeB · byte-exactness ap-fMsMp4N2eYHHmfwRXqOCGG. Receipt JSON: receipts/batch9_128k_2026-08-14/.
Three arms, same command, same GPU (A100-80GB), retrieval gate per arm: fraQtl D2 vs fp16 (stock vLLM) vs fp8-KV (kv_cache_dtype=fp8).
| ctx | arm | decode tok/s | NIAH | KV pool (tokens) |
|---|---|---|---|---|
| 8K | fraQtl D2 | 115.9 | PASS | 1,012,896 |
| 8K | fp16 | 125.74 | PASS | 424,496 |
| 8K | fp8 | 128.08 | PASS | 834,912 |
| 32K | fraQtl D2 | 101.52 | PASS | 984,432 |
| 32K | fp16 | 104.18 | PASS | 412,992 |
| 32K | fp8 | 105.81 | PASS | 811,920 |
| 128K | fraQtl D2 | 69.5 | PASS | 845,136 |
| 128K | fp16 | 52.59 | PASS | 361,776 |
| 128K | fp8 | 72.7 | PASS | 709,488 |
At 8K and 32K, batch-1 decode is weight-bound and D2 sits below fp16 and fp8 (115.9 vs 125.74 vs 128.08; 101.52 vs 104.18 vs 105.81). 128K cells: chunked prefill disabled, max_num_batched_tokens=131072, 1024 decode tokens (32-token subtractive timing is a measurement artifact at 128K). Receipt run IDs: 8K ap-E4uir4SUtT5YLmKSJq2diz, 32K ap-cQiTpbsP6wBYPUkZeXLjnm, 128K ap-QuvidvohDU0DVYEKUomCKt / ap-uCQmv4B5FI985G1lfyxitA / ap-Dgvl6PXVpRBc9JLx1b1U7M.
| users | fraQtl D2 | fp8 | fp16 |
|---|---|---|---|
| 1 | 69.5 | 72.7 | 54.0 |
| 2 | 100.8 | 97.8 | 66.6 |
| 3 | 116.5 | 110.2 | 62.1 |
| 4 | 130.1 | 117.6 | 67.3 |
| 5 | 118.7 | 105.0 | 61.8 |
| 6 | 140.9 | 99.6 | 67.2 |
| 7 | 121.4 | 105.0 | 65.0 |
Preemption-free capacity: D2 6 users, fp8 5, fp16 2. At each arm's own clean maximum: D2 140.9 tok/s @ 6 · fp8 105.0 @ 5 · fp16 66.6 @ 2, 2.1× fp16 and 1.3× fp8 aggregate throughput. D2 unchunked prefill holds ≈3,950 tok/s at every rung (fp16 3,820, fp8 2,720). Run IDs: ap-uCQmv4B5FI985G1lfyxitA, ap-Dgvl6PXVpRBc9JLx1b1U7M, ap-BLmwYqynVkeUk5S6i8s9zG.
| users | fraQtl D2 | fp8 | fp16 |
|---|---|---|---|
| 8 | 335.9 | 349.6 | 265.6 |
| 12 | 363.0 | 259.5 | 254.9 |
| 16 | 408.5 | 333.9 | 233.0 |
| 20 | 409.1 | 379.8 | 249.1 |
| 24 | 429.7 | 432.3 | 246.0 |
At 24 users @32K: 1.75× fp16; fp8 is at 432.3 vs D2's 429.7. The gap over fp8 opens at 128K, where fp8's pool runs out first. Run ID: ap-sISJ6RY7TOm3WQASL8YCJX.
Needle-in-a-haystack passkey grids, 7 depths × 3 keys per context, exact-match gated: 21/21 per arm at 8K, 32K, and 128K; 189/189 total. Run IDs: ap-l7cHhG8pwbKqndTqKRxBfU, ap-YzAkwaYtW7FrUuFf5vAfhC, ap-zjyaiI4b7Rl4VujTuozW7O, ap-utQpsypNBJQrwrXHzRqhTN.
ngram drafting over the compressed cache at 9 users × 131K (c=9): 1.04× with the bit-exact paired-token kernel (run ap-7Mfjwuudo5qNpDG86QHfaO). Mean acceptance length ~4.0 with plain ngram lookup at 131K. Middle-rung receipts (c=4) in progress.
All nine batch-1 cells were re-run (2026-07-06) through the exact public path in this repository (prebuilt wheel + these sidecars, no internal source): all cells NIAH PASS; D2 KV pools byte-identical at all three contexts (1,012,896 / 984,432 / 845,136); decode within run-to-run band. Receipt JSONs: receipts/repro_2026_07_06/.
| File | What |
|---|---|
sidecar_real_u_qwen3-4b-instruct-2507.bin | V eigenbasis sidecar (per-layer, per-kv-head U matrices) |
sidecar_real_u_qwen3-4b-instruct-2507.meta.json | V sidecar provenance (dims, calibration corpus/size, sha256) |
qwen3-4b-instruct-2507-k16-int3.fraqtl-k-eigenbasis.bin | K eigenbasis sidecar (RAQK format, rank 16) |
qwen3-4b-instruct-2507-k16-int3.fraqtl-k-eigenbasis.receipt.json | K sidecar build receipt |
receipts/repro_2026_07_06/ | Independent reproduction receipts |
fraqtl_repro_receipts.py | One-command reproduction script |
Same recipe, receipt format, and measurement standards per model: Mistral-7B-Instruct-v0.3 (live). Next: Llama-3.1-8B, DeepSeek-R1-Distill-Llama-8B, Mistral-Nemo. Request a model by opening a discussion; calibration is one factory run.
The sidecars in this repository also power the fraQtl membrane inside a llama.cpp fork, with compressed KV read natively at decode.
Decode tok/s, single sequence (identical prompts; baseline stated per row):
| Context | membrane | q8_0 baseline | ratio | baseline build |
|---|---|---|---|---|
| 8K | 120.8 | 127.5 | 0.95× | same binary |
| 32K | 101.4 | 90.0 | 1.13× | upstream master (commit pinned in receipt) |
| 128K | 68.8 | 38.4 | 1.79× | upstream master (commit pinned in receipt) |
Upstream fp16 KV at 128K: 74.1 tok/s; membrane reaches 92% of fp16 decode with ~2.45× less KV memory. At 8K the membrane is below q8_0 (120.8 vs 127.5).
Concurrent capacity ("users" = N parallel sequences via -np N, each with an independent KV stream and a full 32,832-token context allocation; exact command line embedded in every receipt JSON):
| 32K tokens/user | membrane-exclusive | q8_0 |
|---|---|---|
| Max users before OOM | 36 | 28 |
| Aggregate tok/s at max | 335.5 | 164.2 |
Retrieval verification: needle-retrieval at 7 depths (0.05–0.95) × distinct high-entropy keys at 8K and 32K, plus 128K at depths 0.1/0.5/0.9: 24/24 retrieved; q8_0 baseline 7/7 at every depth. Multi-user isolation verified (distinct keys per stream, users finishing at different times).
Prefill: 128K ingest 54.5s vs q8's 31.6s (1.7×); numbers in receipts.
Runtime ships as a prebuilt binary, same posture as the vLLM wheel: you get the built runtime and the calibrated sidecars; kernel source is not distributed. Download: fraQtl/fraqtl-membrane-llamacpp-runtime (Linux x86_64, CUDA, SM80+SM90, manifest-sha'd). Flags: FRAQTL_MEMBRANE=1 FRAQTL_MEMBRANE_EXCLUSIVE=1 + --fraqtl-kv --fraqtl-eigenbasis <v-sidecar> --fraqtl-kv-protect 32 --fraqtl-k-eigenbasis <k-sidecar> --fraqtl-sink-tokens 0 --fraqtl-residual-window 0
Weights are quantized Q4_K_M (bartowski GGUF); fraQtl compresses the KV cache, not the weights. Short-context (≤8K) decode is ≈95% of q8_0. All receipts (JSON, with exact cmd + env + upstream commit) are reproducible via the fraQtl gate harness.
The on-device lane, calibration-aware Hi-Fi GGUFs: Gemma-4-E2B (runs an offline phone agent) · Llama-3.1-8B + Llama-3.2-1B draft pair · Qwen3.6-35B MoE. Org page: huggingface.co/fraQtl.