mistral-7b-instruct-v0.3-kv-sidecars

Mistral-7B-Instruct-v0.3: fraQtl KV-Cache Compression Sidecars

Model Overview

Description

2.4× the KV-cache capacity of fp16 (1.2× of fp8-KV) for Mistral-7B-Instruct-v0.3 in vLLM on NVIDIA A100: 997,200 tokens vs 412,544 (fp16) and 825,104 (fp8) at 8K. Batch-1 decode ≈95% of fp16 at 8K (85.7 vs 90.39 tok/s) and 76.84 vs 77.98 tok/s at 32K. Needle-in-a-haystack grids 21/21 per arm at every published point.

The sidecars are calibrated per-layer, per-KV-head eigenbasis artifacts consumed by the fraqtl runtime (a prebuilt binary vLLM attention backend). Base model weights are not modified or redistributed.

License/Terms of Use

Sidecars and receipts: Apache 2.0. Base model weights download from mistralai/Mistral-7B-Instruct-v0.3 under its own license. The fraqtl runtime wheel is proprietary, free to install and run for verification and evaluation (license).

Deployment Geography

Global.

Use Case

Teams serving LLMs with vLLM who need more concurrent users per GPU, longer contexts within a fixed memory budget, or lower KV-cache cost per token.

Release Date

Hugging Face 07/2026 via this repository. Companion model: Qwen3-4B-Instruct-2507 kit (includes 128K-context and concurrency receipts).

References

• Runtime wheel + build provenance: fraQtl/fraqtl-sm80-runtime
• Original three-arm receipts: receipts/ · Independent reproduction: receipts/repro_2026_07_03/

Model Architecture

Artifact type: Calibrated KV-cache eigenbasis sidecars (K + V) consumed by a compressed-page attention runtime for vLLM.

Compression architecture: fraQtl inserts a compression membrane between the model and vLLM's paged KV cache. On write, K and V are stored in a compressed page format: a calibrated protected subspace kept at high precision (rank 16 of 128 for K, rank 32 of 128 for V, per layer per KV head, the sidecars in this repository) plus an INT4 tail on the remaining dimensions. Logical rank stays 128; nothing is truncated. On read, the attention kernel consumes compressed pages directly at tensor-core speed, with no decompress-then-attend step, so capacity gains do not cost decode bandwidth. Recipe label: sm80_k16prot_k112i4_v32prot_v96i4_rv128.

How the protected subspaces are calibrated is not published. The sidecars are the calibrated artifacts, sufficient to run and verify every number on this page.

Input / Output

• Input: text (any workload the base model supports)
• Output: text
• Context length: receipted at 8K and 32K (Mistral-7B's native window). Not receipted at 128K: the model collapses under YaRN extension past 32K, and fp16 fails the retrieval gate there too (receipted). For 128K receipts see the Qwen3-4B kit.

Software Integration

Supported runtime engine: vLLM 0.20.2 (torch 2.11.0, the receipt-validated stack)

Supported hardware microarchitecture: NVIDIA Ampere (SM80 / A100). Hopper/Blackwell not yet supported.

Operating system: Linux

Integration uses vLLM's standard out-of-tree plugin mechanism (vllm.general_plugins entry point): pip install the wheel and the backend registers in every engine worker process.

Calibration Dataset

• Dataset: wikitext-2-raw-v1
• Data collection method: Automated
• Properties: K sidecar: 16 sequences × 1024 tokens (test split). V sidecar: 128 sequences, 65,536 total tokens. Per-layer, per-KV-head. No task data, no benchmark data, no retrieval-test content.

Usage

Reproduce every number on this page with one command (≈$12 of rented A100 time):

pip install modal && modal setup
modal secret create huggingface HF_TOKEN=hf_...
modal run fraqtl_repro_receipts.py            # mistral is the default model

Prints the three-arm table with per-arm needle-in-a-haystack results. Docker fallback: --local. Script: fraqtl_repro_receipts.py.

Evaluation

Three arms, same command, same GPU (A100-80GB), retrieval gate per arm: fraQtl D2 vs fp16 (stock vLLM) vs fp8-KV (kv_cache_dtype=fp8). CUDA graphs on, prefix caching off, batch 1.

Batch-1 decode and KV capacity

Batch-1 decode is weight-bound at these contexts; D2 is below fp16 and fp8 in every decode cell (85.7 vs 90.39 / 90.05 at 8K; 76.84 vs 77.98 / 82.81 at 32K). The 32K three-arm receipt JSONs are in receipts/.

Retrieval verification

Needle-in-a-haystack passkey grids, 7 depths × 3 keys per context, exact-match gated: 21/21 per arm at 8K and 32K; 126/126 total.

Independent reproduction

All six cells were re-run (2026-07-03) through the exact public path in this repository (prebuilt wheel + these sidecars, no internal source): all cells NIAH PASS; fp16/fp8 KV pools byte-identical to the receipts; D2 decode 85.51 @8K / 78.24 @32K and D2 pools within 2% of the receipts. Receipt JSONs: receipts/repro_2026_07_03/. Decode speeds vary a few percent run-to-run with host clocks; pools and retrieval results are deterministic.

Scope

• Retrieval = NIAH passkey grids, exact-match gated. The evidence standard is retrieval verification, not bit-identity; compression is lossy by construction.
• fp8-KV appears in every table. Batch and context are stated on every number; batch-1 short-context decode is weight-bound, so no KV method moves it.
• Behavior on tasks other than retrieval is not separately benchmarked here.

Methods note

• Pool size at 32K vs 8K: the pool is what remains after vLLM reserves working memory at engine init (0.85 × 80 GB − weights − peak activation scratch − CUDA-graph memory). Activation scratch is profiled against the largest possible prefill, which grows with max_model_len, so a 32K engine holds back more scratch for every arm (fp16 shrinks 428K→408K the same way). Page compression is unchanged; the ratios are context-independent.
• 2026-07-03: a batch-routing bug (mixed prefill+decode steps) dropped needles at batch ≥ 2; fixed, re-receipted. Pre-fix multi-user results were never published.

Model Limitations

• Hardware: SM80 (A100) only; the runtime binary will not run on H100 or consumer GPUs.
• Context: 8K–32K for this model (base-model YaRN limitation, see Input/Output). 128K and concurrency receipts live on the Qwen3-4B kit.
• The base model's own limitations and biases are inherited unchanged (weights are not modified).

Files

More Models

Same recipe, receipt format, and measurement standards per model: Qwen3-4B-Instruct-2507 (live, with 128K + concurrency receipts). Next: Llama-3.1-8B, DeepSeek-R1-Distill-Llama-8B, Mistral-Nemo. Request a model by opening a discussion; calibration is one factory run.

More from fraQtl

The on-device lane, calibration-aware Hi-Fi GGUFs: Gemma-4-E2B (runs an offline phone agent) · Llama-3.1-8B + Llama-3.2-1B draft pair · Qwen3.6-35B MoE. Org page: huggingface.co/fraQtl.