qwen3.8-27b-yarn-nvfp4-sm120

Qwen3.8-27B · YaRN + NVFP4 KV on one RTX 5090 (SM120) — reproduction page

This repository contains no model weights. The weights are upstream's unchanged NVFP4 artifact neroued/Qwen3.8-27B-nvfp4-NInfer (qwen3_8_27b_nvfp4.ninfer, 21,492,695,040 bytes, sha256 bb3360522a06e136e0367f5703414d26272b7285c8a6ab6194135c17dbd81b32). Nothing here was retrained, fine-tuned or re-quantized, so duplicating 20 GB of weights would add nothing. This page documents a runtime configuration — the engine build, flags and measurements — that runs that artifact past its native context on a single 32 GB RTX 5090.

Code: github.com/splickz/ninfer-yarn-nvfp4 (release v0.1.0-yarn-nvfp4-sm120, commit b5f2d1a9).

What this is, and is not

The memory side (how many tokens fit) is upstream's NVFP4 KV compression. The positional side (how positions past the trained window are encoded) is the YaRN option added in the linked repository. They are separate problems; this page is about the combination.

Measurements

One RTX 5090 (32,607 MiB), SM120, WSL2 Ubuntu 22.04, CUDA 13.1, driver 610.47. Weights 19.0 GiB resident. CUDA graphs on, speculative decoding off, prefix reuse off, single request, --rope-yarn-factor 4. Retrieval: 5 codewords at 10/25/50/75/90% depth, question at the end, each codeword scored individually, finish_reason=stop required. Prompt tokens are the engine's own usage.prompt_tokens. Prefill tok/s from streamed time-to-first-token.

Planner frontier: nvfp4 650,000 and k8v4 500,000 are refused by the engine's reservation check on this card (12,056,834,048 bytes available). Nothing bypasses the allocator.

Control: at 262,144 tokens, factor 1 and factor 4 reserve identical KV (4.82 GiB) and leave identical free memory (6.12 GiB). Factor 1 with the flag is byte-identical to upstream without it.

Reproduce

git clone https://github.com/splickz/ninfer-yarn-nvfp4.git
cd ninfer-yarn-nvfp4
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel

hf download neroued/Qwen3.8-27B-nvfp4-NInfer qwen3_8_27b_nvfp4.ninfer --local-dir models

# practical: 500K, factor 4, NVFP4 KV
./build/apps/ninfer-serve models/qwen3_8_27b_nvfp4.ninfer \
  --port 5800 --kv-dtype nvfp4 \
  --rope-yarn-factor 4 --rope-original-max-position 262144 \
  --max-context 500000

# retrieval harness (needles at 10/25/50/75/90%, streamed timing)
python3 bench/ctxbench.py 5800 nvfp4-f4 270000 390000 490000

For the 600K edge use --max-context 600000; expect ~400–500 MiB free after startup and roughly half the 500K throughput. Requirements and the full flag reference are in the GitHub README.

Caveats

• Needle retrieval is a smoke test. It does not demonstrate lossless quality at 4× the trained window; YaRN past ~2× typically costs perplexity. Evaluate your own task before relying on it.
• 600K is at the edge of 32 GB: prefill 1,357 → 655 tok/s and decode 48.6 → 33.1 from 500K to 600K, cause not isolated (memory pressure at ~98% VRAM under WSL2 or the split-K clamp are the candidates).
• MTP / DFlash speculative decoding is not validated with this port.
• k8v4 costs ~40% more KV memory per token than nvfp4 and does not fit 500K on this card.
• Experimental; three small commits over a moving upstream.

Attribution

• Engine, NVFP4/K8V4 KV cache and the model artifact: Neroued/ninfer and neroued on Hugging Face. Apache-2.0.
• Base model: Qwen/Qwen3.8-27B (Apache-2.0), quantized via unsloth/Qwen3.8-27B-NVFP4.
• YaRN option, ceiling changes, attention fixes and these measurements: splickz.
• Separate earlier work by the same author on a KV-compression approach (E8 lattice, rk4v4-e8): splickz/ninfer-rk4v4-e8 / splickz/qwen3.8-27b-rk4v4-e8-sm120.