Downloads · 30 days
0
eemin/Qwen3.8-27B-NVFP4-lmhead4-recipe
Qwen3.8-27B-NVFP4-lmhead4-recipe is a machine learning model from eemin. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
No weights here. Runnable scripts + every number measured on the box, with error bars.
Downloads · 30 days
0
Access
Public
Updated Aug 23, 2026
Repo size
—
Likes
0
Public
Click a slice to open those files.
.py25.8 KB · 47%
From the Hugging Face model README
No weights here. Runnable scripts + every number measured on the box, with error bars.
| value | |
|---|---|
| Short context | 162.6 tok/s |
| 63K context | 81.0 tok/s |
| Usable context | 134,016 tokens |
| Hardware | RTX 5090 32 GB, SM120, PCIe 3.0 |
| Engine | SGLang lmsysorg/sglang:qwen38-27b |
| Model | Qwen3.8-27B NVFP4 + lm_head quantized to NVFP4 |
Starting point was 68 tok/s. Everything below is how it got to 162.
./scripts/00-preflight.sh # checks GPU, driver, docker, disk
./scripts/01-download-model.sh # pulls the NVFP4 checkpoint
./scripts/02-quantize-lmhead.py # lm_head BF16 → NVFP4 (+27% speed, +53% context)
./scripts/03-serve.sh # tuned launcher
Verify:
python3 scripts/bench/agentic_bench.py --host 127.0.0.1 --port 30000
python3 scripts/bench/correctness.py --host 127.0.0.1 --port 30000 # quality gate
--attention-backend flashinfer # 2.7× vs triton at long context
--kv-cache-dtype fp8_e4m3 # 4-bit KV is 4× SLOWER
--speculative-algorithm EAGLE
--speculative-num-steps 5
--speculative-eagle-topk 1
--speculative-num-draft-tokens 6 # 5/1/6 — beat 2/1/3, 3/1/4, 4/1/5, 7/1/8, 9/1/10, 11/1/12
--mamba-radix-cache-strategy extra_buffer_lazy
--max-running-requests 1
--max-mamba-cache-size 8 # = requests × 8. See the coherence rule below.
--mamba-full-memory-ratio 0.9 # do NOT raise. See below.
--page-size 32
--chunked-prefill-size 2048
--mem-fraction-static 0.97
--reasoning-parser qwen3 --tool-call-parser qwen3_coder
--max-mamba-cache-size must equal requests × 8Hybrid GDN models reserve a recurrent-state pool. Each request needs 8 slots
(4 for the target with extra_buffer_lazy + 4 for the MTP draft).
Ask for more requests than your slots allow and SGLang silently clamps — the log prints your requested value and then the clamped one, with no warning:
max_running_requests=4 ← what you asked for
max_running_requests=2 ← what you actually got
And concurrency is expensive. Measured, same battery, only this pair changing:
| requests | slots | KV pool | short | 63K |
|---|---|---|---|---|
| 1 | 8 | 134,016 | 162.6 | 81.0 |
| 2 | 16 | 100,480 | 160.6 | 78.0 |
| 4 | 32 | 33,440 | 160.0 | doesn't fit |
Concurrency costs 75% of your context and returns nothing in single-stream speed.
--context-length does not reserve anythingIt only declares. The number that matters is max_total_num_tokens, printed at startup.
--context-length 159776 ← what we declared
max_total_num_tokens 134016 ← what actually fits
Tell your client the second number. A client that believes the first will send a request that fails at runtime instead of truncating cleanly.
--mamba-full-memory-ratio — the official value is wrong for long contextThe SGLang playground raises it to 8.26 when MTP is on. Measured effect on our box:
| ratio | KV pool |
|---|---|
| 0.3 | 143,008 |
| 0.9 (default) | 94,112 |
| 3.0 | 48,640 |
| 8.26 (official w/ MTP) | 39,072 |
The official value cuts context by 58%. It is sized for many concurrent users, not for a single long-context stream. Copying it without measuring halves your context.
| item | GB | note |
|---|---|---|
| Model weights (NVFP4) | 18.59 | |
| MTP / speculative decoding | 5.53 | only 0.85 GB is weights; the rest is buffers |
| GDN state pool (8 slots) | 1.94 | ~83 MB per slot |
| KV cache | 3.88 | |
| free | 0.68 |
34 KB per token of context. The MTP costs ~160K tokens of context — that is the central trade of this model.
mem-fraction 0.90)| with MTP | without | |
|---|---|---|
| context | 62,880 | 272,800 |
| short | 165.0 | 74.2 |
| 63K | 83.0 | 57.1 |
| 128K | doesn't fit | 45.7 |
| 200K | doesn't fit | 36.5 |
MTP roughly doubles speed and costs 4× context. Below ~130K it always wins.
⚠️ Without MTP you must drop to --mem-fraction-static 0.90. At 0.97 the pool grows so
large that no workspace is left and the first request dies with
Triton Error [CUDA]: out of memory — the server boots and answers /v1/models
perfectly, then breaks the moment it generates.
| technique | why |
|---|---|
--fp4-gemm-backend flashinfer_cudnn | −20% short, −30% at 63K. auto→cutlass is already best on SM120 |
--fp4-gemm-backend flashinfer_cutedsl | not supported on SM120 (it's SM100) |
--attention-backend triton | +11% at 1K, −63% at 50K |
--speculative-attention-mode decode | no gain, nominally worse at long context |
--enable-fused-qk-norm-rope | zero gain |
--speculative-adaptive | −21% at 63K, doubled variance |
--speculative-algorithm NGRAM | 3 attempts, 3 OOM — allocates outside the static pool |
HiCache (--enable-hierarchical-cache) | crashes: 'HybridLinearKVPool' object has no attribute 'layer_num' |
--enable-int8-mamba-checkpoint | −20% KV pool, no gain |
--mamba-track-interval 64 | nothing measurable |
--tokenizer-worker-num 4 | ~890 MB VRAM per worker → OOM |
--enable-mixed-chunk | silently ignored with spec decoding (server_args shows False) |
--max-mamba-cache-size 24 / 32 | 24 → context drops to 12,288 · 32 → server hangs with no error |
Deeper chains 7/1/8 … 11/1/12 | accept len rises 3.39→3.73 while throughput falls 25% |
| KV 4-bit / TurboQuant / PolarQuant | 4× slower; TurboQuant needs plain GQA, not hybrid |
| DSpark · DFlash · Weaver | all lost to the native MTP head |
| Quantizing the MTP head | SGLang forces quant_config=None for modelopt NVFP4; mtp.fc is a plain nn.Linear |
The prefix cache matches on exact prefix. Anything volatile at the top of your system
prompt (session_id, turn counter, clock, current file, token count) changes every call
and invalidates everything after it.
| layout (~50K stable context) | TTFT |
|---|---|
| volatile at the top | 5.54 s |
| volatile at the end | 0.43 s (steady state) |
Same information, same tokens, same answer. Only the order changes. Costs nothing.
LiteLLM running with --detailed_debug serializes and logs the whole payload per request.
With a 63K-token prompt that is ~250 KB formatted in Python before the model is called.
| at 63K context | with | without |
|---|---|---|
| effective tok/s | 39.5 | 81.8 |
| streaming TTFT | 2.87 s | 1.00 s |
Its own litellm_overhead_latency_metric reported 0.6 ms while the real cost was
2.1 s — the metric only wraps the LLM call. The signal is in
litellm_request_queue_time_seconds.
Five methodology errors produced five confident, wrong conclusions in one day.
input throughput. That metric divides by the
interval since the previous prefill log — in multi-turn chat it includes the last turn's
entire decode. Reading 96 tokens / 4.5 s as prefill duration produced a whole false theory.0.0 tok/s
with plausible TTFT — the prompt had overflowed the pool and the script counted zero chunks
without checking HTTP status.--mem-fraction-static to 0.93 first. Three OOMs came from adding
a feature at 0.97. But then compare against a baseline at the same fraction — 0.93 costs
−30% at 63K on its own.| file | what |
|---|---|
scripts/00-preflight.sh | GPU / driver / docker / disk checks |
scripts/01-download-model.sh | pulls the NVFP4 checkpoint |
scripts/02-quantize-lmhead.py | lm_head BF16 → NVFP4 via modelopt |
scripts/03-serve.sh | tuned launcher |
scripts/bench/agentic_bench.py | throughput by workload type (prose / code / tool call / thinking) |
scripts/bench/correctness.py | quality gate by objective correctness, not string diff |
scripts/bench/multiturn_bench.py | long-context multi-turn |
AGENTS.md | step-by-step for an AI agent to reproduce this |
Switching between two ~70K contexts costs 12.3 s of TTFT, and no memory configuration changes it. Five configs from 112K to 143K of KV pool: 12.31–12.49 s, every time.
The log shows #cached-token: 0 with 43% of the pool free — the radix tree loses a
prefix it saw one turn earlier while having room to spare. Four hypotheses tested and
rejected (KV eviction, GDN state replay, general space, state-slot count).
Likely a limitation of SGLang's hybrid cache with alternating contexts, reinforced by
HiCache failing in the same HybridLinearKVPool. Unresolved — reported here rather
than explained away.
Built by Calney Gerhardt · @calneymgp