Qwen3.8-27b-Terse-Coder-NVFP4

Qwen3.8-27B Terse-Coder — NVFP4

NVFP4 (modelopt W4A16) quantization of Shockem/Qwen3.8-27b-Terse-Coder, a fine-tune of Qwen/Qwen3.8-27B with ~1/10 the chain-of-thought reasoning tokens on coding tasks and correctness preserved. This is the tested deployment artifact — every number below was measured on this checkpoint.

Final release. This is the last round of the Terse-Coder study; this quant is the final deployment build.

Background

Qwen-3.8-27b is an excellent dense model. It's a breakthrough in locally hosted models on consumer-grade hardware. The biggest challenge it faces is that its reasoning can be, at times, overly verbose. This isn't necessarily an issue, as it can pull itself out of hallucinations, but with user hardware around this model size, it ends up causing long waits, oftentimes into hours, before any outputs or edits occur.

Recently, agentionai released Signal-3.8-27B. This opened the gates to the idea of lowering reasoning by finetuning the model, rather than handling it via templates or configuration. The outcome was a model that thought significantly less than the base model, with marginal differences in error. This sent me down a rabbit hole of testing its reasoning and how it affected the model's output. To my surprise, it was incredibly close to the base, even its traces were very close, just cleaner overall.

My experiment is to continue this research and push it further. So far, I've trained on traces from Signal, the base NVIDIA provided NVFP4 quant, and my own abliterated variant, against HumanEval, 600 questions per round. This is now round 7, and it has shown significant improvement. We are now at a 95% reduction in reasoning against the NVIDIA quant, and 52% against Signal. This was originally created as a LoRA adapter that is then merged into a custom recipe for Qwen-3.8-27b. I have provided both a merged model and a LoRA adapter. Thank you for testing and providing feedback!

Results

vLLM 0.28 on 2× RTX 5060 Ti 16 GB, MTP spec decode on, sampling temp 0.6 / top_k 20 / top_p 0.95 / rep-penalty 1.05. Independent benchmarks (reasoning = completion_tokens_details.reasoning_tokens):

Every external benchmark improved over the round-7 quant (98.0 / 78.3 / 90.2 / 78.6 respectively). CRUXEval was not re-run for round 8 (the round-7 quant measured 92.1% I / 92.9% O on the full 800).

A note on GPQA-Diamond: this is where a terseness fine-tune is supposed to bleed — PhD-level science, far outside the coding training distribution, where long deliberation is the whole game. Holding 79.8% at ~1.7k mean reasoning tokens (thinking models typically burn 10–20k here) means the training cut the deliberation budget, not the capability — the model still scales effort up on hard problems (median 884 → max 16k) instead of answering blindly fast.

Internal agentic harness (30 tests across easy/medium/hard — instruction following, coding, reasoning, compaction handoff, tool/JSON contracts — ×3 runs, this checkpoint served by vLLM): easy 12/12, medium 27/27, hard 41/42 — the single miss is a known LRU-codegen flake that also appears in the round-7 profile.

Transparency note — internal held-out probe. Our internal 40-problem held-out coding set (disjoint from training; 20 HumanEval + 20 MBPP-sanitized, same sampling, pass@1 by automated test execution) measures 61% on this checkpoint across two runs (mean ~38 reasoning tokens, vs ~701 for the stock NVFP4 base). The same round-8 adapter served as a runtime LoRA measures 70% on that probe, and the round-7 quant measured 67.5% — a small merge-attenuation tax specific to the merged 4-bit form (documented on the adapter repo). Every external benchmark above was measured on this exact checkpoint and improved.

Runs at stock-base wall speed with MTP acceptance 0.412 (identical draft stack to round 7) — the reasoning cut is free end-to-end.

Quantization recipe

This is a v3-recipe house quant, built to preserve the adapter effect through 4-bit compression:

• modelopt 0.45 W4A16 NVFP4, per-tensor streaming PTQ (the same 400-tensor quantize set + ignore list as the published house Signal quants)
• FP8 attention (absmax — byte-matches NVIDIA's checkpoint at 97–99%)
• Local-Hessian-weighted calibration on MLP + lm_head (Hessian captured from 2048 house-traffic chunks; Hessian-weighted MSE scale solve with per-block e4m3 bracketing). This matters: an absmax-calibrated quant of the same weights attenuates the terse-reasoning effect to roughly half (−49.5% vs −92.4% cut measured). Geomean Hessian-weighted error ratio 0.805 vs the absmax baseline on the stock base.
• MTP draft stack included (1 MTP layer, BF16, vocab-truncated 40960-id draft head) so speculative decoding works out of the box.

Serving (vLLM, tested path)

Chat template: use Shockem/froggeric-terse-coder (based on froggeric/Qwen-Fixed-Chat-Templates) (also bundled here as chat_template.jinja) — it carries the anti-rumination tool rules the model was evaluated with; serving without it changes agentic behavior.

vllm serve Shockem/Qwen3.8-27b-Terse-Coder-NVFP4 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --kv-cache-dtype fp8

Turn MTP spec decode on — outputs are target-verified (lossless) and acceptance is 0.41. If you serve with spec decode, make sure the generation config has no min_p — vLLM 0.28 rejects min_p under spec decode.

Recommended sampling (mirrors testing): temp 0.6, top_k 20, top_p 0.95, repetition_penalty 1.05.

On 2×16 GB cards cap context at ~200k with a ~3.9 GiB FP8 KV pin; single-card 24 GB+ rigs are unaffected.

Notes

• Do not stack the Terse-Coder adapter on this checkpoint — the preference is already merged in; double application over-shortens reasoning (63% pass with no_code failures).
• The fp16 source weights are at Shockem/Qwen3.8-27b-Terse-Coder if you want to quantize differently or merge further.
• Behavioral edit, not a knowledge edit — targeted at coding with thinking enabled. Should work on other backends (SGLang, TabbyAPI/EXL3), but only vLLM has been measured; validate before relying on them.

Harness recommendations

How you drive the model matters as much as which build you run. Measured on Terminal-Bench 4.0 (Sep 2026, local vLLM serving): prompt-level guardrails did not reduce token burn or stop grinding — the dominant cost was the harness re-sending the full transcript every step. Zero tool-call loops observed in any trajectory.

• Let the server own sampling — omit temperature/top_p/top_k; the checkpoint config applies temp 0.6 / top_k 20 / top_p 0.95 / repetition_penalty 1.05. Never send min_p (vLLM rejects it under MTP spec decode).
• Serve with the bundled chat template (chat_template.jinja; also at Shockem/froggeric-terse-coder). The anti-rumination rules inject even alongside custom system prompts — bypassing the template (bare completions API, client-side template) is the "repeats already-done steps" failure mode.
• Per-turn max_tokens 4–8k is generous (routine coding answers are 25–150 tokens; thinks median ~38). Use 49152 only as a session floor for agents, not a per-turn target.
• Context management is the biggest lever. Keep ~3 recent tool results, cap older blobs at ~4k chars. Unbounded agent history was worth ~270k+ input tokens per task in our probe — that is a harness property, not a model property.
• Think text arrives in reasoning_content — empty content with non-empty reasoning_content is a normal turn, not an error.
• Raise reasoning_effort for genuinely hard problems — the model still scales deliberation up when the problem needs it (GPQA median ~900 reasoning tokens); terseness targets waste, not deliberation.

Attributions & licenses

This checkpoint is a quantized derivative of Shockem/Qwen3.8-27b-Terse-Coder, itself a derivative of Qwen/Qwen3.8-27B, © Qwen Team, Alibaba Cloud, licensed Apache 2.0; this checkpoint remains Apache 2.0 and the original license and copyright notices are retained. Credits:

• Qwen Team (Alibaba Cloud) — the Qwen3.8-27B base model (Apache 2.0).
• NVIDIA — TensorRT Model Optimizer 0.45 (Apache 2.0) drove this NVFP4 quantization; NVIDIA's published Qwen3.8-27B-NVFP4 checkpoint informed the Hessian-calibrated recipe.
• agentionai and p-e-w (Heretic) — Signal and a heretic-ara variant were two of the three trace-generation policies in the upstream adapter's preference data.
• OpenAI (HumanEval, MIT) and Google (MBPP, CC-BY 4.0) — prompt sources for training and held-out evaluation.
• Hugging Face TRL (Apache 2.0) — DPO training; Datacurve — DeepSWE, independent evaluation only.

None of these parties endorse this model; all remaining errors are ours.