Qwythos-9B-v2-Full-Ladder-KLD-GGUF

Qwythos-9B-v2 — full GGUF ladder · measured KLD fidelity + loop-rate test

This is NOT my model. Weights and training are by empero-ai (Qwythos-9B-v2, Apache-2.0). This is an independent GGUF repack whose value-add is what the other community GGUFs don't publish: an objective, measured fidelity sweep (KLD / PPL / Top-1 vs the Q8 reference) plus a real loop-rate benchmark on the quantized weights. No weights were modified.

What v2 fixes (per empero-ai): eliminated looping (6.7% → 0% at greedy decoding), restored the MTP head, cleaner identity, FTPO-trained. This repack is built from the official MTP-Q8_0 so the MTP head is preserved.

✅ Recommended files

The imatrix wins at every bit level. IQ4_XS dominates Q4_K_M (5.3 vs 5.8 GB, 96.2% vs 94.6% Top-1) and IQ3_M dominates Q3_K_M (4.5 vs 4.7 GB, 93.0% vs 90.8% Top-1, KLD 0.065 vs 0.102). For a 6 GB VRAM card, IQ3_M is the sweet spot — smaller and more faithful than plain Q3_K_M, with room left for context. Prefer the IQ tiers over the plain Q tiers at 3- and 4-bit.

📊 Measured fidelity — vs the Q8_0 reference

llama-perplexity --kl-divergence over 24×512-token chunks. KLD (Kullback–Leibler divergence) is the gold-standard quant-fidelity metric — it measures how far the quant's full next-token distribution drifts from the reference, catching damage that PPL and Top-1 alone miss. Top-1 match = how often the quant's argmax equals the reference's.

Reference = Q8_0, not BF16. Qwythos-v2 is a hybrid qwen35 arch (3:1 Gated-DeltaNet linear-attention : full attention + MTP), which llama.cpp does not lower cleanly from safetensors — so the ladder is requantized from empero-ai's official Q8_0 (near-lossless) with --allow-requantize + an imatrix, and fidelity is measured against that Q8_0.

🧪 Real loop-rate benchmark (measured on the quant)

v2's headline is "looping 6.7% → 0% at greedy decoding." We verified it on the quantized weights (Q5_K_M): greedy decode (temp 0), 10 diverse prompts × 220 tokens, flag any generation that degenerates into a repeated cycle or collapsed vocabulary.

Result: 0.0% loop rate (0/10 prompts) — confirms the quantization preserves v2's loop-fix — greedy (temp 0), 300 tokens each, mean repetition 0.038, every generation coherent (852–1527 chars incl. a naturally-repetitive counting prompt). (Method + raw per-prompt output in scripts/loop_bench.py + reports/loop_bench.json.)

🚀 Run it

ollama run hf.co/KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q5_K_M
llama-server -m Qwythos-9B-v2-Q5_K_M.gguf -c 32768

Sampling: qwen3.5-family reasoning models want temp 1.0 / top_p 0.95 / top_k 20. For Ollama use RENDERER qwen3.5 + PARSER qwen3.5 (recent build). v2 no longer loops at greedy, but temp 1.0 is still recommended for reasoning quality.

⚠️ Honest scope

Two things are measured here: (1) KLD/PPL/Top-1 fidelity vs Q8, and (2) a loop-rate test. This is not an independent capability benchmark — for agentic/reasoning scores (MMLU, GSM8K, HumanEval, GPQA), see empero-ai's card (their numbers, from their internal harness, not re-verified here).

🔁 Provenance

Requantized from the official MTP-Q8_0 with llama-quantize --allow-requantize + imatrix. Scripts in scripts/ (pipeline_v2.sh, parse_metrics.py, loop_bench.py), per-quant KLD logs in reports/, SHA-256 in reports/artifact-sha256sums.txt.

Importance matrix (qwythos-v2.imatrix, shipped in this repo): computed fresh on this model's own Q8 (not reused from other weights), over a general-purpose calibration corpus — a ~250 KB mix of general English prose and code (standard for imatrix generation; not weight-specific data). Reuse it directly for your own quants. Note: the imatrix does not cover the MTP/nextn head (block 32) — a normal forward pass doesn't activate it — so IQ3_XS bails ("missing importance matrix … very low-bit"); IQ3_M quantizes cleanly (it tolerates the uncovered head) and is the recommended 3-bit tier.

Credit

Model © empero-ai (Apache-2.0). This repack + measured metrics: KikoCis.