Qwable-9B-Claude-Fable-5-SHQ8-GGUF

Qwable-9B-Claude-Fable-5 — SHQ Family (Selective Hybrid Quants)

Adaptive quantization family for Qwen3.5-9B hybrid Gated DeltaNet + Full Attention architecture. Each variant uses real imatrix data to selectively assign precision per-tensor.

Model: empero-ai/Qwable-9B-Claude-Fable-5 — Claude Fable 5 reasoning fine-tune of Qwen3.5-9B.

Note: The model file SHQ8-Q5_K_M.gguf is named for HF parser compatibility. This is NOT a pure Q5_K_M quantization — it's a hybrid using Q5_K_M base with Q8_0, Q5_K, Q4_K, and F16 selectively applied across different tensor types. Check the config for exact per-tensor types.

Note: SHQ8-v2-Q5_K_M.gguf is also a hybrid — Q5_K_M base with Q8_0 on critical attention, Q6_K on mid attention, IQ4_XS on low-importance tensors. See configs/SHQ8_v2.sh for the exact per-tensor map.

Speed: ~22 tokens/sec on GTX 1070 (8 GB VRAM) with flash-attn for SHQ-OptA, ~26 tokens/sec for SHQ8-v2 (fewer Q8_0 tensors → less data through Pascal's PCIe).

I've put a lot of work into hand-tuning these quants — let me know how they run on your hardware! Drop a comment or open a discussion with your setup and any feedback.

SHQ Family Tree

SHQ Series (Selective Hybrid Quantization)
│
├── SHQ6 ── Gemma 4 12B IT QAT
│   (QAT model, IQ4_XS all FFN + Q8_0 attention)
│   └── gemma4/README.md
│
├── SHQ7 ── Gemma 4 12B OBLITERATED
│   (non-QAT, gradient IQ4_NL + Q4_K + IQ4_XS + Q8_0)
│   └── gemma4/OBLITERATED_SHQ7_README.md
│
├── SHQ8 ── Qwen3.5-9B fine-tunes (non-QAT, Gated DeltaNet hybrid)
│   │
│   ├── Qwable-9B-Claude-Fable-5 (this)
│   │   ├── SHQ-OptA ★ WINNER
│   │   │   Q5_K_M base + Q8_0 on attn_gate/qkv/ssm_alpha/beta + imatrix
│   │   │   └── 6303 MiB, PPL 7.3747 — beats Q6_K in size + quality
│   │   ├── SHQ8-v2 — compact champion
│   │   │   Q5_K_M + selective Q8_0/Q6_K/IQ4_XS by imatrix
│   │   │   └── 5726 MiB, PPL 7.4559 — 18% smaller than OptA
│   │   ├── SHQ8-v3 — IQ4_NL experiment
│   │   │   Q5_K_M + all attn Q8_0 + IQ4_NL on imp ffn_down
│   │   │   └── 6041 MiB, PPL 7.4456 — 4% smaller than OptA
│   │   ├── SHQ8-v3a — all-attn Q8_0 test
│   │   │   Q5_K_M + all attn Q8_0 + ffn_down blk.20-30 Q8_0
│   │   │   └── 6301 MiB, PPL 7.4310 — same size, +0.056 vs OptA
│   │   └── Exp1 ── aborted
│   │       IQ4_XS all FFN (wrong for non-QAT)
│   │
│   └── Qwythos-9B-Claude-Mythos-5-1M
│       ├── SHQ8-OptA ★ quality champion
│       │   Same formula as OptA above
│       │   └── 6303 MiB, PPL 7.4831 — best PPL on Qwythos
│       ├── SHQ8-v2 — compact champion
│       │   Same tiered IMatrix approach as v2 above
│       │   └── 5726 MiB, PPL 7.6542 — 577 MiB less than OptA
│       └── MTP variants (built-in draft head for speculative decoding)
│           ├── M-SHQ8-MTP-OptA
│           │   OptA + blk.32 (MTP) at Q8_0
│           │   └── 6.40 GB, PPL 7.4831 — quality champion
│           └── M-SHQ8-MTP-v2
│               v2 + blk.32 (MTP) at Q8_0
│               └── 5.83 GB, PPL 7.6542 — compact champion

TL;DR

SHQ-OptA is the quality champion (best PPL). SHQ8-v2 is the compact champion (18% smaller than OptA at +0.081 PPL).

Key Result

Q5_K_M base + Q8_0 on attn_gate/qkv/ssm_alpha/ssm_beta + blk.31.ffn_down Q8_0 + imatrix per-row refinement.

• PPL 7.3747 vs Q6_K 7.4394 (Δ = -0.065, within margin but directionally better)
• Size 6303 MiB vs Q6_K 7008 MiB (10% smaller)
• Imatrix auto-promotes high-importance tensors (ffn_down in deep layers → Q6_K)
• Manual overrides take precedence: attn_gate/qkv → Q8_0 everywhere, blk.31.ffn_down → Q8_0

SHQ8-v2 — Real IMatrix Tiered Precision

Q5_K_M base + imatrix per-row + tiered precision by real importance data.

Real importance values from the actual Qwable-9B-Claude-Fable-5.imatrix.gguf (248 tensors, 319 chunks) revealed a 33,000× range — from blk.31 ffn_down at 323K to blk.0-5 ssm_out at 15. This enables safe aggressive quantization on low-importance tensors.

Key insight — real imatrix ≠ README claims: The README's importance matrix table (1M+ for attention) was from a different file. The actual model's imatrix shows ~2× lower values and a different layer ordering (L0 dominates at 494K, not mid layers). This changes which tensors need high precision.

Size: 5,726 MiB (5.36 BPW) — 577 MiB (-9.2%) less than SHQ-OptA.

PPL: 7.4559 ± 0.05 — matches Q6_K (7.4394) within margin at 18% smaller size. Slightly higher than SHQ-OptA (7.3747) but saves 577 MiB (9.2%). A favorable size/quality trade-off.

Speed: ~26 tokens/sec on GTX 1070 — 18% faster than SHQ-OptA thanks to fewer Q8_0 tensors reducing VRAM bandwidth pressure.

Architecture

Same as Qwen3.5-9B — hybrid Gated DeltaNet + Gated Attention (3:1 pattern):

Full architecture details in SHQ-QWEN35-INSIGHTS_CODER.md.

How We Made It

Step 1: Q6_K baseline (PPL 7.4394)

Standard Q6_K with mradermacher's imatrix. Fits 8 GB VRAM at 7008 MiB.

Step 2: The IQ4_XS trap (aborted)

We tried IQ4_XS on all FFN tensors across all layers — blk.\d+.ffn_(gate|up|down)=IQ4_XS. This was a bad design for a non-QAT model: ffn_gate/up have 572K importance (far above the 150K IQ4_XS safety threshold). The run was aborted before completion.

Step 3: CODER insights rescue

Existing experiments on Qwopus3.5-9B-Coder-MTP (same architecture, different fine-tune) provided critical evidence (tested at ctx=2048):

150K importance threshold: IQ4_XS safe only on tensors with importance < 150K:

Full analysis in QUANT_HISTORY_CODER.md.

Step 4: SHQ-OptA — Winner

Chose the proven SHQv2 formula adapted for Qwable:

Base:        Q5_K_M (imatrix per-row refinement)
Attention:   Q8_0 — attn_gate, attn_qkv, ssm_alpha, ssm_beta (ALL layers)
Outlier:     Q8_0 — blk.31.ffn_down (most important tensor)
Output:      Q5_K
Token embd:  Q4_K
Norms:       F16, attn_q/k_norm Q8_0

Imatrix auto-promotes:

• ffn_down in deep layers (18-30) → Q6_K (higher importance)
• attn_v in some layers → Q6_K

Manual Q8_0 overrides are never downgraded.

Imatrix Notes

The importance matrix (Qwable-9B-Claude-Fable-5.imatrix.gguf) was computed by mradermacher from imatrix-training-full-3 — the same dataset used for fine-tuning the model itself. This gives unusually accurate importance signals.

Key findings from imatrix:

• Inverted importance: Upper layers more important (blk.31 = 12× blk.1)
• blk.31.ffn_down is the single most important tensor (3.97e+09)
• Attention = 42.5% of total importance, FFN = 30.1%, SSM = 27.4%
• Layer 1 is anomalously low (7.26e+08) — embedding boundary

Quantization Commands

SHQ-OptA

llama-quantize \
  --imatrix Qwable-9B-Claude-Fable-5.imatrix.gguf \
  --output-tensor-type Q5_K \
  --token-embedding-type Q4_K \
  --tensor-type "output_norm.*=Q8_0" \
  --tensor-type "blk\.\d+\.attn_gate=Q8_0" \
  --tensor-type "blk\.\d+\.attn_qkv=Q8_0" \
  --tensor-type "blk\.\d+\.ssm_alpha=Q8_0" \
  --tensor-type "blk\.\d+\.ssm_beta=Q8_0" \
  --tensor-type "blk\.31\.ffn_down=Q8_0" \
  --tensor-type ".*attn_q_norm.*=Q8_0" \
  --tensor-type ".*attn_k_norm.*=Q8_0" \
  --tensor-type ".*ssm_conv1d.*=Q8_0" \
  --tensor-type ".*ssm_dt.*=F16" \
  --tensor-type ".*ssm_a=F16" \
  --tensor-type ".*attn_norm.*=F16" \
  --tensor-type ".*post_attention_norm.*=F16" \
  --tensor-type ".*ssm_norm.*=F16" \
  Qwable-9B-Claude-Fable-5-bf16.gguf \
  Qwable-9B-Claude-Fable-5-SHQ-OptA.gguf \
  Q5_K_M

SHQ8-v2

Critical: --tensor-type rules are first-match-wins. Specific overrides must come before generic.

llama-quantize \
  --imatrix Qwable-9B-Claude-Fable-5.imatrix.gguf \
  --output-tensor-type Q5_K \
  --token-embedding-type Q4_K \
  --tensor-type "output_norm.*=Q8_0" \
  --tensor-type ".*attn_q_norm.*=Q8_0" \
  --tensor-type ".*attn_k_norm.*=Q8_0" \
  --tensor-type ".*ssm_conv1d.*=Q8_0" \
  --tensor-type "blk\.31\.ffn_down=Q8_0" \
  --tensor-type "blk\.31\.attn_output=Q8_0" \
  --tensor-type "blk\.0\.attn_gate=Q8_0" \
  --tensor-type "blk\.0\.attn_qkv=Q8_0" \
  --tensor-type "blk\.0\.ssm_alpha=Q8_0" \
  --tensor-type "blk\.0\.ssm_beta=Q8_0" \
  --tensor-type "blk\.(26|27|28|29|30|31)\.attn_gate=Q8_0" \
  --tensor-type "blk\.(26|27|28|29|30|31)\.attn_qkv=Q8_0" \
  --tensor-type "blk\.(26|27|28|29|30|31)\.ssm_alpha=Q8_0" \
  --tensor-type "blk\.(26|27|28|29|30|31)\.ssm_beta=Q8_0" \
  --tensor-type "blk\.(3|27|31)\.attn_q\.weight=Q8_0" \
  --tensor-type "blk\.(3|27|31)\.attn_k\.weight=Q8_0" \
  --tensor-type "blk\.(3|27|31)\.attn_v\.weight=Q8_0" \
  --tensor-type "blk\.([1-9]|1[0-9]|2[0-5])\.attn_gate=Q6_K" \
  --tensor-type "blk\.([1-9]|1[0-9]|2[0-5])\.attn_qkv=Q6_K" \
  --tensor-type "blk\.([1-9]|1[0-9]|2[0-5])\.ssm_alpha=Q6_K" \
  --tensor-type "blk\.([1-9]|1[0-9]|2[0-5])\.ssm_beta=Q6_K" \
  --tensor-type ".*attn_norm.*=F16" \
  --tensor-type ".*post_attention_norm.*=F16" \
  --tensor-type ".*ssm_norm.*=F16" \
  --tensor-type ".*ssm_dt.*=F16" \
  --tensor-type ".*ssm_a$=F16" \
  --tensor-type ".*ssm_out=IQ4_XS" \
  --tensor-type ".*attn_output=IQ4_XS" \
  --tensor-type ".*ffn_down=IQ4_XS" \
  Qwable-9B-Claude-Fable-5-bf16.gguf \
  SHQ8-v2-Q5_K_M.gguf \
  Q5_K_M

All Experiments

Usage

llama.cpp

llama-cli \
  -m SHQ8-Q5_K_M.gguf \
  -p "Write a Python function for merge sort" \
  -ngl 99 \
  --flash-attn on \
  -c 4096

For server mode (recommended for agent/coding use):

llama-server \
  -m SHQ8-Q5_K_M.gguf \
  -c 65536 \
  -fit off \
  -ngl 99 \
  --jinja --chat-template-file /path/to/chat_template.jinja \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --port 8080 \
  --mmap \
  --temp 1 \
  --top-p 0.95 \
  --min-p 0 \
  --top-k 20 \
  --seed -1 \
  --parallel 1

Note: The chat template is from Unsloth (chat_template.jinja). For agent/coding tasks, temp 1 + top_k 20 + top_p 0.95 works well. Use -c 65536 for long context tasks.

VRAM usage: ~7399 MiB / 8192 MiB (90%) at -c 65536 with flash-attn + q8_0 KV cache. Room for more context thanks to the Gated DeltaNet architecture's efficient memory footprint.

From wepiqx: I would love to see this fine-tune released with MTP (Multi-Token Prediction) support — would make for an even better quantization. If Empero adds MTP, I'll update this quant with a specialized MTP draft model for faster inference.

Ollama

1. Create a Modelfile:

FROM ./SHQ8-Q5_K_M.gguf

TEMPLATE """{{ .System }}
{{ .Prompt }}"""

SYSTEM """You are a helpful coding assistant."""

PARAMETER num_ctx 8192
PARAMETER temperature 1
PARAMETER top_k 20
PARAMETER top_p 0.95

1. Build and run:

ollama create qwable-SHQ8 -f Modelfile
ollama run qwable-SHQ8

Important — sampling (wepiqx): I strongly recommend not using repeat_penalty (keep at 1.0 / off). Testing shows it degrades output quality on this quant. Use temperature 1 (general) or temperature 0.6 (precise coding tasks). top_k 20 recommended. The model's native repetition handling is deliberately tuned and doesn't need penalizing.

LM Studio

1. Open LM Studio
2. Drag SHQ8-Q5_K_M.gguf into the app
3. Set GPU Offload to 99 layers
4. Enable flash-attention
5. Set context length to 8192+
6. Crucial — disable repeat_penalty (set to 1.0 or off). Use temperature 1 (general) or 0.6 (precise tasks), top_k 20
7. Start chatting

Coding Examples

These models generate full, working HTML/CSS/JS websites in a single pass. Both examples used the same prompt at temperature 1:

"I'm a dev, my audience is youth. I like a creative/tech style. Write the full website code. This HTML will be our foundation."

SHQ-OptA — SHQ8_temp-1.html

A complete, styled developer portfolio in 324 lines:

• CSS variables, background grid effect
• Typewriter animation (pure JS)
• Card grid with hover glow
• Terminal-style contact form
• Responsive design (mobile tweaks)
• JetBrains Mono font integration

SHQ8-v2 — SHQ8-v2_temp-1.html

Same prompt, 355 lines — cleaner code, no external dependencies:

• Ambient floating orb background (pure CSS)
• Typewriter effect + 3D tilt card (mousemove parallax)
• Code window with syntax-highlighted preview
• Custom cursor glow trail
• Responsive design with mobile-safe tilt disable
• Zero external dependencies — no Google Fonts, everything shipped inline

Lessons Learned

1. Imatrix from same dataset as training = gold. Importance signals are unusually clean.
2. IQ4_XS on FFN gate/up destroys non-QAT models. 150K importance threshold is real.
3. CODER experiments transferred 1:1. Same architecture, different fine-tune, same optimal strategy.
4. Q5_K_M + imatrix > Q6_K. The per-row refinement compensates for lower base bpw.
5. Manual --tensor-type overrides beat imatrix promotions. Q8_0 stays Q8_0.
6. Temperature 2 + SHQ-OptA gives incredible coding output. User reported impressive results.
7. Real imatrix data matters more than README claims. The actual importance matrix had 2× lower values and different layer ordering than documented, leading to a different optimal strategy.
8. ssm_out is quantization noise at 15-155 importance (10,000× below top tensors). Safe for IQ4_XS or even IQ2_XXS.
9. L0 and deep layers (26-31) need Q8_0; middle layers (1-25) suffice at Q6_K. Attention position matters more than attention type.

Future Work

• Try IQ2_XXS on ssm_out (importance 15-155 — near zero)
• Profile speed impact of IQ4_XS tensors on GTX 1070 (Pascal lacks IQ4 hardware)

Files

References

• Qwen3.5-9B
• Qwable-9B-Claude-Fable-5
• mradermacher i1 quants
• CODER quantization history
• CODER SHQ insights
• Gated DeltaNet paper