Qwable-9B-Claude-Fable-5-GPTQ-Int4

<p align="center"> <img src="qwable9b.jpg" alt="Qwable-9B-Claude-Fable-5" width="500"/> </p>

Qwable-9B-Claude-Fable-5 · GPTQ Int4

This is a 4-bit GPTQ quantization of empero-ai/Qwable-9B-Claude-Fable-5, a full-parameter SFT distillation of Qwen3.5-9B on Claude Fable 5 and GPT-5.5 agentic-coding/reasoning traces. All capabilities, usage notes, limitations, and licensing from the original model card apply verbatim — please read it first: original README.

This is a community quantization by malvavisc0. It is not an official release from Empero. All credit for the model itself goes to Empero and the Qwen team.

---

TL;DR

---

What is (and isn't) quantized

GPTQ is applied only to the text backbone's linear layers. The following remain in bf16 (standard and recommended — they are small and/or accuracy-critical):

• the vision tower (Qwen3.5-Vision, ~0.6 GB) — the fine-tune was text-only anyway,
• lm_head (lm_head=false),
• embed_tokens (untied, 248,320 × 4096 — the largest single unquantized block),
• norms / embeddings / non-linear params.

That is why an "Int4 9B" model is ~8.1 GB rather than ~4.5 GB: roughly half the bytes are unquantized embedding tables. If you only need text inference, this is still a large VRAM win over the bf16 original (~18 GB) and runs on a single 12 GB GPU with offloading.

Quantized module classes (per layer, all 32 layers covered):

• linear_attn.in_proj_qkv, linear_attn.in_proj_z, linear_attn.out_proj (Gated DeltaNet linear-attention layers)
• self_attn.q_proj, self_attn.k_proj, self_attn.v_proj, self_attn.o_proj (full-attention layers, every 4th)
• mlp.gate_proj, mlp.up_proj, mlp.down_proj

---

Quantization configuration

{
  "bits": 4,
  "group_size": 128,
  "desc_act": false,
  "sym": true,
  "lm_head": false,
  "method": "gptq",
  "pack_dtype": "int32",
  "meta": {
    "quantizer": ["gptqmodel:7.3.4"],
    "uri": "https://github.com/modelcloud/gptqmodel",
    "damp_percent": 0.05,
    "damp_auto_increment": 0.01,
    "true_sequential": true,
    "static_groups": false,
    "act_group_aware": true,
    "mse": 0.0,
    "fallback": { "strategy": "rtn", "threshold": "0.5%", "smooth": null },
    "pack_impl": "cpu",
    "gc_mode": "interval"
  }
}

Calibration

⚠️ Calibration caveat. The base model is a coding/agentic/reasoning distill trained on traces up to ~74k tokens. This quantization was calibrated on general educational web text at 2048 tokens — it does not match the model's specialty domain and does not exercise long context. Expect the largest (still small, given the loss figures below) quality deltas on heavy coding tasks and very long inputs compared to the bf16 original. A coding/agent-trace calibration set at longer seq_len would likely improve this further.

Quantization loss (per-module MSE, from quant_log.csv)

Losses are uniformly tiny across all layers — no module tripped the 0.5% RTN fallback threshold. This indicates a clean, well-conditioned quantization with no outlier layers.

---

How to use

The architecture is multimodal (Qwen3_5ForConditionalGeneration). Load it the same way as the original — GPTQ weights are picked up automatically from quantization_config. You need gptqmodel (or a recent transformers + optimum GPTQ backend) and the Qwen3.5 kernels (flash-linear-attention + a CUDA-matched causal_conv1d), otherwise the linear-attention layers fall back to slow PyTorch ops.

import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer

model_id = "malvavisc0/Qwable-9B-Claude-Fable-5-GPTQ-Int4"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, dtype="bfloat16", device_map="auto",
)

messages = [{"role": "user", "content": "Write a Python function that merges two sorted lists."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to(model.device)

out = model.generate(
    **inputs, max_new_tokens=2048, do_sample=True,
    temperature=0.7, top_p=0.95, top_k=20, repetition_penalty=1.05,
)
# Output starts with a <!thinking>...</thinking> reasoning block, then the final answer.
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Sampling notes (inherited from the original card): the model is a reasoning model — every response opens with a <!thinking> block; parse/strip it for end users. repetition_penalty≈1.05 prevents rare non-terminating reasoning loops; allow generous max_new_tokens.

For vLLM / ExLlamaV serving, prefer their native GPTQ loader and a Marlin kernel for best throughput.

---

Known issues / caveats

• Spurious top-level rope_parameters in config.json (rope_theta: 10000.0, plain default rope). This field is not present in the original model and was injected during quantization; it conflicts with the correct text_config.rope_parameters (rope_theta: 10000000, mrope with section [11, 11, 10]). In practice the text sub-config takes precedence, but loaders that read the top-level field may emit warnings or apply the wrong RoPE. If you see odd long-context behavior, delete the top-level rope_parameters from config.json before loading.
• quantization_config.meta.offload_to_disk_path contains a leftover local scratch path (/tmp/gptqmodel_cfwi2by5) from the build host. It is metadata only and has no effect at load time.
• Vision path is untested. The fine-tune was text-only; the vision tower is preserved unquantized but is not validated. Treat this as a text model.
• No quantitative benchmarks were run on this quantization. The GPTQ loss figures above are the only quality signal; compare against the bf16 original on your own workload before relying on it.

---

Provenance

{
  "source_model": "empero-ai/Qwable-9B-Claude-Fable-5",
  "quantization": { "format": "gptq", "bits": 4, "group_size": 128, "desc_act": false },
  "calibration": { "dataset": "fineweb-edu", "samples": 256, "seq_len": 2048, "chat_template_applied": true },
  "versions": { "aft": "0.0.1", "python": "3.14.6", "torch": "2.13.0+cu130",
                "transformers": "5.14.1", "gptqmodel": "7.3.4" }
}

License

apache-2.0, inherited from the base Qwen3.5-9B weights and the original fine-tune. The fine-tuning data originates from generated traces of Claude Fable 5 and GPT-5.5 — see the original card's Provenance & licensing section for the third-party-terms caveat that applies to downstream commercial use.

Acknowledgements

• Original model: empero-ai/Qwable-9B-Claude-Fable-5 (Empero)
• Base: Qwen3.5-9B (Alibaba Qwen team)
• Quantization: gptqmodel by ModelCloud