Downloads · 30 days
11
1% of all-time downloads
sahilchachra/Qwen3.6-27B-Fable-5-AWQ-W4A16
Qwen3.6-27B-Fable-5-AWQ-W4A16 is a image-text-to-text model from sahilchachra. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
AWQ 4-bit (W4A16) quantization of TeichAI/Qwen3.6-27B-Fable-5-Experimental — a Fable-5 reasoning fine-tune of Qwen3.6-27B (qwen35: a hybrid GatedDeltaNet (linear-attention) + full-attention model with a vision tower a…
Downloads · 30 days
11
1% of all-time downloads
All-time downloads
995
Public
Parameters
27.5B
26.8 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors26.8 GB · 100%
How the weights are stored.
I3218.8B · 68%
From the Hugging Face model README
AWQ 4-bit (W4A16) quantization of
TeichAI/Qwen3.6-27B-Fable-5-Experimental
— a Fable-5 reasoning fine-tune of Qwen3.6-27B (qwen3_5: a hybrid GatedDeltaNet
(linear-attention) + full-attention model with a vision tower and an MTP head).
Variant: AWQ W4A16 — 4-bit symmetric integer weights, group size 128, with
activation-aware scaling. Activations stay BF16.
Disk size: ~25 GB (24.98 GiB; vs ~52 GB BF16, ~2.1×)
Quantized by: sahilchachra
Tooling: llm-compressor 0.12 (AWQModifier + QuantizationModifier) → compressed-tensors (pack-quantized)
Why this model and not
Mia-AiLab/Qwable-3.6-27b: that repo ships only aQ4_K_MGGUF (no full-precision weights), so it cannot be honestly AWQ-quantized (AWQ needs BF16/FP16 source weights). This checkpoint quantizes the genuine Fable-5 BF16 fine-tune instead — the same Fable-5 reasoning lineage.
Only the language tower's standard linear layers are quantized (int4):
mlp.{gate,up,down}_proj on all 64 decoder layersself_attn.{q,k,v,o}_proj on the 16 full-attention layersKept in BF16 (not quantized): the 48 GatedDeltaNet linear_attn (mamba) layers
(no standard AWQ mapping, quantization-sensitive), the vision tower (model.visual.*),
the MTP head (mtp.*), token embeddings, lm_head, and all norms. The headline name
reflects the LM-tower quantization; on-disk size averages the int4 and BF16 parts.
AWQ is calibration-based. This checkpoint was calibrated on 256 sequences of 2048 tokens
drawn from armand0e/claude-fable-5-claude-code
— the fine-tune's own Fable-5 reasoning training data (Claude Code agent traces).
Transcripts are reconstructed turn-by-turn (user prompts + assistant reasoning/answers),
rendered through the model's chat template, and windowed — so AWQ's per-channel scales
reflect the model's real reasoning workload. Group size 128, AWQ scale search, duo_scaling.
Structure verified: 256 LM-tower modules carry weight_packed (int4) + weight_scale;
the 48 GatedDeltaNet layers, vision tower, MTP head, embeddings and lm_head remain BF16.
Functionally verified on an NVIDIA Thor by dequantizing the int4 weights to dense BF16 and
generating: the model reasons correctly (e.g. the "17 sheep, all but 9 run away" trick →
answers 9, and recognizes 17 is irrelevant) and produces coherent code reasoning,
matching the BF16 source's behavior. This is a functional smoke test, not a quality benchmark.
Load with trust_remote_code and sdpa attention. The GatedDeltaNet kernels
(causal-conv1d / flash-linear-attention) fall back to a pure-PyTorch path where
unavailable (e.g. ARM/Jetson) — correct, just slower.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "sahilchachra/Qwen3.6-27B-Fable-5-AWQ-W4A16"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo, dtype=torch.bfloat16, device_map="cuda",
attn_implementation="sdpa", trust_remote_code=True).eval()
msgs = [{"role": "user", "content": "Explain why the sky is blue, briefly."}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True,
return_tensors="pt", return_dict=True).to("cuda")
print(tok.decode(model.generate(**enc, max_new_tokens=256)[0]))
Loader note:
compressed-tensors == 0.17.1has a bug in its eagerdecompress_modelpath (it mis-readsgroup_sizeas 0 and raisesstrategy group requires group_size to be set to a positive value). This affects allpack-quantizedcheckpoints, not just this one. Use a compressed-tensors version where this is fixed, or serve with vLLM (its own loader is unaffected). The weights/scales on disk are standard and correct.
compressed-tensors pack-quantized. Custom code, processor/tokenizer and config
carried over from the source.See TeichAI/Qwen3.6-27B-Fable-5-Experimental and base Qwen/Qwen3.6-27B for architecture, intended use, capabilities and license (Apache-2.0, inherited).