Qwen3.8-27B-TeichAI-Fable-INT8-W8A16-MTP

<div align="center"> <img src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/logo_qwen3.png" width="380" alt="Qwen"> <h1>Qwen3.8-27B TeichAI Fable · INT8 W8A16 · BF16 MTP</h1> <p><strong>A Fable-5 distilled, Ampere-optimized quantization for dual RTX 3090 inference.</strong></p> <p> <a href="https://huggingface.co/TeichAI/Qwen3.8-27B-Fable-Distill">Base model</a> · <a href="https://huggingface.co/Qwen/Qwen3.8-27B">Qwen3.8-27B</a> · <a href="https://github.com/vllm-project/vllm">vLLM</a> · <a href="https://github.com/vllm-project/llm-compressor">llm-compressor</a> </p> <p> <img alt="Base Fable-Distill" src="https://img.shields.io/badge/base-Fable--Distill-8B5CF6"> <img alt="Format W8A16" src="https://img.shields.io/badge/format-W8A16-0A2EFE"> <img alt="Weights INT8" src="https://img.shields.io/badge/weights-INT8-6C4CF1"> <img alt="Activations FP16 or BF16" src="https://img.shields.io/badge/activations-FP16%20%2F%20BF16-555555"> <img alt="Target Ampere" src="https://img.shields.io/badge/target-Ampere%20sm__86-76B900"> <img alt="License Apache 2.0" src="https://img.shields.io/badge/license-Apache--2.0-blue"> </p> </div>

[!NOTE] This is a numerical W8A16 quantization of TeichAI/Qwen3.8-27B-Fable-Distill, a Fable-5 light tune (distill) of Qwen/Qwen3.8-27B. All model credit belongs to TeichAI (the Fable-5 tune) and Qwen (the base model); refer to the upstream model card for architecture, capabilities, and usage guidance. This repository changes numerics only.

[!TIP] This build targets vLLM on Ampere-class GPUs where native FP8 tensor-core execution is unavailable. It retains the model's vision tower, thinking controls, native context, and the byte-preserved BF16 MTP head. The Fable-5 tune is a language-model weight change; the vision tower, recurrent GDN gates, lm_head, and MTP head fall outside the W8A16 target set and are carried through at source precision.

Relationship to the sibling quantization

This checkpoint is the Fable-5-distilled counterpart of lued/Qwen3.8-27B-INT8-W8A16-MTP. The repositories share:

• the identical W8A16 compressed-tensors recipe (400 packed GEMMs, same BF16 preservation set, group-128 symmetric data-free RTN);
• the identical native BF16 MTP head (15 tensors, byte-identical to Qwen's original MTP — same sha256 as the sibling release);
• the identical tokenizer, chat template, processor configs, and bounded-shard layout;
• the same total checkpoint footprint (29.44 GiB).

The only deviation is the base language-model weights: this checkpoint's LM weights are TeichAI's Fable-5 light-tune weights instead of stock Qwen3.8-27B.

Fable-5 distillation (inherited from the base)

Per the upstream card, the source is a light tune of Qwen/Qwen3.8-27B on the Fable-5 datasets (armand0e/claude-fable-5-claude-code, armand0e/Fable-5-Chat) plus a large corpus of private Fable 5 data, trained with Unsloth and Hugging Face TRL. Upstream-reported benchmark results versus the base: ARC Challenge 0.637 (0.591), ARC Challenge (Easy) 0.832 (0.782), BoolQ 0.911 (0.896). These are the upstream card's claims, not measurements made here. The W8A16 step touches only the 400 GEMMs; the recurrent GDN gates (96), vision tower (333 tensors), lm_head, and MTP head are preserved at source precision.

Quantization fidelity

Measured on this exact checkpoint against its own BF16 teacher (the TeichAI Fable-Distill source) on 2026-08-18 with tools/measure_kld.py (teacher-forced, full-vocabulary FP32 log-softmax, FP64 accumulation, nats/token; PyTorch 2.11.0+cu128, Transformers 5.10.1, compressed-tensors 0.17.1). This is the v0.2 revision: every preserved tensor is a bit-exact copy of the source (input_layernorm, post_attention_layernorm, q_norm/k_norm, vision tower, embeddings, lm_head, GDN gates); only the 400 projection GEMMs are quantized.

The KLD result measures checkpoint-weight drift over 467 teacher-forced positions (fixed 8-prompt short suite). It is not a functional quality score; tool use, coding, multimodal quality, and long-context recall require separate behavioral evaluation. A greedy KV-cache smoke generation (finite logits, non-degenerate output) passes on every release.

Short-prompt suite

kld_result.json in this repository is the single source of truth for these numbers.

Checkpoint profile

MTP preservation

The MTP head is the exact BF16 MTP from the original Qwen3.8-27B — never re-quantized:

• 15 tensors in model_mtp.safetensors (849,400,392 bytes).
• sha256 1d8268aa85ace093a561e3e7b63b9d390dac1cd55a90cd55b5ec509c3c9da9fe — identical to the MTP bytes in the Qwen3.8-27B source and every family release. Verify with sha256sum model_mtp.safetensors.
• All 15 tensors verified bfloat16, shape-equal, and bit-equal to the TeichAI Fable-Distill source (strict audit, not a count check); the built model_mtp.safetensors sha256 matches the family canonical hash.

Reproduction contract

The immutable source is TeichAI/Qwen3.8-27B-Fable-Distill commit ac2b133895580be05ee90a6a7cdcc23cdf998bd0. The exact selective W8A16 configuration is recorded in recipe.yaml and config.json. A conforming rebuild must take native MTP from that source, reproduce the source-derived packed projection set, and prove all 15 MTP tensors bit-identical before resharing for publication.

Status

v0.2 revision (2026-08-18). This checkpoint is built on the v0.2 pipeline: every preserved (non-quantized) tensor is a bit-exact copy of the TeichAI source — machine-audited (STRUCTURAL AUDIT PASS including the value round-trip and preserved bit-identity checks; MTP AUDIT PASS) — and KLD re-measured on the v0.2 bytes at 0.000700 nats/token, 98.29% top-1 (v0.1: 0.000660/98.50%; both within the family's measured range, and the difference is noise-scale — see the family CHANGELOG.md for the v0.1→v0.2 provenance story). Originally published 2026-08-18 at commit f41c1353d1cf238f631d42a1d0845d0471666f44 (v0.1); the v0.2 revision is a new commit on the same repo — prior commits remain fetchable by hash, so existing downloads keep working. Serving was validated on the v0.1 bytes: boots through llama-swap from the lued/* cache entry, loads the CompressedTensorsWNA16 → MarlinLinearKernel path, detects the native MTP draft model (method=mtp, 3 speculative tokens), and serves a smoke request with the qwen3 reasoning parser. The BF16 parent source remains cached; deletion is the pipeline's final step and has not been requested.