Qwen3.6-35B-A3B-compressed

Newer version: this repo predates our Hi-Fi line. The current, calibration-aware build of this model, with KLD receipts, is fraQtl/Qwen3.6-35B-A3B-Hi-Fi-GGUF. Start there.

Looking for a GGUF for llama.cpp / Ollama / LM Studio? This repo ships fraQtl's compressed safetensors (transformers-runtime, V-cache compression). For a GGUF Q4_K_M of the same base model, drop-in for llama.cpp / Ollama / LM Studio / koboldcpp / Jan, see fraQtl/Qwen3.6-35B-A3B-GGUF (21.4 GB, ~30% lower KLD vs the Q8 teacher than a standard Q4_K_M on our measured code/math + general slices).

Qwen3.6-35B-A3B: fraQtl Compressed

25.53 GB on disk (2.74× smaller than ~70 GB FP16). 128K context on a single A100-80GB at 51.7 GB peak VRAM, where the stock FP16 setup OOMs at 64K. ∞Bench Passkey 30/30 at ~125K tokens. MMLU 82.24% vs 82.40% FP16 on the full 14 042 questions.

Weight-compressed Qwen3.6-35B-A3B artifact for transformers: INT3-packed MoE expert tensors, WikiText-2 calibration (v4). This release is the public weight-compressed artifact; fraQtl runtime KV-cache compression is a separate early-access layer.

For production inference pilots, custom compression workflows, and newer artifacts, see fraqtl.ai/contact or email [email protected].

Headline results

All measurements on a single NVIDIA A100-80GB. Single-seed; multi-seed re-verification in progress.

† ∞Bench KV-retrieval: n=1, 10, 100, 500, and 1500 needle settings (~101K-token context at n=1500). Compressed accuracy 80% (4/5) at every pair count, identical to FP16. The one failing case also fails at FP16 with the same example seed. Peak VRAM at n=1500: 48.7 GB on a single A100-80GB.

Installation

This model requires one extra package before you can load it from transformers. The compressed weights need a custom loader that reconstructs the packed MoE expert tensors at load time. That loader ships as a compiled binary in the fraqtl-runtime wheel; without it, from_pretrained(..., trust_remote_code=True) raises ImportError and the model does not load.

pip install fraqtl-runtime

No license token is required for loading or inference; the wheel is free to install. A token is only required for the optional runtime KV-cache compression layer (separate paid feature, see section below).

Platform: Linux x86_64 + CUDA validated (v0.1). macOS / ARM coming.

Once installed, use the standard transformers API as shown in the Quick Start below.

Quick Start

pip install fraqtl-runtime

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "fraQtl/Qwen3.6-35B-A3B-compressed"

model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained(repo)

prompt = "The capital of France is"
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
# → "The capital of France is Paris, a city renowned for its rich history…"

You never write import fraqtl yourself. The chain:

1. You call AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, ...).
2. transformers downloads modeling_qwen36_fraqtl.py from this repo (a small stub).
3. That stub does from fraqtl.modeling_qwen36_fraqtl import Qwen36CompressedForCausalLM.
4. The fraqtl-runtime wheel ships the compiled .so that this import resolves to.
5. The compiled loader reconstructs the packed MoE expert weights and returns the ready-to-use model.

Compression is part of how the artifact is stored, not a runtime mode you toggle. If the wheel is missing, the import fails with an error pointing back at pip install fraqtl-runtime.

What this release includes

This repository contains:

• A public compressed-weight artifact for Qwen3.6-35B-A3B.
• A lightweight loader for the compressed format.
• Benchmark results: perplexity, MMLU sample, ∞Bench passkey, ∞Bench KV-retrieval, qualitative generation, VRAM profile across contexts.

This repository does not include:

• The fraQtl calibration pipeline.
• The full compression engine.
• Runtime KV-cache compression.
• Custom model-packing tools.

Those components are distributed through the fraQtl runtime for approved early-access users.

Runtime KV-cache compression (separate layer)

This artifact is weight-compressed only. fraQtl runtime KV-cache compression, the additional live-cache compression at long context, is distributed separately as the fraQtl runtime for approved early-access users. It is not required to load or evaluate this model. To request access, contact fraQtl directly.

Benchmarks

Long-context retrieval

Stock FP16 baseline OOMs at 64K context on the same GPU.

Perplexity

FP16/BF16 baseline : 11.6041
fraQtl compressed  : 11.6930
Delta              : +0.0889 (+0.77%)

WikiText-2 test split, 16 × 256-token windows, greedy decode.

MMLU (full 14 042 questions, 57 subjects)

Matched shot-count and subject sampling against the FP16 reference run.

HumanEval pass@1 (full N=164)

HumanEval has a ~±3 pp 95% CI at N=164, so the +2.43 pp is inside the interval. Greedy decode, full 164 problems, single seed. The running average was 95% at N=20 and converged toward the baseline as the harder later problems came in, consistent with HumanEval's easier-early/harder-late ordering.

HumanEval+ pass@1 (full N=164)

Same harness as HumanEval, same N and CI.

BigCodeBench (N=200)

Identical pass count on identical problems.

TruthfulQA-MC2 (N=500)

FP16 reference measured to 450/500, with the final value within ±0.005 of 0.4553.

Qualitative generation (greedy)

No loops or gibberish observed across the qualitative smoke set.

Throughput (single A100-80GB)

100 fresh tokens generated after a real prefill at each context length, measured after one warmup pass.

This is not a real-time chat backend. The PackedFusedExperts loader dequantizes only the routed MoE experts per forward pass, trading persistent VRAM (25 GB resident vs ~70 GB FP16) for per-token compute. At ~0.85 tok/s, the use cases are long-context analysis, document processing, batch inference, and research workloads, not interactive chat.

Long-context perplexity (WikiText-103)

32 K PPL was not measured: transformers' default loss path materializes full fp32 logits and OOMs at that context. 32 K throughput and ∞Bench retrieval at 125 K both ran on the same hardware.

Calibration corpus and task-aware variants

This artifact (v4) uses WikiText-2 as the calibration corpus, chosen for general-purpose preservation across the benchmarks above. The calibration corpus determines which input directions are protected during compression, and protected directions correlate most strongly with downstream tasks that share input distribution with the calibration data.

GSM8K-CoT 5-shot is not in the headline table. v4 measures 57.67% under the matched harness; the FP16 baseline is 81.00%. WikiText calibration biases protected dimensions toward general-text fidelity, at the cost of math chain-of-thought reasoning.

A separate math-corpus-calibrated variant (MBPP calibration, internal codename Phase 1.5) measures above FP16 on GSM8K-CoT under the same harness:

Phase 1.5 is releasing as a separate artifact (fraQtl/Qwen3.6-35B-A3B-compressed-math or similar) with its own model card and full matched-FP16 sweep, as a study on calibration-corpus / task-aware compression.

Pick the artifact for your workload:

• General-purpose / code / long-context retrieval: v4 (this artifact)
• Math-CoT-heavy: the task-aware variant

Scope

• Single seed. Long-context numbers (passkey, KV-retrieval, VRAM) and MMLU are single-seed; multi-seed re-verification in progress.
• Long-context workload variance: evaluate on your own long-context workload before production use.
• English MMLU only. Multilingual performance has not been systematically re-measured after compression.
• Calibration: WikiText-2 (v4). GSM8K-CoT 5-shot: 57.67% vs 81.00% FP16.
• transformers version: requires transformers >= 4.51 for the Qwen 3.6 MoE architecture. Older versions do not recognize the model type.
• Platform: Linux x86_64 with CUDA validated. Untested on Apple Silicon / AMD ROCm.
• Fine-tuning: out of scope. The compressed artifact is inference-only.
• Live KV-cache savings require the fraQtl runtime (early access). Without it, only the weight compression is active; KV cache stays FP16.
• Runtime memory depends on context length, batch size, backend, and hardware.

License

The Qwen3.6-35B-A3B base model is licensed by Alibaba under its own license terms. fraQtl's compression artifacts and loader code in this repository are Copyright © 2026 fraQtl AI. The compressed weights are released for research and evaluation use; see LICENSE for terms. The fraQtl runtime package and full compression engine are proprietary; contact for commercial licensing.

Measure your own model

fraQtl maintains an open diagnostic tool for measuring model compressibility before deeper compression workflows:

pip install fraqtl-diagnostic
fraqtl analyze <any-HF-model>

• Source: https://github.com/fraqtl-ai/fraqtl-diagnostic
• PyPI: https://pypi.org/project/fraqtl-diagnostic/
• Apache-2.0 (the diagnostic only, not the compression engine).

Contact

• Site: https://fraqtl.ai
• Contact / pilots: https://fraqtl.ai/contact or [email protected]
• GitHub: https://github.com/fraqtl-ai
• Hugging Face: https://huggingface.co/fraQtl