Downloads · 30 days
53
6% of all-time downloads
ala-la/Qwen3.6-35B-A3B-pipeline-mixed34
Qwen3.6-35B-A3B-pipeline-mixed34 is a text generation model from ala-la. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
Compressed Qwen3.6-35B-A3B-abliterated-v2 (uncensored MoE, 70.2 GB FP16) to 19.94 GB using a 4-stage pipeline: AWQ + QuaRot + mixed 3/4-bit quantization + bias compensation.
Downloads · 30 days
53
6% of all-time downloads
All-time downloads
911
Public
Parameters
9.1B
187 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors19.9 GB · 99%
How the weights are stored.
U83.8B · 42%
From the Hugging Face model README
Compressed Qwen3.6-35B-A3B-abliterated-v2 (uncensored MoE, 70.2 GB FP16) to 19.94 GB using a 4-stage pipeline: AWQ + QuaRot + mixed 3/4-bit quantization + bias compensation.
PPL: +0.03% (28.377 → 28.387) measured in-memory on 50 calibration texts (~682 tokens).
| Metric | Value |
|---|---|
| FP16 PPL | 28.377 |
| Compressed PPL | 28.387 |
| Delta | +0.03% |
| FP16 size | 70.2 GB |
| Compressed size | 19.94 GB |
| Compression ratio | 3.5x |
This model uses custom QuantizedLinear layers not recognized by transformers.AutoModelForCausalLM. Loading via standard HF API produces random weights — the model will not work.
To use this model you need the pipeline code which defines QuantizedLinear and QuantizedMoeExperts classes. A proper load_quantized_model() function is not yet implemented — this is a research artifact, not a production-ready quantized model.
Not compatible with: llama.cpp, GGUF, vLLM, SGLang, Ollama, MLX, or any standard inference engine.
For production inference use lmstudio-community/Qwen3.6-35B-A3B-GGUF (Q4_K_M = 21.2 GB, works everywhere).
Scales salient weight channels based on activation magnitudes. Scales fused into RMSNorm weights.
Rotates weight matrices with random Hadamard matrix to remove outlier features. RMSNorm weights fused into adjacent linear layers before rotation (required for rotation equivariance). Final RMSNorm fused into lm_head.
Per-expert sensitivity analysis → 3-bit for less sensitive experts, 4-bit for sensitive ones. Group quantization (group_size=64) with scale + zero-point.
Corrects systematic quantization bias by comparing FP16 vs quantized layer outputs on calibration data.
7 bugs found and fixed during development:
| # | Bug | Effect | Fix |
|---|---|---|---|
| 1 | GatedDeltaNet wrong projection names | 30/40 layers unrotated | Fixed: in_proj_qkv/z/b/a + out_proj |
| 2 | MoE gate/router not rotated | Routing broken | Added gate + shared_expert_gate rotation |
| 3 | --skip-deltanet-rotation default=True | 30/40 layers skipped | Default=False |
| 4 | Per-layer hooks caused double rotation | Residual stream rotated N times | Removed hooks, rotate embed+lm_head instead |
| 5 | AWQ gate not scaled | Routing broken | Added gate + shared_expert_gate scaling |
| 6 | RMSNorm fusion used w instead of (1+w) | x2 multiplier instead of (1+w) | Multiply by (1+w), set weight to zeros |
| 7 | Final RMSNorm not fused into lm_head | +47% PPL (norm not rotation-equivariant) | Fuse final norm weight into lm_head before rotation |
Key insight: Qwen3.6 RMSNorm uses output = x_norm * (1 + weight), not x_norm * weight. Weight init=zeros, effective multiplier = 1+w. Fusion must multiply by (1+w) and set weight to zeros after.
| Run | Config | PPL | Delta | Size |
|---|---|---|---|---|
| run1 | QuaRot only (pre-fix) | 71893 | +253246% | 17.9 GB |
| run2 | QuaRot + 3/4-bit (pre-norm-fix) | 45.8 | +61.5% | 17.9 GB |
| run3 | QuaRot + 3/4-bit (all fixes) | 30.89 | +8.87% | 17.9 GB |
| run4 | AWQ + QuaRot + bias (30% 4-bit) | 30.04 | +5.86% | 17.9 GB |
| run5 | AWQ + QuaRot + bias (50% 4-bit) | 29.64 | +4.45% | 18.5 GB |
| run6 | All 4-bit, gs=64 | 28.23 | -0.53% | 21.2 GB |
| run7 | 50% 4-bit, gs=64 | 28.98 | +2.12% | 19.6 GB |
| run8 | 60% 4-bit, gs=64 | 28.39 | +0.03% | 19.94 GB |
| Test | Status | Notes |
|---|---|---|
| PPL (50 samples, 682 tokens) | ✓ | +0.03% vs FP16 (in-memory) |
| QuaRot rotation lossless | ✓ | -0.11% without quantization |
| HumanEval | ✗ | Not run |
| MMLU | ✗ | Not run |
| HellaSwag | ✗ | Not run |
| ARC | ✗ | Not run |
| GSM8K | ✗ | Not run |
| Long context | ✗ | Not tested |
Limitations:
load_quantized_model() function not implementedmodel-*.safetensors — quantized weights (custom format)config.json — model config (quantization_config: custom)quantization_meta.json — per-layer quantization metadatazero_loss_pipeline_cuda.py — pipeline code (QuantizedLinear, QuantizedMoeExperts)METHODS_LOG.md — full development historyRESULTS.md — run results tableTongyi Qianwen (same as base model).