Downloads · 30 days
0
haihengh/Qwen3.6-35B-A3B-finchmoe-3bit
Qwen3.6-35B-A3B-finchmoe-3bit is a machine learning model from haihengh. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for finchmoe. The card lists the license as apache-2.0.
Optimized weights for the FinchMoE C/Metal inference engine (SSD-streamed MoE, ~2 GB RAM, no llama.cpp needed).
Downloads · 30 days
0
Access
Public
Updated Aug 15, 2026
Repo size
16.1 GB
Likes
0
Public
Click a slice to open those files.
.bin16.1 GB · 100%
From the Hugging Face model README
Optimized weights for the FinchMoE C/Metal inference engine (SSD-streamed MoE, ~2 GB RAM, no llama.cpp needed).
Why this quant: 3-bit routed experts + 4-bit GDN/attention + 8-bit embed/lm_head — 14.9 GB total (~4.8× smaller than the 71.9 GB BF16 source), tuned for the engine's zero-copy expert streaming.
| File | Size | Content |
|---|---|---|
model_weights_quant.bin + .json | 1.95 GB | Non-expert weights: 4-bit affine (group 64) GDN/attention/shared experts, 8-bit embeddings + lm_head, BF16 norms/gates |
packed_experts_3bit/layer_00..39.bin | 14 GB | Routed experts, 3-bit per layer (256 experts × 1.31 MB) |
vocab.bin, tokenizer.bin | 8 MB each | Custom BPET tokenizer |
hot_sets.bin | 10 KB | Optional prefill hot-set prefetch data |
shaders.metal | 128 KB | Runtime-compiled Metal shaders |
The MTP files (model_weights_mtp.bin, layer_40.bin) are optional and not
included — the MTP head is not used.
git clone https://github.com/haihengh/finchMoE
cd finchmoe && make # requires macOS + Xcode CLT (Metal)
# download this repo's files next to the binary (or symlink them):
./finchmoe-infer -m . \
--weights model_weights_quant.bin --manifest model_weights_quant.json \
-P "Tell me a story about a lighthouse keeper." --tokens 200
| Component | Bits | CosSim vs BF16 |
|---|---|---|
| Routed experts (256×40) | 3-bit | 0.966-0.979 |
| Non-experts (GDN, attention, shared) | 4-bit | ≥ 0.995 |
| Embeddings + lm_head | 8-bit | near-lossless |
End-to-end quality passes the llama.cpp Q4_K_M baseline level on edge prompts; no quantization-driven long-generation drift (fresh-prefill differential cos 0.99942).
Weights derived from Qwen/Qwen3.6-35B-A3B (Apache-2.0).