Downloads · 30 days
48
2% of all-time downloads
hellohazime/Qwen3.8-2.4T-A95B-REAP-512GB-GGUF
Qwen3.8-2.4T-A95B-REAP-512GB-GGUF is a text generation model from hellohazime. Use it when you need the model to write or continue text. The card lists the license as other.
The unpruned Q2 quant of Qwen3.8 (UD-IQ2XXS, 656.6 GB) cannot run on a 512 GiB machine at all — it exceeds total RAM. This build prunes the experts an English + code deployment doesn't route to, keeping the healthier…
Downloads · 30 days
48
2% of all-time downloads
All-time downloads
2.2K
Public
Repo size
404 GB
Likes
0
Public
Click a slice to open those files.
.gguf404 GB · 100%
From the Hugging Face model README
The unpruned Q2 quant of Qwen3.8 (UD-IQ2_XXS, 656.6 GB) cannot run on a 512 GiB machine at all — it exceeds total RAM. This build prunes the experts an English + code deployment doesn't route to, keeping the healthier Q2 quantization intact: 404 GB (376 GiB), fully resident on a 512 GiB Mac at ~9.5 tok/s.
No fork, no PR branch, no custom runtime. This file loads and generates
on stock mainline llama.cpp — verified by running it there (greedy outputs
identical to our fork). Download and llama-server, nothing else.
| experts | 304 of 512 per MoE layer (uniform), selected by measured routing counts; width untouched (unlike our 256 GB sibling) |
| calibration | English web + code corpus (200k tokens) through the unpruned model; per-(layer, expert) router selection counts |
| selection coverage | 89.7% of routing selections retained (worst layer 83.2%) |
| quantization | untouched — byte-slab copy of UD-IQ2_XXS (layer-mixed ~1.9 bpw experts), no requantization |
| size | 404 GB = 376 GiB — full -ngl 99 offload fits under macOS's Metal working-set ceiling with room for KV |
| KLD vs unpruned UD-IQ2_XXS | held-out en: mean 0.110, median 0.024, argmax agreement 87.6%, PPL 8.70 → 9.31 (×1.07) · held-out code: mean 0.196, median 0.007, argmax agreement 90.0%, PPL 1.75 → 2.03 (×1.16) |
Expert selection is count-based (router hit counts), not gate-weighted REAP saliency — on Kimi-K3 this shortcut recovered 90.9% of saliency mass vs 93.5%. The counts come from the same measured workload as the 256 GB sibling; the two builds share their keep-set.
Like every calibration-pruned build: what the corpus leaves out is what gets deleted. Non-English languages and off-domain abilities are deliberately sacrificed.
Provenance: the exact pruning plan is published — plans/reap_plan_qwen_304.json (sha256 299b756b01231c43…, full hash in SHA256SUMS); plan + source quant + the MIT slicer reproduce this file's bytes. Split-half recalibration (same-distribution halves): keep-set overlap 92.7%, count-mass retention 98.7–98.9% (worst layer 97.0%) — details in plans/README.md.
Measured exactly as on the 256 GB sibling's card (teacher-forced held-out text, ~128k tokens per domain, base logits from the unpruned quant): the numbers answer how far does this build drift from the model it was cut from. Median KLD 0.007 on code means half of all code tokens are essentially untouched; the top-1 token survives 90% of the time. Side-by-side with the 256 GB build (which also cuts expert width and starts from a 1.56 bpw base), this build keeps roughly twice the fidelity on every metric — en argmax 87.6% vs 79.2%, code 90.0% vs 86.6%, PPL overhead ×1.07-1.16 vs ×1.20-1.26. That gap is the measured price of the last 158 GB.
Verified:
| claim | evidence |
|---|---|
| KLD / argmax agreement vs the unpruned quant | table above, held-out en and code |
| Slicing is lossless for surviving weights | identity prune byte-identical — pinned by tests in the tooling repo |
| Loads and generates | stock mainline llama.cpp (9.3 tok/s) and our fork (9.5 tok/s), greedy outputs identical, full -ngl 99 resident on an M3 Ultra 512 GB |
| Agentic sanity check | two labeled conditions, per our protocol: bare prompt 1/3 — two one-turn deaths obeying the benchmark's phantom ```python scaffold instruction (log-verified, same trap class as the parent's single bench fail) · with the standard counter-note (promptv1m) 3/3 — the trapped cells become normal grind-passes (20-min death → 74-min $1,000 win). Full data: swelancer-local-subset-evals |
What the sanity check actually demonstrates — and what most quant/prune releases never show: this build drove Qwen's own CLI agent through three real, paid SWE-Lancer tasks end-to-end — multi-hour tool-calling sessions (29 / 74 / 185 minutes) against a real React Native codebase — and solved all three ($2,000 in original prize value) under the disclosed conditions. Not a perplexity table: the model plans, edits files, runs shells, and finishes.
An owner-requested extension then ran the five hard tasks (the ones a 2-bit K2.7 baseline failed): 2/5, $6,000 — including the heaviest task in the set (a ~104k-token issue) solved in 99 minutes where the streamed unpruned parent needed 183, resident speed converting directly into solve time. Total across all eight: 5/8, $8,000 vs the parent's 7/8 — the KLD ladder (parent > this build > the 256 GB sibling) reproduces exactly as a bench ladder (7/8 > 5/8 > 3/8). Fidelity costs capability; this build is the measured middle point, and every cell is public: swelancer-local-subset-evals.
Verified operating envelope — every number on this card was measured at a stated point: quality (KLD / argmax / PPL) on 2048-token teacher-forced windows — the same regime the expert selection was calibrated in; agentic work served at context 131,072 through multi-hour sessions (up to 185 min). The model's trained window is 262,144 (GGUF metadata). Beyond these points — longer contexts, non-English work (deliberately cut by calibration), off-corpus domains — you are past our measurements.
llama-server -m Qwen3.8-2.4T-A95B-REAP-512GB-IQ2_XXS.gguf \
--port 8090 -ngl 99 -c 131072 --jinja \
--temp 1.0 --top-p 0.95 --top-k 20
Works on stock mainline llama.cpp (no fork needed; Qwen3.5-MoE support). Sampling per the Unsloth card; hybrid thinking model.
Same pipeline as the 256 GB sibling (tooling, MIT), minus the width cut: count the router selections over the calibration corpus, keep the top 304 experts per layer, slice the GGUF as raw byte slabs, renumber the router. blk.92 (a stored, never-executed MTP block) is sliced blind to keep the file loadable. No requantization anywhere.
Credits: Qwen team, Unsloth (UD-IQ2_XXS this inherits), Cerebras REAP, kimi-k3-mlx.
Qwen3.8のQ2量子化(UD-IQ2_XXS、656.6GB)は512GiB機のRAM総量を超えていて
そもそも載りません。これは英語+コード用途でほぼ呼ばれないexpertを実測
ルーティングで削って404GB(376GiB)にした版で、512GiB機に-ngl 99で
全載せ・約9.5 tok/sで動きます。幅は削っていないので、256GB版より忠実度が
全指標で約2倍良い(code argmax一致90.0%、KLD中央値0.007)。多言語は設計上
壊れています。素のmainline llama.cppで動作検証済み(fork不要)。単なる
perplexity表ではなく、Qwen公式CLIで実在の有償SWE-Lancerタスク3問を
数時間のツール往復の末に3問とも解き切ったことを検証済みです。