Downloads · 30 days
9
4% of all-time downloads
mmnga-o/Qwen3.8-REAP30-Width50-UD-gguf
Qwen3.8-REAP30-Width50-UD-gguf is a machine learning model from mmnga-o. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
Downloads · 30 days
9
4% of all-time downloads
All-time downloads
227
Public
Repo size
358 GB
Likes
4
Public
Click a slice to open those files.
.gguf357 GB · 100%
From the Hugging Face model README
An experimental reduced GGUF build of Qwen3.8-2.4T-A95B.
The routed-expert count is reduced by about 30%, then the intermediate width of each retained expert is reduced by 50%. The number of experts selected per token remains 10.
The calibration set contains 97 completed English and Japanese samples covering general chat and reasoning, code generation, and tool use. It contains 94,874 profiled tokens, of which 78,061 are semantic assistant tokens used for scoring. Prompt and structure tokens are excluded.
For expert e, the score is:
REAP(e) = mean[t routed to e](
top-10-renormalized router weight(t,e)
× ||unweighted expert output(t,e)||₂
)
Experts are ranked independently in each of the 92 MoE layers. The top 358 of 512 are retained. No experts are reserved manually.
After the REAP30 slice, the reduced model is profiled again with the same calibration set. Every intermediate channel is scored independently for each expert:
a(t,e,c) = SiLU(gate(t,e,c)) × up(t,e,c)
score(e,c) = Σ[t routed to e](router_weight(t,e) × a(t,e,c))²
× ||Wdown,e[:,c]||₂²
The top 1,024 of 2,048 channels are retained for each expert. Low-coverage experts use a blend of the activation score and a static weight-energy prior.
| Original | This build | |
|---|---|---|
| MoE layers | 92 | 92 |
| Routed experts per layer | 512 | 358 |
| Routed expert FFN width | 2,048 | 1,024 |
| Experts selected per token | 10 | 10 |
The router and grouped expert tensors are sliced with the same per-layer expert map. Shared experts are retained. MTP tensors are omitted.
Width selection is channel-level rather than 256-channel-block-level. Retained gate/up rows are copied directly. ffn_down_exps.weight is dequantized, gathered by the selected channel map, packed into new QK256 blocks, and requantized to the source GGUF type.
Both folders use the same REAP and Width rankings.
| Folder | Shards | Size |
|---|---|---|
UD-IQ1_S | 12 | about 184.72 GiB |
UD-Q1_0 | 10 | about 148.54 GiB |
Load the first shard in the selected folder. Each folder also includes slice_manifest.json with the applied ranking hashes and slice settings.
UD-Q1_0 uses the experimental IQ1_XXXS GGML type. Use the iq1-narrow branch:
git clone --branch iq1-narrow https://github.com/unslothai/llama.cpp
cmake -S llama.cpp -B llama.cpp/build \
-DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp/build -j --target llama-server
Example for a 32 GB GPU while keeping MoE weights on CPU:
./llama.cpp/build/bin/llama-server \
-m ./UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf \
--cpu-moe \
-ngl 80 \
--ctx-size 8192 \
--flash-attn on \
--cache-type-k q8_0 \
--cache-type-v q8_0
This configuration was tested with 512 GB system RAM and one 32 GB RTX 5090. It offloaded 80 of 92 layers and used about 30.6 GiB of VRAM. Adjust -ngl and context size for your hardware.
This is a heavily reduced experimental model. Quality and stability may differ from the original model and depend on how closely the calibration distribution matches the target workload.