Downloads · 30 days
0
haihengh/Qwen3.8-Flash-Next-125B-finch-4bit-ple4bit
Qwen3.8-Flash-Next-125B-finch-4bit-ple4bit is a machine learning model from haihengh. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
A FinchMoE install of Qwen3.8-Flash-Next-125B (a 48-layer MoE, 512 experts, top-8) repacked into the .finch format for SSD-streaming inference on memory-constrained Apple Silicon. Every tensor class is int4/int8 affin…
Downloads · 30 days
0
Access
Public
Updated Sep 16, 2026
Repo size
104 GB
Likes
0
Public
Click a slice to open those files.
.bin104 GB · 100%
From the Hugging Face model README
ple4bit)A FinchMoE install of Qwen3.8-Flash-Next-125B
(a 48-layer MoE, 512 experts, top-8) repacked into the .finch format for
SSD-streaming inference on memory-constrained Apple Silicon. Every tensor
class is int4/int8 affine; this release is the first in which the PLE n-gram
hash table is too.
97 GiB on disk, down from 162 GiB in the earlier release, with the difference almost entirely the PLE table.
Qwen3.8-Flash-Next-125B-finch-4bit| previous release | this release | |
|---|---|---|
| PLE n-gram table | raw BF16 | int4 affine, group 32 |
ple_shards/ | 128 × 800,003,840 B | 128 × 250,001,200 B |
| install size | 162 GiB | 97 GiB |
| PLE read per token | 5 KB | 1.4 KB |
Nothing else moved: model_weights.bin and packed_experts/ are byte-identical
to the previous release, and the prompt becomes the same tokens. The manifest
declares the new table in an additive quant.pleNgram slot
({scheme: affine, weightBits: 4, groupSize: 32, scaleType: BF16, biasType: BF16}),
so an engine that does not understand it will not silently misread the table.
Each row is laid out fixed-stride as [packed nibbles][BF16 scales][BF16 biases],
so one pread still fetches a row's whole quantized state, and the decoder
derives the group size from the row itself.
Same engine, same flags, same prompts, previous release vs this one — the only variable is PLE row precision:
| prompt (prefill tokens) | argmax | top-10 logit cos (overlap) | top-100 | whole-vocab |
|---|---|---|---|---|
| 62 | match | 0.999953 (9/10) | 0.999865 | 0.996408 |
| 426 | match | 0.999940 (10/10) | 0.999880 | 0.997876 |
| 2940 | match | 0.999897 (10/10) | 0.999806 | 0.996318 |
For scale — not an apples-to-apples comparison, since that one differs in both weights and engine — the project's cross-quantization comparison against a llama.cpp GGUF sits at top-10 cos 0.995, so the PLE term is roughly an order of magnitude below the error the port already carries in the bands that decide behaviour.
Two further checks, independent of the engine: 4,000 sampled rows across all 128
parts have |new − old| / scale with max exactly 0.5000 and zero rows above
the half-step bound, which is the signature of round-to-nearest and nothing else;
and the previous release's table is a byte-identical copy of the source snapshot
(3,200 rows across 16 shards verified).
One honest caveat. The quantization perturbs the top-1 logit by a fraction of its margin, and on a prompt with a thin margin it can flip a greedy token — measured, not speculative: a prompt whose top-1 margin was 0.47 nats diverged after ~25 generated tokens. Both continuations stayed fluent. If you sample rather than decode greedily, or your margins are wide, this does not arise.
Measured against the previous release, same harness and protocol:
| previous release | this release | |
|---|---|---|
| base pass@1 | 0.945 (155/164) | 0.945 (155/164) |
| HumanEval+ pass@1 | 0.921 (151/164) | 0.909 (149/164) |
Three of 164 problems moved, in both directions on base (HumanEval/140
pass → fail, HumanEval/163 fail → pass), so base is unchanged and the stricter
suite loses two. Those flips are real rather than noise: every HumanEval cell
stays below the context length at which this engine stops being bit-reproducible
(2051 tokens), so the PLE quantization is the only variable between the two runs.
Build FinchMoE and point it at this directory:
FinchMoECLI --model /path/to/Qwen3.8-Flash-Next-125B-finch-4bit-ple4bit \
--prompt "..." --temperature 0.7
Expect roughly 8.5 tok/s prompt processing and ~3 tok/s generation on a 16 GB M4 Mac mini, where the install streams from an external SSD. Smaller machines and the iPhone target run the same tables; memory is dominated by the KV cache and scratch, not by weights.
manifest.json — tensor map, quantization slots, architecturemodel_weights.bin (3.6 GB) — embeddings, attention, hyper-connections, normspacked_experts/ (63 GB) — routed and shared experts, int4/int8ple_shards/ (30 GB) — the PLE n-gram table, int4 affine group 32tokenizer/, verified-install.json — tokenizer and the install receipt
(per-file SHA-256, used to skip re-hashing on load)Repacked with FinchMoERepack from a local BF16 safetensors snapshot,
sourceSnapshotHash sha256:99e815241ef03325536b0aaa4441deea45174c17fae31e10f0bb456410c590de,
on 2026-09-15. The source snapshot is unmodified apart from the repack and
quantization described above.
The base model is licensed under the Qwen Community License 1.0, which is
included here as LICENSE and permits publishing, distributing and creating
derivative works subject to its conditions. This repository is a derivative
work: the modifications are the repack into the .finch format and the
quantization described above. The copyright and permission notice is retained
in full.