Downloads · 30 days
9
5% of all-time downloads
robbiemu/leanstral-1.5-119b-a6b-mxfp4-w4a8
leanstral-1.5-119b-a6b-mxfp4-w4a8 is a text generation model from robbiemu. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Experimental runtime-enablement release. The calibration profile is locked in and backed by a clean replicated study. But a genuinely fresh quality gate on a clean runtime missed its predeclared Top-1 and KL-tail thre…
Downloads · 30 days
9
5% of all-time downloads
All-time downloads
176
Public
Repo size
65.7 GB
Likes
0
Public
Click a slice to open those files.
.safetensors65.7 GB · 100%
From the Hugging Face model README
Experimental runtime-enablement release. The calibration profile is locked in and backed by a clean replicated study. But a genuinely fresh quality gate on a clean runtime missed its predeclared Top-1 and KL-tail thresholds, so I'm not calling this a production-ready replacement. Current complete SGLang serving doesn't demonstrate an end-to-end advantage over W4A16 either—the measured gain is confined to the native prefill expert GEMMs.
This checkpoint packs MXFP4 weights together with a frozen, corpus-calibrated static FP8 activation profile. It's built for Blackwell's native mixed FP8 × FP4 tensor path and keeps the activation contract fully explicit rather than implicit.
source: official Leanstral 1.5 FP8 E4M3 checkpoint
weights: packed MXFP4 E2M1
weight scales: UE8M0, one scale per K/32 block
activation profile: static per-tensor FP8 E4M3, frozen Variant 1
native GEMM input: normalized E4M3 values with unit UE8M0 K/32 scales
post-GEMM: reapply the calibrated per-tensor activation scale
accumulation: FP32 in the mixed GEMM
output: BF16
This checkpoint uses the frozen Variant 1 static activation profile; it does not choose activation scales dynamically at runtime. The project also includes a dynamic MXFP8 implementation, but only as a reference for testing and comparison.
Every evaluation in this card used the same Variant 1 profile, recorded under SHA-256 5480257595178836022477080bc6132e2accff2d6e26c959b092e57a0f911531. Before publishing it, I replaced one local filesystem path with the public Variant 1 dataset reference. None of the 9,432 activation scales changed. PUBLICATION_MANIFEST.json records the original and published file hashes, plus a separate hash of the scale data, so that metadata-only change can be verified.
Approximately 61.2 GiB of checkpoint storage and 66 GiB of CUDA-visible memory were consumed in the validated 8,192-token configuration. We recommend at least 96 GB of GPU or coherent unified memory; 128 GB unified memory was tested.
We also evaluated
kamiyugi/Leanstral-1.5-119B-A6B-NVFP4A16,
a well-documented community W4A16 release, at revision
85048640a8998ed424a0abba9a4d928252f5c3dc.
The two checkpoints make different choices. That release quantizes the routed experts to NVFP4 while retaining attention and shared experts at higher precision, and it has a mature vLLM/Marlin serving path. Our release quantizes more language matrices and uses MXFP4 weights with static FP8 activations, making it smaller but dependent on an experimental SGLang integration.
| This MXFP4 W4A8 release | Community NVFP4A16 release | |
|---|---|---|
| Seven-shard size | 61.16 GiB | 67.16 GiB |
| Primary Top-1 agreement with our dynamic anchor | 56.67% | 51.67% |
| Primary Top-5 overlap | 53.50% | 42.67% |
| Primary token-weighted NLL | 13.68898 | 13.52535 |
| Current single-request decode | 10.24 tok/s | 23.24 tok/s |
These are our measurements on one GB10, not claims made by the other checkpoint’s author. The serving comparison is also cross-runtime: each model used the runtime that currently supports its contract. It therefore measures practical deployment today rather than isolating W4A8 versus W4A16 arithmetic.
The results suggest different use cases. The NVFP4A16 release is the stronger choice for users who prioritize working, high-performance serving today. This MXFP4 W4A8 release is smaller and preserves ordinary next-token decisions more closely in our validation, but remains an experimental artifact for developing a GPU-resident mixed FP8×FP4 runtime.
See docs/hf-quantization-postmortem.md
for the complete methodology, quality distribution, serving workloads, and
claim boundaries.
NVIDIA ModelOpt's W4A8_MXFP4_FP8 preset describes static per-tensor FP8 activation quantization. A dynamic MXFP8 kernel works differently: it generates E4M3 values and UE8M0 scales for every K/32 block at runtime. Both end up using mixed FP8 × FP4 arithmetic, but their scale semantics and serialized metadata aren't interchangeable. Any runtime supporting this checkpoint needs to treat it as the static per-tensor contract, not the dynamic one.
Blackwell's accelerated mixed FP8 × FP4 path follows the OCP MX block-scale contract:
MXFP4 weights:
FP4 E2M1 values
UE8M0 scale per K/32 block
MXFP8-form GEMM activations:
FP8 E4M3 values
UE8M0 scale per K/32 block
So MXFP4 matches the native mixed-MMA weight representation directly. NVIDIA ModelOpt emits canonical packed MXFP4 values and row-major UE8M0 scales, and the Blackwell runtime handles the required load-time scale-layout transformation.
NVFP4 still has its place for native W4A4 and weight-only W4A16 execution. But its fractional E4M3/K16 weight-scale contract doesn't line up with the mixed MX block-scaled MMA used here. An earlier prototype reconstructed those scales correctly, just slowly. A clean pinned-runtime rerun confirmed the picture: SGLang still rejects the exact NVFP4-weight/static-FP8 signature, all five retained scalar arithmetic controls check out, and W4A16 wins all 15 matched benchmark rows on the deterministic real-shape fixture. So this specific NVFP4 W4A8 contract is out—that doesn't say anything against NVFP4 W4A4 or NVFP4 W4A16.
The format choice stays inside NVIDIA's quantization ecosystem either way: ModelOpt supports MXFP4 conversion and export directly. MXFP4's standardized representation might help other runtimes too, but the main reason for picking it here is Blackwell's native mixed-precision contract.
model: mistralai/Leanstral-1.5-119B-A6B
revision: 3fe4e64d70e6873420fe19108a3a61ec6a9f1460
format: released FP8 E4M3 weights and published scales
tool: NVIDIA ModelOpt 0.37.0 MXFP4QTensor
note: verified the results are bitwise identical at modopt 0.45
The MXFP4 weights come from requantizing the official released FP8 checkpoint directly. Expanding those tensors back to BF16 would change how they execute, but it wouldn't recover any higher-precision source model that isn't there.
The streaming conversion produced seven consolidated-* shards and converted 14,112 eligible language weights: 180 attention matrices, 13,824 routed-expert matrices, and 108 shared-expert matrices. The old FP8 weight and activation scale tensors were removed and replaced with MXFP4 UE8M0/K32 weight scales.
The model repository includes the conversion manifest, release manifest, validation receipt, and every shard hash.
The official source revision ships the Mistral consolidated layout with params.json and tekken.json, but no tokenizer_config.json or generation_config.json. This derivative keeps that same layout instead of inventing Hugging Face defaults for the missing files. Use the Mistral config parser and loader shown below, and set your sampling parameters explicitly in the client.
I built five 512-record calibration candidates, calibrated each one as a complete static profile, then evaluated the frozen artifacts against the same 100-example iterative-development pack. Once cross-restart variability turned out to trace back to normal FlashInfer autotune choices, I reran the whole comparison across five independent, normally autotuned cold starts. Every profile used the same dynamic reference within each start.
We started with 4 variants then tried to build a composite from the datasets to best reflect their strenghts, variant 5.
| Profile | Candidate NLL | Mean absolute sample NLL delta | Top-1 agreement |
|---|---|---|---|
| V1 | 13.93530 ± 0.00145 | 0.08660 ± 0.00144 | 0.620 ± 0.016 |
| V2 | 13.96571 ± 0.00128 | 0.12148 ± 0.00181 | 0.578 ± 0.013 |
| V3 | 13.96561 ± 0.00129 | 0.09765 ± 0.00085 | 0.550 ± 0.029 |
| V4 | 14.02996 ± 0.00105 | 0.13983 ± 0.00045 | 0.550 ± 0.012 |
| V5 | 13.97669 ± 0.00120 | 0.09480 ± 0.00038 | 0.574 ± 0.009 |
V1 came out ahead on candidate NLL, signed NLL delta, mean absolute sample NLL delta, top-1 agreement, and flip count—in all five starts.
V5 consistently improved the severe KL tail: it won mean and maximum KL in all five starts, and fixed all ten recurring V1 tail failures in at least four of them. But V5 also lost V1's central NLL/top-1 behavior in every single start and brought recurring failures of its own.
So I tried a constrained Variant 6: keep 464 V1 records, swap in 48 D/E short-or-medium records from V5, and recalibrate the whole thing as one coherent profile. Across five new starts, V6 improved mean KL every time—but its NLL was worse than V1's every time, and its top-1 agreement was lower every time too. The V6-minus-V1 mean NLL came out to +0.028728, past the frozen +0.01 rejection margin, and the top-1 difference was -0.042, past the frozen -0.01 margin. Dropping the first start doesn't change that outcome. V6 got rejected; V1 stayed.
V1 is the profile selected for central preservation under this release's frozen priorities. V5 is the demonstrated alternative for the severe tail. The constrained attempt just didn't manage to keep both advantages at once.
Under normal FlashInfer startup variation, V1 consistently gives the best ordinary predictive preservation. The KL-tail improvements on the table are real, but neither V5 nor a targeted V1-centered composite gets there without an unacceptable hit to central quality.
That's where simple record-substitution composite development stops, for now. Reopening it would need a different mechanism entirely: layer-specific calibration, constrained scale optimization, or a different activation-quantization scheme altogether.
Full methods, per-start distributions, recurring failures, routing coverage, fallback exposure, and hashes all live in the model repository, alongside:
docs/activation-profile-selection.mddocs/replicated-autotuned-profile-study.mddocs/constrained-v1-composite-result.mdLow-level canaries established that:
These canaries validate the tensor and arithmetic contract—nothing more. On their own, they don't establish full-model quality or end-to-end serving speed.
The full 36-layer text model also loaded through the pinned experimental SGLang/FlashInfer path and returned finite outputs. That said, release validation covered text generation only. Vision/projector tensors are preserved from the official source artifact, but multimodal inference wasn't validated, and this release doesn't claim to support it.
After freezing V1, I built a fresh 144-example pack spanning 138 source files, with zero overlap against V1–V6, the 100-example development pack, or the earlier quality pack—checked across identifier, source, fingerprint, and rendered text. Pack SHA-256: 029680debf51f64815dab3582acaa0d756e54977bfdf2214a8232860a63bcbec.
The frozen primary gate compared V1 static FP8 with the within-start dynamic MXFP8 reference on the pinned clean runtime. The table covers the pack's 120-example primary stratum:
| Metric | Result | Frozen criterion | Status |
|---|---|---|---|
| Absolute token-weighted NLL delta | 0.04455 | ≤0.10 | Pass |
| Mean absolute sample NLL delta | 0.07578 | ≤0.15 | Pass |
| Top-1 agreement | 59.17% (71/120) | ≥60% | Fail |
| Top-5 overlap | 0.5383 | ≥0.50 | Pass |
| Dynamic Top-1 retained in static Top-5 | 0.7583 | ≥0.70 | Pass |
| Median partition KL | 0.03895 | ≤0.10 | Pass |
| p95 partition KL | 3.49075 | ≤1.00 | Fail |
| Examples above KL 5 | 6 | ≤2 | Fail |
| Minimum category Top-1 | 0.48 | ≥0.40 | Pass |
The separate 24-example context-length stress stratum passed all six of its frozen checks. Expert coverage across the full pack hit 4,560 of 4,608 layer/expert pairs, and only 19 of 26,308,512 routed assignments needed fallback scales. Even so, the primary gate failed—and I'm disclosing that result here rather than quietly kicking off another calibration-design round to fix it.
If you care about SGLang performance today, use W4A16. This experimental W4A8 path is not faster overall yet.
The underlying W4A8 expert GEMMs do show promise. In an instrumented local SGLang implementation, they ran 1.43–1.89× faster than W4A16 Marlin for prefill sizes of 128, 512, and 2048 tokens. At decode-sized workloads, the two were effectively tied.
But the complete routed-MoE operation was slower. SGLang moved routing information to the CPU with counts.tolist() and gathered the active expert weights with weight[active_experts] at every layer. That overhead cost more than the faster GEMMs saved.
Those timings came from an earlier instrumented implementation, not current upstream SGLang. The later clean port revalidated the arithmetic and loaded the full model successfully. Its Leanstral adapter still contains the same CPU-visible routing and expert-gathering boundaries, though, so the practical conclusion has not changed:
The likely fix is to keep the packed expert weights and routing data on the GPU and run them through a fused routed-MoE backend. SGLang is already working on this architecture for dynamic MXFP8 activations, but this checkpoint needs the same path to support its serialized static FP8 profile.
That remaining integration work is tracked in SGLang #31235. Until it lands and is benchmarked, W4A16 remains the better choice for current SGLang serving.
See docs/performance-potential-and-sglang-limitations.md for the complete benchmark table, timing breakdown, and supporting runtime details.
Stock SGLang doesn't load this serialized static-FP8 profile through the resident FlashInfer path yet. The validated text canary used the pinned project patch with exactly FlashInfer 0.6.14. With that patch applied, you need to supply the selected profile explicitly:
export SGLANG_MXFP4_FP8_PROFILE=/path/to/model/mxfp4_fp8_activation_profile.json
sglang serve /path/to/model \
--model-config-parser mistral \
--load-format mistral \
--quantization mxfp4 \
--moe-runner-backend flashinfer_mxfp4 \
--attention-backend flashinfer \
--dtype bfloat16 \
--tp-size 1
The loader will fail outright on a missing profile entry or a non-finite/non-positive calibrated scale. The repository records the exact SGLang base, patch digest, container digest, and FlashInfer Python/cubin/JIT versions used for validation.
Completed:
The frozen quality failure means I can't call this production-ready. What this is: an experimental MXFP4 W4A8 runtime-enablement artifact with a strongly justified calibration profile, a disclosed failed quality gate, demonstrated native prefill-compute potential, and a known SGLang orchestration limitation. It's not a proven faster or production-quality replacement.
The 100-example pack got reused to diagnose profiles and design composites, so its results support engineering selection—they're not an untouched estimate of final generalization. The separate 144-example pack was frozen after profile selection and used exactly once for the release-quality gate. I'm reporting it as the final validation result for this artifact, and it wasn't used to redesign V1.
Other limitations: