Qwen3.8-27B-MXFP8-CRACK-NInfer-v3

Qwen3.8-27B "MXFP8-CRACK", NInfer v3 artifact

A NInfer v3 artifact of the community fine-tune dealignai/Qwen3.8-27B-MXFP8-CRACK. It runs on NInfer-all, the master branch of iamwavecut/ninfer-all, which serves the RTX 3090, RTX 4090, RTX 5090 and RTX PRO 6000 Blackwell. Its first (v2) release targeted stock NInfer, which builds only for the RTX 5090; this v3 file was verified on NInfer-all.

Download & run

hf download WaveCut/Qwen3.8-27B-MXFP8-CRACK-NInfer-v3 Qwen3.8-27B-MXFP8-CRACK-ninfer-v3.ninfer --local-dir models

# OpenAI/Anthropic-compatible server from the Docker image, MTP speculation, on http://localhost:8080/v1:
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all serve /models/Qwen3.8-27B-MXFP8-CRACK-ninfer-v3.ninfer \
  --max-context 32768 --kv-capacity auto --spec mtp --draft-tokens 3 --lm-head-draft

# From a build of NInfer-all, CLI, one request:
./build/apps/ninfer models/Qwen3.8-27B-MXFP8-CRACK-ninfer-v3.ninfer \
  --prompt "Summarize the difference between prefill and decode." --max-context 16384

# OpenAI/Anthropic-compatible server, MTP speculation:
./build/apps/ninfer-serve models/Qwen3.8-27B-MXFP8-CRACK-ninfer-v3.ninfer --max-context 32768 \
  --kv-capacity auto --spec mtp --draft-tokens 3 --lm-head-draft

The artifact carries no DFlash2 adapter. The Docker image needs an NVIDIA driver of the CUDA 13 branch (580 or newer) and the NVIDIA Container Toolkit; a build (its README) sets CMAKE_CUDA_ARCHITECTURES to 86, 89 or 120a. On 24 GB cards the official artifact of the same identity and size fits with a long rk4v4 or rk8v4 KV cache.

How it was built

The source checkpoint stores weights in two packed 8-bit schemes, so it was first dequantized to BF16 and renamed into the official Qwen3.8-27B tensor layout, then converted with the unmodified stock converter:

1. Text + Vision (582 tensors), MXFP8: 4×FP8-E4M3 packed per U32 word, one U8 UE8M0 scale (2^(b-127)) per 32-element group → BF16.
2. MTP draft layer (8 tensors), MLX affine int8: uint8 codes with per-128-group F16 scale and bias, w = q*scale + bias → BF16. The formula was verified against the official Qwen/Qwen3.8-27B MTP weights (cosine similarity ≈ 1.0, σ matches to 5 decimals).
3. Renames: language_model.model.* → model.language_model.*, language_model.lm_head.* → lm_head.*, vision_tower.* → model.visual.*; the vision patch embedding was permuted from the MLX conv layout [O,T,H,W,I] to torch [O,I,T,H,W]. config.json of the fork passes the converter's registered validation unchanged.
4. Frontend resources (tokenizer, preprocessor configs, generation_config.json) are the official Qwen/Qwen3.8-27B files, pinned by SHA-256 in the converter.
5. Converter: python3 -m tools.convert.qwen3_8_27b.convert --model <bf16-dir> --out crack.ninfer wrote the v2 artifact of the first release; its report is reports/v2/crack.ninfer.conversion.json.
6. v3: python3 tools/upgrade_ninfer_v2_to_v3.py crack.ninfer Qwen3.8-27B-MXFP8-CRACK-ninfer-v3.ninfer rewrote the container. Every weight byte is unchanged; the maintained Qwen3.8 chat template is installed.

Caveats

• Double quantization. The source is already 8-bit (MXFP8 / affine int8); this artifact re-quantizes it into NInfer's groupwise-int profile (Q4/Q5/Q6/W8 groupwise + BF16). Expect marginally lower fidelity than an artifact built from a native BF16 checkpoint of the same tune.
• The engine binds this file to the registered qwen3.8-27b / groupwise-int profile: architecture, tokenizer and chat template are the official ones; only the weights carry the fine-tune.
• Per the upstream card: set your sampler explicitly (temperature=1.0, top_p=0.95, top_k=20) if you rely on the tune's intended behavior.

Lineage

Qwen/Qwen3.8-27B
  └─ dealignai/Qwen3.8-27B-MXFP8-CRACK   (fine-tune + MXFP8/MLX-affine quant, Apache-2.0)
       └─ BF16 dequant + official-layout rename (adapter described above)
            └─ crack.ninfer (v2)  →  Qwen3.8-27B-MXFP8-CRACK-ninfer-v3.ninfer (this repo)

Licensed Apache-2.0, same as both upstreams.