Downloads · 30 days
61
22% of all-time downloads
kvnxiao/swift-qwen3.8-27b-orcarouter-dflash2-nvfp4-ninfer
swift-qwen3.8-27b-orcarouter-dflash2-nvfp4-ninfer is a image-text-to-text model from kvnxiao. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for ninfer. The card lists the license as other.
This is an NInfer v3 conversion of ajgazin's Swift 1.0 NVFP4 checkpoint. One .ninfer file has the text and vision model, built-in MTP head, separate DFlash2 draft model, and indexed proposal head. Ajgazin applied the…
Downloads · 30 days
61
22% of all-time downloads
All-time downloads
276
Public
Repo size
22.8 GB
Likes
3
Public
Click a slice to open those files.
.ninfer22.8 GB · 100%
From the Hugging Face model README
This is an NInfer v3 conversion of ajgazin's Swift 1.0 NVFP4 checkpoint. One .ninfer file has the text and vision model, built-in MTP head, separate DFlash2 draft model, and indexed proposal head. Ajgazin applied the OrcaRouter refusal direction to UkisAI's Swift 1.0 weights before quantization. This conversion adds no training or further abliteration.
| Model at a glance | Details |
|---|---|
| Artifact | swift-qwen3.8-27b-orcarouter-dflash2-nvfp4.ninfer · 22,783,240,452 bytes (22.78 GB) |
| Components | Text, vision, MTP, DFlash2, and proposal head |
| Tested setup | RTX 5090; 32,768-token context and KV capacity; FP8 KV cache; one concurrent request |
| Smoke tests | 15/15 passed across speculation off, MTP, and DFlash2 |
| Custom MMLU-Pro subset | 229/280 correct in DFlash2 mode; protocol and detailed results |
| License | Swift Open License v1.0, with Qwen and DFlash2 Apache 2.0 notices retained |
The source chain is Qwen3.8-27B → UkisAI Swift 1.0 → ajgazin's OrcaRouter derivative → ModelOpt NVFP4/FP8 checkpoint → this NInfer artifact. Ajgazin's source card documents the refusal-direction edit and calibration. The pinned inputs and recipe checksum make this conversion reproducible.
The unchanged CaptainArni recipe imports the source's calibrated NVFP4 and FP8 weights. Conversion changes storage layouts and quantizes selected components without another calibration pass:
| Component | Representation in this artifact |
|---|---|
| Text MLPs | Import NVFP4 codes and block scales for all 64 layers; store reciprocal FP32 global scales. |
| Attention and GDN | Import FP8 E4M3 codes with BF16 row scales; retain GDN a/b projections and direct tensors as BF16. |
| Embeddings and output head | Encode as row-scaled FP8. The source NVFP4 output head is decoded before re-encoding. |
| Vision | Use Q4/Q5 projections, Q6 patch embedding, Q8 merger, and BF16 direct tensors. |
| MTP and DFlash2 | Quantize draft projections to Q8; retain their direct tensors as BF16. The DFlash2 checkpoint comes from z-lab's mirror of Inco AI's release. |
| Proposal head | Encode a 131,072-token shortlist as Q4 for speculative token proposals. |
| Frontend | Preserve the source tokenizer, chat template, generation settings, and vision resources byte for byte. |
The source output head is NVFP4. NInfer's output-head kernel uses row-scaled FP8, so the converter decodes and re-encodes it. Sampled output-head rows have 1.86263752%–2.96464051% relative L2 reconstruction error. This measures changed tensor values, not changed answer quality.
NVFP4 MLP weights permit NInfer's A4 activation path, but the whole transformer layer is not four-bit. The source's chat template is embedded without an override. MTP uses the built-in draft component; DFlash2 uses the separate companion. The two speculation modes are alternatives at inference time. The proposal head supplies shortlist token proposals for MTP.
Reader tests cover NVFP4 nibble order, block scales, FP8 row scaling, partial reads, and malformed or non-finite inputs. Full checkpoint preflight checked 2,194 base tensors and 81 DFlash2 tensors. Across 63 sampled rows near the beginning, middle, and end of selected matrices, imported codes and block scales matched the source. Artifact inspection checked geometry, component bindings, the proposal head, and byte-identical frontend resources. RESULTS.md has the artifact checksum and sample error ranges. These checks do not establish exact numerical equivalence to the source runtime.
The GPU smoke suite ran text, multi-turn chat, thinking, a structured weather tool call, and image input in each speculation mode. All 15 checks passed at 32K context. The benchmark used the same runtime settings with temperature zero, seed 42, disabled prefix reuse, and one warm-up per mode.
The benchmark has three workloads:
Answer: X line; wrong, unparsed, truncated, and failed answers count as incorrect.| Mode | Peak total VRAM in smoke test | Prose decode tokens/s | Code decode tokens/s | Reasoning decode tokens/s |
|---|---|---|---|---|
| Speculation off | 23,659 MiB | 79.96 | 79.87 | 79.85 |
| MTP, 3 draft tokens | 24,504 MiB | 143.25 | 210.86 | 210.33 |
| DFlash2, 7 draft tokens | 25,944 MiB | 140.91 | 245.00 | 280.26 |
Decode rates are server-reported output-token intervals, shown as medians across the three repeats. All generation runs reached the cap, so these figures measure bounded decoding throughput, not answer quality or end-to-end request speed. VRAM includes desktop use and other processes. RESULTS.md also reports prompt processing, request durations, draft acceptance, and individual smoke-test rates.
The quality result was 229/280 correct (81.79%), with 1 unparsed and 0 truncated responses. This is a custom balanced subset and prompting protocol, not the standard MMLU-Pro score. Quality was not compared across speculation modes.
The release includes benchmark.json, benchmark-inputs.json, gpu-validation.json, and conversion reports. The reports have the exact prompts, selected questions, responses, timing measurements, and GPU memory samples. The checks do not establish source-runtime parity, preserved refusal behavior, or long-context answer quality.
The artifact is intended for local text generation, tool-call generation, and image understanding through NInfer. Generated answers can be incorrect or harmful. This project has not evaluated refusal behavior, safety, bias, or multilingual quality; assess those properties for your application.
Use Linux with uv, hf, git, just, and rootless Podman with NVIDIA CDI. Download the repository files, build the pinned runtime, and fetch the model:
hf download kvnxiao/swift-qwen3.8-27b-orcarouter-dflash2-nvfp4-ninfer --exclude '*.ninfer' --local-dir swift-ninfer
cd swift-ninfer
just setup
just runtime-build
hf download kvnxiao/swift-qwen3.8-27b-orcarouter-dflash2-nvfp4-ninfer swift-qwen3.8-27b-orcarouter-dflash2-nvfp4.ninfer --local-dir models
Check available VRAM before starting NInfer. The tested configuration uses 32K context and FP8 KV cache:
nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv,noheader
podman run --rm --device nvidia.com/gpu=all --security-opt=label=disable \
-p 127.0.0.1:8080:8080 -v "$PWD/models:/models:ro" \
localhost/swift-orcarouter-ninfer:9e163eee4b8a-cuda13.1.2 \
ninfer-serve /models/swift-qwen3.8-27b-orcarouter-dflash2-nvfp4.ninfer \
--host 0.0.0.0 --port 8080 --max-context 32768 --kv-capacity 32768 \
--kv-dtype fp8 --max-concurrency 1 --vision \
--spec mtp --draft-tokens 3 --lm-head-draft
Omit the speculation flags for ordinary decoding, or use --spec dflash2 --draft-tokens 7 for DFlash2. Requests use model ID swift-qwen3.8-27b-orcarouter.
The project scripts read pinned revisions, component settings, and the artifact name from sources.json. Convert on CPU, then build the runtime:
just setup
just download
just check
just preflight
just convert
just inspect
just runtime-build
just download verifies the Hub files. Preflight needs at least 8 GiB available RAM and 30 GiB free disk after download. just convert refuses to overwrite an existing artifact or conversion report. Verify the output checksum with (cd output && sha256sum --check SHA256SUMS).
Get the computer user's approval before allocating GPU memory. Check that at least 28 GiB is free before each NInfer run. Prepare benchmark inputs without GPU use, then run validation and benchmarks separately:
just benchmark-prepare
nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv,noheader
just gpu-validate --allow-gpu
nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv,noheader
just benchmark --allow-gpu
just results
just hf-stage
just hf-stage prepares output/huggingface/ after GPU validation passes for the artifact checksum. PUBLISHING.md describes the release files and upload command.
UkisAI created Swift 1.0 from Qwen3.8-27B. Ajgazin applied the OrcaRouter refusal direction and quantized the source checkpoint. CaptainArni supplied the conversion recipe; Inco AI supplied DFlash2; Neroued's NInfer supplies the format and runtime. This is an independent conversion, not an upstream release.
The weights inherit Swift Open License v1.0. Its commercial-use threshold is US$1,000,000 or more in gross revenue over the most recently completed fiscal year, including affiliated entities. Read sections 1, 4, and 5 for the exact terms and redistribution requirements. Qwen and DFlash2 retain Apache 2.0 terms; see LICENSE-APACHE-2.0 and NOTICE. UkisAI's original notice remains intact.