Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL

Qwen3.8-27B Heretic ARA NVFP4 MTP VL

A compressed-tensors NVFP4 W4A4 release of heretic-org/Qwen3.8-27B-heretic-ara with the original BF16 vision/video tower and all 15 native BF16 MTP tensors retained and runtime-validated on one RTX 5090.

Release artifact: this repository contains the converted nvfp4-pack-quantized checkpoint. The raw ModelOpt export was an intermediate artifact. Conversion to compressed-tensors is required for the tested Qwen3.8 multimodal + MTP vLLM path.

Quick start — RTX 5090 / Blackwell

Primary runtime: vLLM's Qwen3.8 CUDA 13 build. The packed NVFP4 weights, multimodal path, and native MTP drafter were validated with the pinned image below. Hugging Face may display an automatic 8-bit badge because two FP4 values are physically packed in each U8 storage element; the quantization is NVFP4 W4A4.

Download

hf download lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL \
  --local-dir ./qwen38-nvfp4

Allow roughly 23 GB for the release plus temporary/cache headroom.

Recommended text-only long-context profile — 192K

This is the practical RTX 5090 profile: FP8 KV, one sequence, chunked prefill, prefix caching, native MTP n=3, FlashInfer/CUTLASS NVFP4 GEMM, and the vision graph disabled to preserve KV capacity.

docker run --rm --gpus all --ipc=host --network=host \
  -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai@sha256:d392f621bb3e372ecc09f0b0cb88099afe9fa05d37a0450de45eeb8c12b6787e \
  --model lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL \
  --served-model-name qwen38-nvfp4-mtp \
  --host 0.0.0.0 --port 8000 \
  --language-model-only \
  --max-model-len 196608 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Aggressive text-only profile — 256K

Use the same command with:

--max-model-len 262144
--gpu-memory-utilization 0.96
--max-num-batched-tokens 8192

This is a single-concurrency capacity target. Add --enforce-eager when CUDA Graph capture or workspace overhead prevents startup; eager mode trades some performance for lower graph-memory pressure. Start with 192K for a stable operational profile.

Full image/video profile

The vision tower and encoder cache need additional VRAM. This pinned profile is the verified smoke-test configuration:

docker run --rm --gpus all --ipc=host --network=host \
  -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai@sha256:d392f621bb3e372ecc09f0b0cb88099afe9fa05d37a0450de45eeb8c12b6787e \
  --model lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL \
  --served-model-name qwen38-nvfp4-mtp-vl \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 4096 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 1024 \
  --enable-prefix-caching \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

For multimodal capacity tuning, 32K is the conservative starting point, 64K–96K is the practical target range, and 128K is aggressive on 32 GB. Validate the chosen image/video limits and encoder-cache budget on your runtime before production.

OpenAI-compatible text request

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"qwen38-nvfp4-mtp",
    "messages":[{"role":"user","content":"Explain speculative decoding briefly."}],
    "max_tokens":256,
    "chat_template_kwargs":{"enable_thinking":false}
  }'

Image request

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="qwen38-nvfp4-mtp-vl",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
            {"type": "text", "text": "Describe this image."},
        ],
    }],
    max_tokens=256,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(r.choices[0].message.content)

RTX 5090 optimization notes

• Keep the GPU power limit at the operator-selected 400 W for sustained runs.
• flashinfer-cutlass is the preferred NVFP4 GEMM backend on SM120; the validated runtime also logged FlashInferCutlassNvFp4LinearKernel.
• FP8 KV is essential for 192K–262K capacity. BF16 KV roughly doubles KV memory.
• max_num_seqs=1 reserves capacity for one very long conversation. Increase concurrency only after lowering context or measuring free KV blocks.
• Chunked prefill controls peak scheduling pressure for long prompts; prefix caching accelerates repeated prefixes.
• expandable_segments:True reduces allocator fragmentation during long-context and multimodal workloads.
• Keep MTP at n=3 as the validated speed/acceptance point. Workload-specific acceptance determines the actual gain.
• The pinned digest is the compatibility contract. A newer vLLM tag should be revalidated for Qwen3.8, compressed-tensors NVFP4, vision/video, MTP, tool parser, and long-context allocation.
• The model card's throughput values come from 4K validation runs. 192K/262K settings are capacity profiles; long-prompt prefill throughput is workload-dependent.

Lineage

Heretic v1.2.0+custom ARA de-censoring of Qwen/Qwen3.8-27B, published by trohrbaugh and mirrored under heretic-org. The two source repositories were checked and their model artifacts are byte-identical; the organization mirror adds base-model metadata.

What is quantized

Checkpoint metadata:

quant_method: compressed-tensors
format: nvfp4-pack-quantized
tensors: 2687
weight files: ~20.56 GB
calibration: 20 samples × 8192 tokens = 163,840 tokens
calibration dataset: abisee/cnn_dailymail, config 3.0.0

Validation

Hardware and runtime:

GPU: NVIDIA GeForce RTX 5090 32 GB (SM120)
GPU power limit: 400 W
Runtime: vllm/vllm-openai:qwen38-x86_64-cu130
Runtime digest: sha256:d392f621bb3e372ecc09f0b0cb88099afe9fa05d37a0450de45eeb8c12b6787e
vLLM build: 0.1.dev19754+g3a0914114
PyTorch: 2.13.0+cu130
KV cache: FP8
MTP: n=3

Verified paths:

• GET /health → HTTP 200
• GET /v1/models → expected served model
• OpenAI-compatible text request → HTTP 200
• Real PNG request → HTTP 200: A red square and a blue circle are displayed on a white background.
• Real MP4 request → HTTP 200: The video shows a red square and a blue circle on a white background.
• Native MTP drafter loaded; embeddings and LM head shared with target model

Measured single-stream results from 1024-token generations:

These are local measurements for the stated prompts/configuration. They are reproducibility evidence, not broad quality benchmarks or a guaranteed speedup over every non-MTP setup.

Compact reference command

docker run --rm --gpus all --ipc=host --network=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:qwen38-x86_64-cu130 \
  --model lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL \
  --served-model-name qwen38-nvfp4-mtp-vl \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --enable-prefix-caching \
  --trust-remote-code \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' 

For short non-thinking responses, pass this at the top level of the request:

{"chat_template_kwargs": {"enable_thinking": false}}

Use the pinned profiles in Quick start for production. The 192K text-only profile is the recommended RTX 5090 long-context setup; 262K is the aggressive single-sequence target. The compact command above remains a readable 32K multimodal example.

Reproduction outline

1. Load the full BF16 VLM with Qwen3_5ForConditionalGeneration using CPU/disk offload.
2. Quantize the language model with ModelOpt NVFP4_DEFAULT_CFG.
3. Calibrate on 20 CNN/DailyMail samples at sequence length 8192.
4. Keep lm_head, token embeddings, linear_attn.conv1d, visual.*, and mtp.* outside NVFP4.
5. Export the ModelOpt HF checkpoint.
6. Retain/graft the 15 MTP tensors from the same upstream source.
7. Convert ModelOpt tensor conventions to compressed-tensors using ModelOptNvfp4Converter:
   • weight → weight_packed
   • input_scale → reciprocal input_global_scale
   • weight_scale_2 → reciprocal weight_global_scale
8. Validate image, MP4, text, and MTP n=3 in the exact Qwen3.8 runtime above.

The converter excludes:

lm_head
model.language_model.embed_tokens
re:.*visual.*
re:.*conv1d.*
re:^mtp.*

See BUILD_MANIFEST.json, VALIDATION_REPORT.json, recipe.yaml, and SHA256SUMS in this repository.

Sources and acknowledgements

• Qwen/Qwen3.8-27B — official architecture, tokenizer, vision/video stack, and trained MTP head
• NVIDIA Model Optimizer — initial NVFP4 W4A4 PTQ/export
• compressed-tensors — ModelOptNvfp4Converter and portable nvfp4-pack-quantized packaging
• vLLM — Qwen3.8 serving, multimodal API, and MTP speculative decoding
• sakamakismile/Qwen3.8-27B-MTP-NVFP4 — working Qwen3.8 compressed-tensors NVFP4+MTP runtime reference
• sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP — earlier ModelOpt NVFP4+MTP recipe reference
• CNN/DailyMail 3.0.0 — calibration text source
• trohrbaugh/Qwen3.8-27B-heretic-ara — original publisher path for this ARA artifact
• Heretic, timrohrbaugh/heretic, and ARA PR #211 — de-censoring method/tool lineage

All upstream behavioral changes belong to their respective source authors. This repository contributes quantization, packaging, and RTX 5090 runtime validation. It does not claim authorship of Qwen3.8, Heretic/ARA, or Blackfrost's model modifications.

Limitations

• NVFP4 requires compatible NVIDIA Blackwell kernels/runtime.
• The tested full multimodal profile uses most of a 32 GB RTX 5090.
• FP8 KV scale warnings may appear when static q/prob scales are absent; the tested runtime uses fallback scale 1.0.
• The calibration set is compact. Task-specific quality should be evaluated against the BF16 source before production use.
• Abliterated/de-censored behavior comes from the upstream model. Operators are responsible for deployment policy and outputs.

License

Apache-2.0, following Qwen3.8 and the cited upstream checkpoint. The included LICENSE is copied from the upstream release.