Qwen3.8-27B-AutoQuant-NVFP4-FP8-5bit

Qwen3.8-27B NVFP4/FP8 5-bit

Mixed-precision NVIDIA ModelOpt quantization of Qwen/Qwen3.8-27B, based on upstream revision 1d4bf0f.

This checkpoint targets a strong size-to-quality balance by assigning compatible language projections to NVFP4 or FP8 according to measured sensitivity. No fine-tuning or additional training was performed.

Quantization

Quantization used NVIDIA ModelOpt 0.46.0. The 5.0-bit target applies to the compatible quantized language weights, not the BF16 components or runtime KV cache. Exact module assignments are recorded in hf_quant_config.json and config.json.

The complete repository occupies approximately 20.39 GB (18.99 GiB).

Calibration

Calibration used 1,024 fully populated 512-token samples from the deterministic version-6 corpus (seed 42). Its token distribution was:

Quality Benchmarks

The first row contains author-reported results for this checkpoint. The comparison rows use the benchmark table published for acyildirimer/Qwen3.8-27B-NVFP4.

Thinking was disabled during these quality benchmark runs to keep the evaluation conditions consistent and avoid mixing reasoning-mode differences into the checkpoint comparison.

4-task mean is the arithmetic mean of MMLU, ARC-Challenge, HellaSwag, and GSM8K. HumanEval is reported separately and is not included in that aggregate. The first-row MMLU and HellaSwag results use their complete evaluation sets: MMLU is 5-shot accuracy over 14,042 questions (11,856 correct), and HellaSwag is 10-shot accuracy over 10,042 questions (9,447 correct). Both were evaluated through vLLM chat completions with thinking disabled, temperature 0, and strict generated-letter scoring. These are generative accuracy results, not log-likelihood leaderboard scores.

The first-row results are author-reported. The remaining comparison results originate from the public witcheer/rtx-5090-benchmarks dataset. The serving stacks differ, so the comparison is indicative rather than a controlled quantization ablation.

Tool-Calling Evaluation

Evaluated with tool-eval-bench v2.2.0 through vLLM at temperature 0, seed 42, across all 69 scenarios:

One failure involved prompt-injection resistance. Tool-enabled deployments should enforce independent authorization and defenses for untrusted tool, file, and web content.

vLLM Usage

This checkpoint was designed for headless deployment on a dual-GPU system with 32 GB of total VRAM, while retaining the full native context window and vision support. Both memory-conscious configurations below are set to serve one request at a time.

A vLLM runtime with ModelOpt mixed NVFP4/FP8 support is required. This checkpoint was smoke-tested with vLLM 0.28.0.

VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
OMP_NUM_THREADS=1 \
CUDA_DEVICE_ORDER=PCI_BUS_ID \
CUDA_VISIBLE_DEVICES=0,1 \
NCCL_P2P_DISABLE=1 \
vllm serve acyildirimer/Qwen3.8-27B-AutoQuant-NVFP4-FP8-5bit \
    --host 0.0.0.0 \
    --port 8000 \
    --quantization modelopt \
    --tensor-parallel-size 2 \
    --safetensors-load-strategy lazy \
    --kv-cache-dtype fp8 \
    --max-model-len auto \
    --disable-custom-all-reduce \
    --gpu-memory-utilization 0.97 \
    --max-num-seqs 1 \
    --max-num-batched-tokens 2048 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --performance-mode interactivity \
    --attention-backend auto \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --generation-config auto \
    --override-generation-config \
      '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \
    --tool-call-parser qwen3_coder \
    --default-chat-template-kwargs \
      '{"preserve_thinking":true,"reasoning_effort":"xhigh"}'

Vision support is enabled by leaving out --language-model-only. This example also disables NCCL P2P communication and vLLM's custom all-reduce path for a dual-GPU system without reliable peer-to-peer connectivity.

The SGLang configuration below provides an equivalent multimodal setup.

SGLang Usage

The equivalent SGLang configuration is:

CUDA_DEVICE_ORDER=PCI_BUS_ID \
CUDA_VISIBLE_DEVICES=0,1 \
NCCL_P2P_DISABLE=1 \
OMP_NUM_THREADS=1 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
sglang serve \
    --trust-remote-code \
    --model-path acyildirimer/Qwen3.8-27B-AutoQuant-NVFP4-FP8-5bit \
    --tp 2 \
    --disable-custom-all-reduce \
    --enable-multimodal \
    --mem-fraction-static 0.93 \
    --attention-backend flashinfer \
    --kv-cache-dtype fp8_e4m3 \
    --chunked-prefill-size 2048 \
    --max-running-requests 1 \
    --cuda-graph-bs-decode 1 \
    --disable-prefill-cuda-graph \
    --mamba-radix-cache-strategy extra_buffer_lazy \
    --max-mamba-cache-size 4 \
    --mamba-ssm-dtype float32 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --default-chat-template-kwargs \
      '{"enable_thinking":true,"preserve_thinking":true,"reasoning_effort":"xhigh"}' \
    --host 0.0.0.0 \
    --port 8000

This configuration intentionally disables NCCL P2P communication and SGLang's custom all-reduce path for dual-GPU systems without reliable peer-to-peer connectivity.

Limitations

This is a lossy post-training quantization. Results can vary with the evaluation protocol, runtime, GPU architecture, context length, and workload. The model also inherits the capabilities and limitations of the upstream checkpoint.

Refer to the Qwen3.8-27B model card for the authoritative architecture description, generation settings, vision preprocessing, and upstream limitations.

Acknowledgements

This checkpoint retains the upstream Apache 2.0 license and was quantized with NVIDIA ModelOpt.