Mixtral-8x22B-Instruct-v0.1-NVFP4

Mixtral-8x22B-Instruct-v0.1-NVFP4

NVFP4-quantized version of mistralai/Mixtral-8x22B-Instruct-v0.1, produced by Enfuse.

Model Overview

Why NVFP4 Matters for MoE

Mixture-of-Experts models store all expert weights in memory even though only a subset are active per token. Mixtral-8x22B has 141B total parameters but only uses ~39B per forward pass. This means memory -- not compute -- is the primary bottleneck. NVFP4 quantization reduces the memory footprint from ~282 GB to ~70 GB, making this model deployable on significantly fewer GPUs without sacrificing the MoE architecture's efficiency advantages.

How to Use

vLLM (recommended)

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_id = "enfuse/Mixtral-8x22B-Instruct-v0.1-NVFP4"
tokenizer = AutoTokenizer.from_pretrained(model_id)

llm = LLM(model=model_id, tensor_parallel_size=2)
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512)

messages = [
    {"role": "user", "content": "Explain the benefits of mixture-of-experts architectures."},
]

prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
outputs = llm.generate([prompt], sampling_params)
print(outputs[0].outputs[0].text)

Hardware Requirements

• Full NVFP4 (W4A4): Requires NVIDIA Blackwell GPU (B200, GB200, RTX 5090) for native FP4 tensor core support
• Weight-only FP4: Older GPUs (H100, A100) can load the model but will only apply weight quantization
• Recommended: 2x B200 with tensor parallelism

Quantization Details

Quantized using LLM Compressor (v0.10.0):

• Method: Post-training quantization (PTQ) with calibration
• Calibration data: 512 samples from HuggingFaceH4/ultrachat_200k
• Sequence length: 2048 tokens
• Scheme: NVFP4
• Excluded layers: lm_head, MoE gate layers (block_sparse_moe.gate)

Infrastructure

Quantized on an NVIDIA DGX B200 (8x B200, 2 TiB RAM, CUDA 13.0).

Evaluation

OpenLLM v1 Benchmarks

Evaluated using lm-evaluation-harness (v0.4.11) with vLLM backend, --apply_chat_template --fewshot_as_multiturn, tensor_parallel_size=2 on NVIDIA B200 GPUs.

BF16 reference scores from Open LLM Leaderboard v1. Average recovery: ~88%.

About Enfuse

Enfuse builds sovereign AI infrastructure for regulated enterprises. The Enfuse platform provides on-prem LLM orchestration and an App Factory for shipping governed, compliant AI applications on your own infrastructure.

This quantization is part of our ongoing work to make large language models more accessible and efficient for on-premise deployment, where memory efficiency directly impacts what models organizations can run within their own data centers.

Acknowledgments

• Mistral AI for the Mixtral-8x22B-Instruct model
• vLLM Project for LLM Compressor
• NVIDIA for the NVFP4 format and Blackwell hardware