gpt-oss-20B-NVFP4-GGUF

GPT-OSS 20B — NVFP4 (Expert-Selective, Thinking Opt-In)

Repository: FreedomAISVR/gpt-oss-20B-NVFP4-GGUF
Source model: openai/gpt-oss-20b
Quantization: NVFP4 experts + Q8_0 non-experts (Blackwell-optimized)

Model Details

GPT-OSS is a 20B-parameter mixture-of-experts (MoE) language model developed by OpenAI, with 2.8B active parameters per token. It uses a 128-expert MoE layer (top-2 routing) with a 28-layer transformer architecture. This is an NVFP4 + Q8_0 hybrid — MoE expert weights are NVFP4, all other tensors are Q8_0.

Architecture

Recommended Inference Configuration

{
  "temperature": 0.7,
  "top_p": 0.9,
  "max_tokens": 32768
}

Quantization Details

This repository uses a hybrid quantization approach:

• NVFP4 quantized: MoE expert weights (72 tensors: ffn_gate_exps, ffn_up_exps, ffn_down_exps — 3 per block × 24 blocks, each 142 MiB)
• Q8_0 quantized: All non-expert tensors — attention projections, router, embeddings, layer norms, LM head, biases (387 tensors)

The NVFP4 expert weights benefit from Blackwell GPU hardware acceleration for 4-bit matrix multiplication. Q8_0 for non-expert tensors provides a good balance between quality and size. OpenAI's GPT-OSS was post-trained with MXFP4 quantization baked into expert weights, so these are requantized from MXFP4 to NVFP4.

File Details

Performance

On NVIDIA Blackwell GPUs (RTX 5060 Ti and higher), the NVFP4 expert weights benefit from hardware-accelerated 4-bit matrix multiplication, while non-expert tensors run at Q8_0 throughput.

Chat Template

The chat template uses opt-in reasoning — reasoning_effort is only applied when explicitly set by the user. This matches the original OpenAI model behavior where no "Reasoning:" instruction is injected into the system prompt.

// No reasoning instruction by default
// Set for explicit control:
reasoning_effort: "low"    // minimal chain-of-thought
reasoning_effort: "medium" // balanced reasoning
reasoning_effort: "high"   // thorough reasoning

Compatibility

This GGUF file is compatible with:

• llama.cpp (commit b93186b or later)
• LM Studio (0.3.10 or later)
• Ollama, text-generation-webui, and other GGUF-compatible inference engines

Blackwell GPU recommended for NVFP4 hardware acceleration.

License

Apache 2.0 (same as the original OpenAI GPT-OSS model)