gpt-oss-15.5b-23E-nf4-GPU

gpt-oss-15.5b-23E-nf4-GPU

NF4-quantized variant of xpuenabler/gpt-oss-15.5b-23E-SFT for NVIDIA GPU inference via bitsandbytes.

Quantization recipe

• Base: xpuenabler/gpt-oss-15.5b-23E-SFT (24 layers, 23 experts, 2880 hidden, intermediate=2880, 8 KV heads, 64 head_dim).
• Attention q/k/v/o_proj: NF4 via BitsAndBytesConfig (bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True).
• MoE experts (gate_up_proj, down_proj, originally fused 3D tensors not caught by BNB's auto-replacement): manually quantized to per-expert bnb.nn.Linear4bit (NF4 / fp16 compute) — see modeling_gpt_oss_nf4.py.
• Kept FP16 (per project decision): router mlp.router.weight, embed_tokens, lm_head. Router quantization is known to materially degrade MoE quality.

Storage breakdown (post-quant):

• NF4-packed: 7.19 GB
• FP16: 2.33 GB (router + embed + lm_head)
• safetensors total: ~9.5 GB (down from BF16 ~31 GB)

Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained(
    "xpuenabler/gpt-oss-15.5b-23E-nf4-awq-GPU",
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "xpuenabler/gpt-oss-15.5b-23E-nf4-awq-GPU",
    trust_remote_code=True,    # required — uses GptOssNF4ForCausalLM
    torch_dtype=torch.float16,
    device_map="auto",
)
prompt = "Explain in one short paragraph what makes a Mixture-of-Experts model efficient."
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=120, do_sample=False, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Evaluation

Benchmarked with lm-evaluation-harness v0.4 against the same task setup as openai/gpt-oss-20b reporting:

Notes

• 23-expert MoE with top-4 routing; per-expert dispatch on inference is ~13 tok/s on a single H100 (fp16 compute, NF4 storage).
• Forward path is the same as upstream GptOssExperts; only the per-expert matmul is dispatched to NF4 Linear4bit.