gpt-oss-120b-oQ8

gpt-oss-120b-oQ8

cjnielson44/gpt-oss-120b-oQ8 is an Apple Silicon / oMLX-ready MLX checkpoint for GPT-OSS 120B. It was produced with oMLX oQ8 quantization and published for local inference through oMLX.

This checkpoint is not a uniform 8-bit conversion of every tensor. GPT-OSS uses MoE expert projection tensors that are already stored in MXFP4 form, so those expert tensors are preserved as MXFP4 passthrough tensors and explicitly marked in config.json.

Quantization Details

• Source: local oMLX-compatible gpt-oss-120b MLX checkpoint, originally derived from openai/gpt-oss-120b.
• Quantizer: oMLX oQ8.
• Main quantized tensors: affine oQ8, bits: 8, group_size: 64.
• GPT-OSS MoE expert projections: MXFP4 passthrough, bits: 4, group_size: 32, mode: mxfp4.
• Expert override coverage: all 36 layers for gate_proj, up_proj, and down_proj under model.layers.<i>.mlp.experts.
• Floating dtype used during quantization: bfloat16.

The MXFP4 expert overrides are required for oMLX/MLX loading. Without them, the loader treats the expert tensors as affine-quantized tensors and expects *.biases parameters that do not exist for these MXFP4 expert weights.

Use With oMLX

Download into an oMLX-discoverable model directory:

hf download cjnielson44/gpt-oss-120b-oQ8 \
  --local-dir ~/.omlx/models/cjnielson44/gpt-oss-120b-oQ8

Restart oMLX, then use this model id:

gpt-oss-120b-oQ8

Example OpenAI-compatible request, assuming your oMLX server is listening locally:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OMLX_API_KEY" \
  -d '{
    "model": "gpt-oss-120b-oQ8",
    "messages": [{"role": "user", "content": "Write a short note about Apple Silicon inference."}],
    "max_tokens": 128
  }'

Choosing This Variant

Use oQ8 if you want the highest quality variant in this release family. Because GPT-OSS expert tensors are preserved as MXFP4 in all three variants, oQ8 may be the best default: it keeps higher precision for the non-expert tensors while only modestly increasing disk usage relative to oQ4 and oQ6.

This is the local variant retained after publish and smoke testing.

Verification

This repo was uploaded after local oMLX discovery and load testing. The GPT-OSS MXFP4 expert override fix in config.json was verified by successful oMLX loading.

Limitations

• Experimental community quantization.
• Requires recent oMLX/MLX support for GPT-OSS and MXFP4 expert tensors.
• No benchmark or perplexity numbers are provided yet.
• This model card does not change the upstream license or usage terms of openai/gpt-oss-120b.