gpt-oss-120b-oQ4

gpt-oss-120b-oQ4

cjnielson44/gpt-oss-120b-oQ4 is an Apple Silicon / oMLX-ready MLX checkpoint for GPT-OSS 120B. It was produced with oMLX oQ4 quantization and published for local inference through oMLX.

This checkpoint is not a uniform 4-bit conversion of every tensor. GPT-OSS uses MoE expert projection tensors that are already stored in MXFP4 form, so those expert tensors are preserved as MXFP4 passthrough tensors and explicitly marked in config.json.

Quantization Details

• Source: local oMLX-compatible gpt-oss-120b MLX checkpoint, originally derived from openai/gpt-oss-120b.
• Quantizer: oMLX oQ4.
• Main quantized tensors: affine oQ4, bits: 4, group_size: 64.
• GPT-OSS MoE expert projections: MXFP4 passthrough, bits: 4, group_size: 32, mode: mxfp4.
• Expert override coverage: all 36 layers for gate_proj, up_proj, and down_proj under model.layers.<i>.mlp.experts.
• Floating dtype used during quantization: bfloat16.

The MXFP4 expert overrides are required for oMLX/MLX loading. Without them, the loader treats the expert tensors as affine-quantized tensors and expects *.biases parameters that do not exist for these MXFP4 expert weights.

Use With oMLX

Download into an oMLX-discoverable model directory:

hf download cjnielson44/gpt-oss-120b-oQ4 \
  --local-dir ~/.omlx/models/cjnielson44/gpt-oss-120b-oQ4

Restart oMLX, then use this model id:

gpt-oss-120b-oQ4

Example OpenAI-compatible request, assuming your oMLX server is listening locally:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OMLX_API_KEY" \
  -d '{
    "model": "gpt-oss-120b-oQ4",
    "messages": [{"role": "user", "content": "Write a short note about Apple Silicon inference."}],
    "max_tokens": 128
  }'

Choosing This Variant

Use oQ4 if you want the smallest non-expert tensor precision among these releases. Because GPT-OSS expert tensors are preserved as MXFP4 in all three variants, the practical disk-size difference between oQ4, oQ6, and oQ8 may be smaller than expected.

For best quality from this release family, prefer cjnielson44/gpt-oss-120b-oQ8.

Verification

This repo was uploaded after local oMLX discovery and load/unload smoke testing. The same GPT-OSS MXFP4 expert override fix used for oQ8 was applied to this oQ4 repo.

Limitations

• Experimental community quantization.
• Requires recent oMLX/MLX support for GPT-OSS and MXFP4 expert tensors.
• No benchmark or perplexity numbers are provided yet.
• This model card does not change the upstream license or usage terms of openai/gpt-oss-120b.