gpt-oss-20b-oQ8

gpt-oss-20b-oQ8

cjnielson44/gpt-oss-20b-oQ8 is an Apple Silicon / oMLX-ready MLX checkpoint for GPT-OSS 20B. It was produced with oMLX oQ8 quantization and published for local inference through oMLX.

Quantization Details

• Source: openai/gpt-oss-20b, downloaded locally without the extra original/ and metal/ artifacts.
• Quantizer: oMLX oQ8.
• Main quantized tensors: affine oQ8, group_size: 64.
• Floating dtype used during quantization: bfloat16.
• GPT-OSS expert handling: the output config and safetensor index were inspected for MXFP4 expert passthrough tensors. 72 explicit per-expert MXFP4 overrides were added after quantization.

GPT-OSS checkpoints use MoE expert tensors, and those tensors need their config metadata to match the actual packed tensor layout. If expert tensors are emitted as MXFP4 passthrough weights, they must be marked as mode: mxfp4, bits: 4, group_size: 32 for oMLX/MLX loading.

Use With oMLX

Download into an oMLX-discoverable model directory:

hf download cjnielson44/gpt-oss-20b-oQ8 \
  --local-dir ~/.omlx/models/cjnielson44/gpt-oss-20b-oQ8

Restart oMLX, then use this model id:

gpt-oss-20b-oQ8

Example OpenAI-compatible request, assuming your oMLX server is listening locally:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OMLX_API_KEY" \
  -d '{
    "model": "gpt-oss-20b-oQ8",
    "messages": [{"role": "user", "content": "Write a short note about Apple Silicon inference."}],
    "max_tokens": 128
  }'

Verification

This repo was generated for oMLX and symlinked under ~/.omlx/models/cjnielson44/ after quantization. The source repo and target Hub repo were checked before generation so this upload does not overwrite an existing cjnielson44 20B oQ variant.

Limitations

• Experimental community quantization.
• Requires recent oMLX/MLX support for GPT-OSS.
• No benchmark or perplexity numbers are provided yet.
• This model card does not change the upstream license or usage terms of openai/gpt-oss-20b.