Step-3.7-Flash-oQ3e

Step-3.7-Flash-oQ3e

Data-driven mixed-precision quantization (oMLX oQ3e, imatrix-calibrated, group size 128) of stepfun-ai/Step-3.7-Flash, in MLX format for Apple Silicon. Runs in mlx-vlm and oMLX.

• ~79 GB on disk (~3.3 bpw text backbone, down from 375 GB BF16)
• Full VLM — step3p7 text backbone (45 layers, MoE with 288 experts / top-8) in oQ3e gs128; vision encoder in 8-bit gs64 (~2.2 GB)
• Peak memory in my tests: ~85 GB text-only, ~87.5 GB with vision — 96GB will do, but you'll want a 128GB Mac for bigger context sizes
• Converted and tested on a MacBook Pro M5 Max 128GB 40 GPU

Quantization

oQ3e allocates bits per layer from an importance-matrix sensitivity pass over calibration data, instead of quantizing everything uniformly — the bulk of the model sits at 3 bits with group size 128, and sensitive layers are kept at 8 bits (group size 64). The vision encoder is quantized separately at 8 bits (group size 64), which is effectively lossless for vision towers. Output is standard MLX affine quantization — no custom kernels or runtime required.

Benchmarks & Variants

mmlu_pro, n=300, greedy. Standard error ≈ 0.03.

These numbers are not representative of what the model can do. Thinking mode wrecks the measurement: left unconstrained, the model wanders into 30k-token reasoning chains, blows the token budget and returns no answer at all — which the harness scores as wrong. So part of what looks like "quantization loss" here is really "ran out of room before answering".

Not apples-to-apples either: the API row used reasoning_effort: low, which keeps the model from running away, and my local runs had no such control. The API can't do the "thinking off" column at all — the endpoint returns "Reasoning is mandatory for this endpoint and cannot be disabled". I'll run more tests later to make each quant's quality relative to the API clearer.

I haven't benchmarked vision quality yet; the encoder passed my qualitative smoke tests.

Thinking control

Step-3.7-Flash is a smart boi. It really likes to think. As a matter of fact, the upstream always enables thinking and doesn't let you turn it off — the chat template unconditionally opens a <think> block, and reasoning_effort barely moves the needle (in my probes, low/medium/high modulate thinking length by ~2×, and the default behaves like max).

I fixed this by implementing the off switch at the template level: when you pass enable_thinking: false (or reasoning_effort: "none"), the template prefills an empty <think>\n</think>\n\n block, so the model skips straight to the answer — no runtime patches, no custom parser, works in any stack that forwards chat_template_kwargs.

Heads up: skipping thinking costs accuracy on hard tasks (0.77 → 0.57 on mmlu_pro in my runs) — but it also turns a 3-hour eval into a 4-minute one. Pick your poison.

Worth knowing: reasoning_effort is a weak lever here. In the open weights the chat template just injects a Reasoning: <level> line into the system prompt — no control token, no budget, nothing that forces the model to stop. It nudges, it doesn't govern. The hosted API behaves the same way; I got the same effect by writing that line into the system prompt myself.

What does work locally is a hard cap. oMLX's thinking_budget closes the <think> block when the budget runs out, and the model takes the hint and answers. In my tests every response came back with a closed block and an actual answer, even when the cut landed mid-sentence.

Changes from upstream

1. tokenizer_class fixed — upstream declares LlamaTokenizerFast for what is a ByteLevel BPE tokenizer, which breaks decode in transformers (literal Ġ/Ċ in output — it silently corrupted my eval scores until I caught it). This repo sets PreTrainedTokenizerFast, restoring correct encode and decode.
2. Chat template extended — enable_thinking=false / reasoning_effort="none" emit an empty-think prefill (see Thinking control). Everything else is unchanged.

Usage

# text and vision (mlx-vlm — plain mlx-lm doesn't support the step3p7 architecture)
mlx_vlm.generate --model mlx-community/Step-3.7-Flash-oQ3e --prompt "..."
mlx_vlm.generate --model mlx-community/Step-3.7-Flash-oQ3e \
  --image photo.jpg --prompt "Describe this image."

# server — thinking off per request:
# POST /v1/chat/completions
#   {"chat_template_kwargs": {"enable_thinking": false}, ...}

# oMLX — discovers the model from the HF cache
omlx serve