Step-3.7-Flash-oQ2e

Step-3.7-Flash-oQ2e

Data-driven mixed-precision quantization (oMLX oQ2e, imatrix-calibrated, group size 128) of stepfun-ai/Step-3.7-Flash, in MLX format for Apple Silicon. Runs in mlx-vlm and oMLX.

• ~59 GB on disk (~2.4 bpw text backbone, down from 375 GB BF16)
• Full VLM — step3p7 text backbone (45 layers, MoE with 288 experts / top-8) in oQ2e gs128; vision encoder in 8-bit gs64 (~2.2 GB)
• Peak memory in my tests: ~63 GB text-only, 65.5 GB with vision — plan for 96 GB+
• Converted and tested on a MacBook Pro M5 Max 128GB 40 GPU

Quantization

oQ2e allocates bits per layer from an importance-matrix sensitivity pass over calibration data, instead of quantizing everything uniformly. Group size 128 lowers metadata overhead (~0.25 bpw saved vs gs64) at a small accuracy cost. The vision encoder is quantized separately at 8 bits (group size 64), which is effectively lossless for vision towers. Output is standard MLX affine quantization — no custom kernels or runtime required.

Benchmarks & Variants

mmlu_pro, n=300, greedy. Standard error ≈ 0.03.

These numbers are not representative of what the model can do. Thinking mode wrecks the measurement: left unconstrained, the model wanders into 30k-token reasoning chains, blows the token budget and returns no answer at all — which the harness scores as wrong. So part of what looks like "quantization loss" here is really "ran out of room before answering".

Not apples-to-apples either: the API row used reasoning_effort: low, which keeps the model from running away, and my local runs had no such control. The API can't do the "thinking off" column at all — the endpoint returns "Reasoning is mandatory for this endpoint and cannot be disabled". I'll run more tests later to make each quant's quality relative to the API clearer.

I haven't benchmarked vision quality yet; the encoder passed my qualitative smoke tests.

Thinking control

Step-3.7-Flash is a smart boi. It really likes to think. As a matter of fact, the upstream always enables thinking and doesn't let you turn it off — the chat template unconditionally opens a <think> block, and reasoning_effort barely moves the needle (in my probes, low/medium/high modulate thinking length by ~2×, and the default behaves like max).

I fixed this by implementing the off switch at the template level: when you pass enable_thinking: false (or reasoning_effort: "none"), the template prefills an empty <think>\n</think>\n\n block, so the model skips straight to the answer — no runtime patches, no custom parser, works in any stack that forwards chat_template_kwargs.

Heads up: skipping thinking costs accuracy on hard tasks (0.68 → 0.54 on mmlu_pro in my runs) — but it also turns a 4-hour eval into a 3-minute one. Pick your poison.

Worth knowing: reasoning_effort is a weak lever here. In the open weights the chat template just injects a Reasoning: <level> line into the system prompt — no control token, no budget, nothing that forces the model to stop. It nudges, it doesn't govern. The hosted API behaves the same way; I got the same effect by writing that line into the system prompt myself.

What does work locally is a hard cap. oMLX's thinking_budget closes the <think> block when the budget runs out, and the model takes the hint and answers. In my tests every response came back with a closed block and an actual answer, even when the cut landed mid-sentence.

Changes from upstream

1. tokenizer_class fixed — upstream declares LlamaTokenizerFast for what is a ByteLevel BPE tokenizer, which breaks decode in transformers (literal Ġ/Ċ in output — it silently corrupted my eval scores until I caught it). This repo sets PreTrainedTokenizerFast, restoring correct encode and decode.
2. Chat template extended — enable_thinking=false / reasoning_effort="none" emit an empty-think prefill (see Thinking control). Everything else is unchanged.

Usage

# text and vision (mlx-vlm — plain mlx-lm doesn't support the step3p7 architecture)
mlx_vlm.generate --model mlx-community/Step-3.7-Flash-oQ2e --prompt "..."
mlx_vlm.generate --model mlx-community/Step-3.7-Flash-oQ2e \
  --image photo.jpg --prompt "Describe this image."

# server — thinking off per request:
# POST /v1/chat/completions
#   {"chat_template_kwargs": {"enable_thinking": false}, ...}

# oMLX — discovers the model from the HF cache
omlx serve