Ornith-1.0-35B-MTPLX-Vision-mxfp4-int4-mtp

Ornith-1.0-35B-MTPLX-Vision (mxfp4 body, int4 MTP head)

A vision-enabled Ornith 1.0 35B model for Apple Silicon via MTPLX native speculative decoding (MTP). The body is the Shiftedx mxfp4 + abliterated build; the MTP head has been re-quantized from its original 5-bit to 4-bit affine (group 64) so the model loads on stock MTPLX without a loader patch.

Why this derivative exists

The Shiftedx mxfp4-vision model has the best MTP acceptance of the Ornith MTP variants I measured, but its MTP head is prequantized at 5-bit. Stock MTPLX (up to 2.1.0) miscomputes the 5-bit group size in _infer_prequantized_group_size (integer-floor 32 // 5 = 6 instead of 6.4), infers group_size 60 instead of 64, and nn.quantize rejects the layer at load:

ValueError: [quantize] The last dimension of the matrix needs to be
divisible by the quantization group size 60 ... shape (2048,4096)

That bug is filed as youssofal/MTPLX#182 with a fix in PR #183. This repo is the model-side workaround: the MTP head is re-quantized to 4-bit affine (group 64), which stock MTPLX loads natively (5-bit packing is the only non-power-of-2 width that triggers the bug; 4-bit does not).

Transform

Per quantized MTP tensor (a weight / scales / biases triple):

1. dequantize at 5-bit, group 64 to bf16
2. re-quantize at 4-bit, group 64, affine

13 quantized triples transformed, 33 norm / layernorm tensors passed through unchanged. The body (model shards) and vision tower are unchanged from the Shiftedx source.

The 5-bit to 4-bit round-trip costs a small amount of draft-head fidelity: mean abs error ~0.0006 per tensor against the original 5-bit dense, which shows up as a modest acceptance drop at the deepest draft position (see below).

Performance (measured)

The signal that matters for an MTP head is draft acceptance at the chosen bit width: high acceptance at lower bits means the draft head matches the target well without paying for extra precision. Wall-clock tok/s is reported for completeness but varies with hardware state and is not the figure of merit here.

Acceptance at depth 3, by MTP head bit width (same M5 Pro config)

Apple M5 Pro 64 GB, MTPLX 2.1.0 (stock, no patch), mtp_history_policy=committed (CLI default), --draft-temperature 0, thinking OFF, warm.

This repo is the highest-acceptance Ornith MTP variant at 4-bit, and the highest-acceptance variant overall that loads on stock MTPLX. At 4-bit it reaches pos3=0.66, close to the 5-bit source's 0.76, while every other 4-bit variant is at 0.52 or collapses. The 5-bit source is marginally higher but costs an extra bit and needs the MTPLX loader fix; this repo gets most of the way there at lower precision with no patch.

The wang-yang source collapse (pos3=0.00) is not a quantization issue: its shipped MTP draft head diverges from its own target body (264 of 1803 body leaf tensors differ from the vision variants; the MTP head is bit-identical across variants). See the comparison repo Jonandrop/Ornith-1.0-35B-MTPLX-Vision README for the weight-diff evidence.

Wall-clock tok/s (this repo, for reference)

tok/s (e2e) is generated_tokens / total_elapsed, wall-clock including prefill. Varies run-to-run with thermal/contention state.

Why naive RTN, not AWQ or GPTQ

The 5-bit to 4-bit re-quantization here is plain round-to-nearest (RTN). I tried two techniques that beat RTN for weight quality on standard LLMs (AWQ, arXiv:2306.00978; GPTQ, arXiv:2210.17323). Both collapsed MTP acceptance below RTN when applied to this draft head:

• AWQ (activation-aware per-channel weight scaling): reduced MTP head weight reconstruction error 13.6% vs RTN across all 10 standard Linears, but acceptance collapsed to pos3 0.02 (vs RTN 0.66). Cause: AWQ scales weights by s per input channel and must inverse-scale activations by 1/s at runtime. In a transformer draft head those rescaled activations feed residual streams, RMSNorms, attention softmax, and the (RTN) MoE experts, none of which are re-quantized with W*s, so the rescale leaks and corrupts them. AWQ's rescale is only invisible when every consumer of the rescaled activation uses the scaled weights.
• GPTQ (sequential Hessian error compensation, no runtime rescale): acceptance collapsed to pos3 0.00. Cause: a correct GPTQ needs the per-column quantizer to exactly match MLX's group-64 affine grid; a column-wise uniform quantizer misaligns the error compensation and perturbs the final weights. Group-aware GPTQ is more work and the proven margin at 4-bit is only ~0.1 to 0.2 perplexity (not acceptance), so the expected upside is small.

The headline finding from this: weight reconstruction error is not acceptance. A 13.6% weight-error win (AWQ) produced acceptance 30x worse. RTN remains the best stock-loadable 4-bit MTP head here. The 5-bit source (0.76 at pos3) stays marginally ahead but needs the MTPLX loader fix (#182/#183).

Usage

mtplx start --model <path-to-this-model-dir> --port 8092 \
  --chat-template-path <path-to-this-model-dir>/chat_template.jinja

OpenAI-compatible endpoint supporting text and image input via POST /v1/chat/completions. The MTPLX CLI defaults to the committed history policy this model requires.

Sources

Released under Apache-2.0, preserving the upstream licenses.