MiniMax-M2.7-exl3

Quantization was performed using exllamav3 v0.0.29 (commit cb1a436).

The original model is distributed in FP8 (float8_e4m3fn), not FP16/BF16 — this is why the 8.0bpw quant is nearly identical in size to the original.

PPL and KL divergence metrics are non-computable for this model due to inf/NaN values produced by layer 61 experts during forward pass. This is not specific to EXL3 — the same issue affects 21-38% of GGUF quantizations across multiple providers. Top-K agreement against the original is provided instead.

<details> <summary>Quantization Notes</summary>

Inf/NaN values during calibration

Some experts in the model produce inf values during calibration (e.g. experts 61 and 74 in the last layer had inf values in their down-projection calibration state). The lm_head layer also exhibited NaN values in its calibration state (445K NaN out of 1.5B elements).

This causes Cholesky decomposition to fail during quantization, as the Hessian matrix is no longer positive-definite. ExLlamaV3 does not handle this case gracefully — quantization crashes after exhausting retry attempts. A local patch was applied to fall back to uncalibrated quantization for the affected tensors. Given that only a handful of experts out of 256 in the last layer are affected, the impact on output quality is expected to be minimal.

Note that inf/NaN values are present in the original model during inference as well — both the quantized and original models produce NaN perplexity. This appears to be caused by numerically unstable expert weights that produce overflow during forward pass, not by the quantizer itself. The same layer (blk.61.ffn_down_exps) has been identified as causing NaN perplexity across GGUF quantizations by multiple providers.

</details>

Setup tool calling for TabbyAPI

Add tool_format: minimax_m2 to your config.yml or per-model tabby_config.yml. Also enable reasoning: true to properly separate thinking blocks from output:

model:                                                          
  tool_format: minimax_m2
  reasoning: true