Qwen3-235B-A22B-Instruct-2507-exl3

Quantization was performed using exllama3 v0.0.15.

The 5.5bpw-opt and 6.23bpw-opt quantizations were created using a combination of the 5.0bpw, 6.0bpw, and 7.0bpw measurements to optimize layer-wise bit allocation.

Full measurements data: json

Metrics explanation

• KL-divergence: Measures the difference between probability distributions of quantized and original models. Lower is better (closer to original).
• Perplexity: Indicates how well the model predicts the next token. Lower values mean better prediction quality.
• Top-K agreement: Shows how often the quantized model's top-K predictions exactly match the original model's predictions. Higher values indicate better preservation of the original model's behavior (1.0 = perfect match):
  • K=1: The fraction of times both models predict the exact same token as their top-1 choice
  • K=5: The fraction of times both models have the same 5 tokens in their top-5 predictions in the exact same ranking order

Tool Calls Support for Qwen/GLM Models

The official tabbyAPI doesn't support tool calls for Qwen and GLM models yet.

If you're using Qwen-Code, OpenClaw, or similar software that need tool call support, you can use my fork with the tools-support branch:

Clone directly:

git clone -b tools-support https://github.com/NeuroSenko/tabbyAPI

Or add to existing tabbyAPI installation:

git remote add neurosenko https://github.com/NeuroSenko/tabbyAPI
git fetch neurosenko
git checkout -b tools-support neurosenko/tools-support

This branch includes native tool calling support for Qwen and GLM model families.