Qwable-v1-oQ8-mtp

Qwable-v1-oQ8-mtp

This model was quantized using oQ (oMLX v0.4.4) mixed-precision quantization.

Quantization details

• Model type: qwen3_5_moe
• Bits: 8
• Group size: 64
• Format: MLX safetensors

Performance Benchmark

• Run on: Apple Mac Studio M4 Max 128GB
• https://omlx.ai/benchmarks/9d9hiut3

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwable-v1-oQ8-mtp
Engine: Force mlx-lm
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128           775.4       10.63  1320.6 tok/s    94.8 tok/s       2.125   542.1 tok/s    35.38 GB
pp4096/tg128          2454.4       10.86  1668.8 tok/s    92.8 tok/s       3.834  1101.7 tok/s    36.16 GB
pp8192/tg128          4894.9       11.11  1673.6 tok/s    90.7 tok/s       6.306  1319.4 tok/s    36.50 GB
pp16384/tg128        10506.6       11.80  1559.4 tok/s    85.4 tok/s      12.006  1375.4 tok/s    37.13 GB
pp32768/tg128        24482.0       12.69  1338.5 tok/s    79.4 tok/s      26.093  1260.7 tok/s    38.47 GB
pp65536/tg128        63223.4       14.87  1036.6 tok/s    67.8 tok/s      65.112  1008.5 tok/s    41.15 GB
pp131072/tg128      190082.3       19.00   689.6 tok/s    53.0 tok/s     192.496   681.6 tok/s    46.53 GB
pp200000/tg128      395018.9       23.64   506.3 tok/s    42.6 tok/s     398.021   502.8 tok/s    52.19 GB

Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x          94.8 tok/s     1.00x  1320.6 tok/s  1320.6 tok/s       775.4       2.125
2x         140.2 tok/s     1.48x   821.5 tok/s   410.8 tok/s      2493.0       4.319
4x         200.5 tok/s     2.11x  1498.1 tok/s   374.5 tok/s      2614.5       5.288
8x         257.6 tok/s     2.72x  1487.3 tok/s   185.9 tok/s      5128.2       9.483

Intelligence Benchmark Comparison

Intelligence Benchmark Comparison

               Mode    Sampled         Qwable-v1-oQ8-mtp
--------------------------------------------------------
MMLU           Sample  1000/14042                  89.6%
TRUTHFULQA     Full    817                         88.9%
HUMANEVAL      Full    164                         86.0%
MBPP           Sample  200/500                     69.5%
LIVECODEBENCH  Sample  300/1055                    49.3%

--- Detail ---

Model: Qwable-v1-oQ8-mtp
Benchmark         Accuracy   Correct   Total   Time(s)   Think
--------------------------------------------------------------
MMLU                 89.6%       896    1000    3046.7     Yes
TRUTHFULQA           88.9%       726     817    1518.2     Yes
HUMANEVAL            86.0%       141     164    1218.3     Yes
MBPP                 69.5%       139     200    1691.5     Yes
LIVECODEBENCH        49.3%       148     300   11606.2     Yes