Qwable-v1-oQ4-mtp

Qwable-v1-oQ4-mtp

This model was quantized using oQ (oMLX v0.4.4) mixed-precision quantization.

Quantization details

• Model type: qwen3_5_moe
• Bits: 4
• Group size: 64
• Format: MLX safetensors

Performance Benchmark

• Run on: Apple Mac Studio M4 Max 128GB
• https://omlx.ai/benchmarks/ico1h93g

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwable-v1-oQ4-mtp
Engine: Force mlx-lm
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128           756.0        8.36  1354.4 tok/s   120.6 tok/s       1.818   633.7 tok/s    19.94 GB
pp4096/tg128          2427.7        8.68  1687.2 tok/s   116.1 tok/s       3.530  1196.4 tok/s    20.71 GB
pp8192/tg128          4853.5        8.93  1687.8 tok/s   112.8 tok/s       5.988  1389.5 tok/s    21.06 GB
pp16384/tg128        10443.8        9.58  1568.8 tok/s   105.2 tok/s      11.661  1416.0 tok/s    21.68 GB
pp32768/tg128        24245.5       10.75  1351.5 tok/s    93.8 tok/s      25.611  1284.5 tok/s    23.02 GB
pp65536/tg128        62700.3       12.98  1045.2 tok/s    77.6 tok/s      64.349  1020.4 tok/s    25.71 GB
pp131072/tg128      188035.9       16.91   697.1 tok/s    59.6 tok/s     190.184   689.9 tok/s    31.08 GB
pp200000/tg128      392351.0       21.43   509.7 tok/s    47.0 tok/s     395.072   506.6 tok/s    36.74 GB

Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x         120.6 tok/s     1.00x  1354.4 tok/s  1354.4 tok/s       756.0       1.818
2x         170.3 tok/s     1.41x   817.6 tok/s   408.8 tok/s      2504.6       4.008
4x         235.1 tok/s     1.95x  1525.1 tok/s   381.3 tok/s      2574.5       4.863
8x         290.8 tok/s     2.41x  1508.9 tok/s   188.6 tok/s      5063.4       8.950