Holo-3.1-35B-A3B-oQ8

Holo-3.1-35B-A3B-oQ8

This model was quantized using oQ (oMLX v0.4.2) mixed-precision quantization.

Quantization details

• Model type: qwen3_5_moe
• Bits: 8
• Group size: 64
• Format: MLX safetensors

Performance Benchmark

• Run on: Apple Mac Studio M4 Max 128GB
• https://omlx.ai/benchmarks/gx4u5a6h

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Holo-3.1-35B-A3B-oQ8
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128           799.5       10.92  1280.7 tok/s    92.3 tok/s       2.186   527.0 tok/s    35.38 GB
pp4096/tg128          2455.9       11.43  1667.8 tok/s    88.2 tok/s       3.908  1081.0 tok/s    36.16 GB
pp8192/tg128          4885.5       11.82  1676.8 tok/s    85.2 tok/s       6.387  1302.6 tok/s    36.50 GB
pp16384/tg128        10476.7       12.11  1563.9 tok/s    83.2 tok/s      12.015  1374.3 tok/s    37.13 GB
pp32768/tg128        24378.5       13.33  1344.1 tok/s    75.6 tok/s      26.072  1261.8 tok/s    38.47 GB
pp65536/tg128        62954.3       15.55  1041.0 tok/s    64.8 tok/s      64.929  1011.3 tok/s    41.15 GB

Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x          92.3 tok/s     1.00x  1280.7 tok/s  1280.7 tok/s       799.5       2.186
2x         132.4 tok/s     1.43x  1249.7 tok/s   624.9 tok/s      1638.3       3.572
4x         192.3 tok/s     2.08x  1472.3 tok/s   368.1 tok/s      2639.1       5.445
8x         245.2 tok/s     2.66x  1473.5 tok/s   184.2 tok/s      5183.5       9.735

Intelligence Benchmark Comparison

Intelligence Benchmark Comparison

              Mode    Sampled                Holo-3.1-35B-A3B-oQ4         Holo-3.1-35B-A3B-oQ8
----------------------------------------------------------------------------------------------
MMLU          Sample  500/14042                             83.0%                        83.0%

--- Detail ---

Model: Holo-3.1-35B-A3B-oQ4
Benchmark         Accuracy   Correct   Total   Time(s)   Think
--------------------------------------------------------------
MMLU                 83.0%       415     500     875.7     Yes

Model: Holo-3.1-35B-A3B-oQ8
Benchmark         Accuracy   Correct   Total   Time(s)   Think
--------------------------------------------------------------
MMLU                 83.0%       415     500    1060.2     Yes