qwen3-32b-bh-galaxy

qwen3-32b-bh-galaxy

Qwen3-32B (32.8B dense, 64 layers, GQA 64/8 heads, QK-norm, 131K context) served on a Blackhole Galaxy: 32 Blackhole chips in one 8×4 mesh, with tensor-parallel prefill and 4 data-parallel decode lanes through vLLM.

Runs on p150x32 (mesh BH-Galaxy): 131,072-token context, up to 32 concurrent sequences, prefix caching on.

This revision is rebuilt on the latest releases: tt-metal v0.79.0, vLLM 0.26.0, vllm-tt-plugin main dd287f9 and tt-inference-server v0.23.0. It also carries the S1-S3 optimizations (see the changelog). Decode is about 29 % faster than the previous revision of this repo: 9.71 ms vs 13.72 ms at 32 users.

Packaged with tt-model-manager 0.1.0. The bundle uses manifest schema 5 and is self-contained: it ships the ttnn, vLLM and vllm-tt-plugin wheels plus the model code, with no container image.

Quickstart

tt-model pull  changh95/qwen3-32b-bh-galaxy --with-weights
tt-model serve changh95/qwen3-32b-bh-galaxy

What the two commands do:

• pull --with-weights installs the bundle into its own Python 3.10 venv, offline, from the vendored wheels. It then downloads the Qwen/Qwen3-32B weights at 9216db5781bf21249d130ec9da846c4624c16137 next to the bundle. The weights are not in this repo.
• serve starts an OpenAI-compatible server with model id Qwen/Qwen3-32B on port 20000, or on the next free port if that one is busy. On the first start it compiles kernels and captures traces for the whole mesh. On this revision a cold first start (empty caches) took about 48 minutes on the reference host; later starts reuse the caches the bundle keeps under .tt_cache and .cache and are much faster. The server is ready when it logs Application startup complete.

Host requirements

• The Blackhole Galaxy driver and firmware.
• The SFPI compiler release 7.78.0 at /opt/tenstorrent/sfpi, as tt-metal v0.79.0 pins it (tt_metal/sfpi-version). The previous revision used 7.76.0. The ttnn wheel does not vendor the compiler; the kernel JIT looks for it at /opt/tenstorrent/sfpi. The JIT checks <venv>/lib/python3.10/site-packages/ttnn/runtime/sfpi before /opt, so instead of upgrading /opt you can unpack the SFPI 7.78.0 release (https://github.com/tenstorrent/sfpi) there.
• Disk: about 900 MB for the bundle plus about 62 GB of weights, and the first start writes about 60 GB of converted weights to .tt_cache and about 16 GB of compiled kernels to .cache; plan for roughly 140 GB free.

Performance

Measurement setup: Blackhole Galaxy, real weights, bfp8 MLP weights and KV cache, greedy decoding, warm traces. All numbers below were measured on this revision's release trees (tt-metal v0.79.0 + 9f69d68757d, plugin dd287f9 + 3fa2f09) on 2026-10-01. The ttnn wheel here is that build's binaries, byte-identical.

tt-metal demo

text_qwen_demo.py, run on this bundle's model code:

Serving through vLLM

Setup: vllm bench serve, random prompts, prefix caching on, --ready-check-timeout-sec 0, seeds that vary with ISL/OSL. Each value is the mean of two full 54-cell matrix runs on the run.sh configuration below. Per-user tok/s is 1000 / median TPOT, and aggregate is users × per-user.

The previous revision's serving numbers were measured on its predecessor build (e3cd84b, 15.3 ms demo decode). Against the S3 serving baseline (main tree on tt-metal f8c78f9), all 54 cells are within +0.0..+0.9 % on t/s/u.

tt-inference-server (v0.23.0 + catalog entry), run.py against the running server

• --workflow benchmarks: PASS.
  • 1 user 128/128: TTFT 95.8 ms, TPOT 10.0 ms.
  • 32 users 128/128: 2,442 tok/s, TTFT 401 ms, TPOT 10.0 ms.
  • The previous revision gave: 1 user TTFT 94 ms, TPOT 15.6 ms; 32 users 1,715 tok/s, TTFT 395 ms.
• --workflow evals: PASS 3/3. AIME24 83.3, MATH-500 94.4, GPQA Diamond 69.2. The previous revision scored 76.7, 93.2 and 60.6.

These TIS runs used the plugin's default decode interleave (on). They were not repeated with it off. At 1 and 32 users × 128/128, the matrix runs with it off give a mean TPOT of 9.9-10.1 ms, the same as the 10.0 ms from TIS.

Serving configuration note: decode interleave is off

vllm-tt-plugin main (dd287f9) turns decode-priority interleave (#119) on by default. On this model it hurts the serving matrix:

• At 32 users × 16384/128, median TPOT goes from 16 ms to 63 ms. TTFT in that cell improves by 15 %.
• At 1-4 users, short-prompt TTFT rises by 2-8 %.

run.sh therefore passes "decode_interleave_enabled": false in --additional-config. Every number in the serving table was measured with that setting. To get the plugin default back, remove that key.

Serving command

run.sh runs the vLLM OpenAI server with:

• DP 4 × 8 sequences (32 global), block size 64, max model length 131072, --max-num-batched-tokens 131072, seed 9472.
• The hermes tool parser and the qwen3 reasoning parser, xgrammar structured outputs, and enable_thinking: false as the default chat-template kwarg.
• --shutdown-timeout 120, so the EngineCore closes the mesh cleanly on stop.
• --additional-config '{"tt": {...}}' with these keys:
  • dispatch_core_axis col
  • sample_on_device_mode all
  • fabric_config FABRIC_2D_TORUS_XY
  • worker_l1_size 1345000
  • trace_region_size 335544320 (320 MiB). The previous revision used 256 MiB. The batched-prefill traces need the larger region.
  • decode_interleave_enabled false

Provenance

These are the exact sources the bundle was built from. metal/models in this repo is the model code the wheels were measured with, and run.sh puts it on PYTHONPATH. The branch commits are local to the build host and have not been pushed yet.

Changelog vs the previous revision (780b268, tt-metal f8c78f9, ttnn 0.75.0rc10.dev1597)

All model items below are on by default. Each one has a QWEN_* switch, and setting it to 0 or off restores the old path.

Releases

• tt-metal is rebased onto v0.79.0: 309 upstream commits on top of f8c78f9. The model-side change is limited to the trace allocation tracker API rename. Decode output is bit-identical to the pre-rebase tree.
• vllm-tt-plugin is updated to upstream main dd287f9. That brings speculative-decoding infrastructure, which this model does not use, and decode interleave #119, which this bundle switches off.
• tt-inference-server is updated to v0.23.0.
• SFPI moves from 7.76.0 to 7.78.0.

S1: collectives and prefill batching

• QKV all-reduce goes direct with the view-8 layout (QWEN_BH_QKV_AR_DIRECT=view8).
• The norm all-gather is async with persistent buffers (QWEN_BH_NORM_AG_MODE=async).
• WO all-reduce goes direct (QWEN_BH_WO_AR_DIRECT=1).
• The SDPA all-gather runs on 2 links with persistent CCL buffers (QWEN_SDPA_AG_LINKS=2, QWEN_CCL_PERSISTENT=sdpa).
• D14 batched prefill: requests from 2 or more users are prefilled together, including a 2-user bucket (QWEN_BATCHED_PREFILL_MIN_USERS=2, QWEN_BATCHED_PREFILL_2USER_BUCKET=1). This cuts TTFT at 16 users × 128 tokens by about 54 %.
• The serving trace region grows to 320 MiB.
• Plugin: the prefill coalescing hold and decode-priority interleave are opt-in and stay off.

S2: decode matmuls

• QKV, WO, FF2 and FF13 move to 1D-multicast interleaved program configs (QWEN_S2_*_MM=il).
• Decode goes from 12.68 to 11.58 ms.

S3A: fusions without C++ changes

• Fused split QK-norm.
• Tail set A: fused on-device argmax plus select, a fast plus-one kernel, and direct LM-head output.
• Decode goes from 11.58 to 10.70 ms.

S3B: C++ ports, built into this ttnn wheel

• SiLU(FF1)·FF3 epilogue fused into the FF13 all-reduce (QWEN_S3_FF13_EPILOGUE).
• all_gather_concat (QWEN_S3_AG_CONCAT).
• all_reduce_create_qkv_heads (QWEN_S3_AR_CREATE_HEADS).
• Fused residual + RMSNorm + stats all-gather (QWEN_S3_FUSED_RMS).
• All four are ported to the 2D torus.
• Decode goes from 10.70 to 9.76 ms (9.71 ms on the release tree).

Known differences vs the pre-release main tree

• Demo TTFT at 1 user is about 15-20 ms higher (130-149 ms vs 113-119 ms). The extra time is in the upstream runtime's semaphore resets during the prefill/decode mode switch. The serving TTFT does not show it: the TIS 1-user TTFT is 95.8 ms vs 95.5 ms.