qwen3.8-27b-p150x2

qwen3.8-27b-p150x2

Qwen/Qwen3.8-27B served with vLLM on two Tenstorrent p150a cards linked by ethernet (2 Blackhole chips, 32 GB each). One image, three profiles: tp2 (default, tensor-parallel over both chips), p1d1 (one chip prefills, the other decodes), tp2-dflash2 (tp2 + DFlash2 speculative decoding). Sibling package for the 4-chip P300x2: qwen3.8-27b-p300x2 (profiles plain, batch8-dflash2, single-user-dflash2).

Runs on p150x2 or p150x2 or p150x2 — see the serve profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/qwen3.8-27b-p150x2 --with-weights
tt-model serve changh95/qwen3.8-27b-p150x2

pull --with-weights downloads the Docker image and the Qwen/Qwen3.8-27B weights at 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 (into your HF cache; they are not in the image). serve starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

Run

tt-model pull  changh95/qwen3.8-27b-p150x2 --with-weights
tt-model serve changh95/qwen3.8-27b-p150x2                        # tp2 (default)
tt-model serve changh95/qwen3.8-27b-p150x2 --profile p1d1
tt-model serve changh95/qwen3.8-27b-p150x2 --profile tp2-dflash2  # needs: hf download incoai/Qwen3.8-27B-DFlash2

• OpenAI-compatible API on port 20000, model id Qwen/Qwen3.8-27B; streaming, tool calls (qwen3_coder) and reasoning content (qwen3) work as in vLLM; "chat_template_kwargs": {"enable_thinking": false} turns thinking off.
• First boot converts the weights (6-8 min); later boots take about 2 min.

curl -s http://localhost:20000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen3.8-27B", "messages": [{"role": "user", "content": "Explain KV caching in two sentences."}], "max_tokens": 256}'

Profiles at a glance

• tp2 is 1.3x faster per user than p1d1 at 1 user and has 3.8x the KV pool; p1d1 keeps TPOT at 71-87 ms at 1-8 users across every prompt length, where tp2 at 8 users degrades to 134-255 ms once prompts reach 16k-32k tokens. At 8 users on prompts up to 4k tokens p1d1 delivers 84-90% of tp2's aggregate throughput.
• tp2-dflash2 is lossless (greedy trajectory of tp2 up to bf16 near-ties); its gain fades with prompt length (2.4x at 128 tokens, 1.9x at 8k, 1.4x at 32k) and grows with answer length (3.1-3.2x at 1k-token answers).

Performance

Method: tt-inference-server --workflow benchmarks grid (random prompts of exactly ISL tokens, OSL output tokens, streaming). t/s = aggregate output tokens/s, t/s/u = per user (= OSL / mean end-to-end latency); TTFT and TPOT are means. Measured on a P300x2 development box emulating the p150x2 (two chips on different cards over a 2-channel ethernet link), not on a p150a pair.

tp2: throughput, t/s (t/s/u)

@18: the KV pool seats 18 prompts of that length.

tp2: TTFT ms / TPOT ms

p1d1: throughput, t/s (t/s/u)

p1d1: TTFT ms / TPOT ms

tp2-dflash2 vs tp2, 1 user

• 4 users: 113 vs 58 t/s aggregate at 128/128 (1.9x), 151 vs 62 at 128/1,024 (2.4x), 47 vs 35 at 8k/128 (1.3x); no gain at 32k prompts with 4 users.

Notes

• Not yet run on real p150a cards: all numbers above come from the P300x2 emulation (two chips on different cards, one ethernet cable = 2 links). tp2 and tp2-dflash2 ship a 2-channel mesh descriptor (models/demos/blackhole/qwen36/mesh/p150_x2_2link_mesh_graph_descriptor.textproto): the channel count is the minimum number of trained links the fabric requires and it uses every link it finds, so a pair joined by one cable (2 links) or two cables (4 links) both boot. tt-metal's stock p150_x2 descriptor asks for 4 links and refuses a one-cable pair at start-up (Expected 4 eth links). p1d1 uses a separate two-mesh descriptor whose link count is TT_PD_CONNS (default 2 = one cable).
• tp2-dflash2 produces greedy output only and serves at most 4 users; tp2 and p1d1 support every vLLM sampling option.
• Speculative decoding is not available in p1d1.

Serve profiles

One image serves every profile below; pick one with --profile.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: