qwen3.8-27b-p150x8

qwen3.8-27b-p150x8

Qwen/Qwen3.8-27B on one Tenstorrent P150x8 (8 Blackhole chips) — one image, three serve profiles. spec-decode (default) runs one vLLM on all eight chips (8-way tensor parallel) with lossless speculative decoding for greedy requests — the DFlash2 block drafter at 1–4 users, the checkpoint's MTP head at 5–8 users or long contexts: ~10 ms/token at one user on natural text and the fastest long-prompt first token on this board (1.1 s at 8K, 21 s at 131K). plain is the same engine without a drafter (sampled workloads, the exactness reference). p4d4 splits the board into a prefill half and a decode half (two vLLMs, Mooncake hand-off) for 4–32 concurrent users: 13.8 ms/token at 4 users, 672 t/s and a 1.1 s first token at 32. Up to 32 users, 262K context, full sampling; GSM8K 0.82–0.835 in every profile.

Runs on p150x8 — see the serve profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

One image, three serve profiles — see How to serve below.

How to query

Every profile speaks the OpenAI API on port 20000 (model id Qwen/Qwen3.8-27B); GET / describes what is running (mode, chips, drafter).

curl -s http://localhost:20000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen3.8-27B", "messages": [{"role": "user", "content": "Explain KV caching in two sentences."}], "max_tokens": 256, "temperature": 0}'

Greedy requests (temperature 0) decode speculatively in spec-decode and p4d4; sampled requests run plain decode on the same server. Streaming ("stream": true), tool calls (qwen3_coder parser) and reasoning content (qwen3 parser) work as in vLLM. --profile spec-decode (default) needs huggingface-cli download z-lab/Qwen3.8-27B-DFlash2 on the host once, into the Hugging Face cache the container mounts; plain and p4d4 need nothing beyond the model weights.

Serve profiles

One image serves every profile below; pick one with --profile.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

Profiles at a glance

<!-- t/s/u = output tokens per second per user = aggregate t/s / users. Sources (clock pinned 1200 MHz, container of the 2026-10-01 image 7dfae48db1ed whose decode/verify/prefill programs this image runs unchanged; the profile was named `hybrid` then, so its logs keep that name): spec-decode = logs/points_merged_pub3_20261001/hybrid/summary.txt (random 128/128 @1 73.3, @4 189.1/4, @32 453.3/32; gsm8k @1 93.1, @4 249.0/4; 8K TTFT 1076 ms), re-measured within the gate's 5 % on the 2026-10-02 images (logs/points_merged_pub4_20261002/hybrid/summary.txt, logs/points_merged_pub5_20261002/spec-decode/summary.txt, logs/points_merged/spec-decode/summary.txt); plain = logs/points_merged_pub3_20261001/plain/summary.txt (42.6, 143.6/4, 475.3/32, 8K TTFT 995 ms); p4d4 = logs/sweeps/v13/pivots.md t/s/u row 128/128 (73.1, 61.3, 21.0) + TTFT 8K 1409 ms, real text logs/served_spec_mtp/summary.txt (74.40, 265.11/4; TP=4 does not run the ring-8 chain, so unchanged) -->

Use when

spec-decode (default) — 1–8 users on natural text, long documents, agent loops; one profile for every greedy drafter band.

• 9.5 ms/token at one user (93.1 t/s), 13.5 at four (249.0 t/s) on GSM8K chat prompts (plain 22.4 / 23.8); on random-like tokens 12.3 / 18.2 ms.
• The fastest first token on this board: 1.1 s at 8K, 2.0 s at 16K, 4.0 s at 32K, 21 s at 131K (p4d4: 1.4 / 2.7 / 6.2 / 30.5 s). Contexts ≤ 12K decode with the block drafter, longer ones switch to the MTP head without a re-prefill (15.5–16.5 ms/token).
• At 5–8 users the engine drafts with the checkpoint's MTP head automatically (22.0 ms/token at 8 users, 307 t/s) — nothing to pick; above 8 users, and for every sampled request, it decodes plainly on the same server.
• Cost: the drafter weights must be in the host's Hugging Face cache before the first serve — once, on the host: huggingface-cli download z-lab/Qwen3.8-27B-DFlash2 (+1 GB/chip; the container mounts that cache and resolves the snapshot offline).

plain — sampled (temperature > 0) workloads at high concurrency, and the exactness baseline.

• Speculation never runs for sampled requests; this profile keeps vLLM's async scheduling: 475 t/s at 32 users (453 spec-decode). 22.4 ms/token at one user, 28.7 at 32 — no drafter, so the same on any text.
• The greedy output of spec-decode equals this profile's, token for token.

p4d4 — 4–32 concurrent users, bursty arrivals, aggregate throughput (its costs — the lone long prompt, sustained heat — are under Known limits).

• 13.8 ms/token at 4 users (245 t/s), 15.4 at 8 (428), 23.2 at 16 (575), 37.5 at 32 (672 t/s) vs spec-decode 18.2 / 22.0 / 26.4 / 28.9 ms and 189 / 308 / 341 / 453 t/s.
• Prompts prefill on their own four chips, so a burst never stalls the decoders: TTFT 290 ms at 8 users, 1.06 s at 32 (spec-decode 419 ms / 5.4 s); 8K prompts at 4 users decode at 16.2 ms/token vs 36.8.
• Formerly published as changh95/qwen3.8-27b-pd4x4-p150x8 (retired 2026-09-30); numbers are that bundle's v13 protocol sweep, reproduced here within 1.5% TPOT.

Benchmarks

All numbers were measured from served instances of this model code on the reference box with tt-inference-server's vllm bench serve (random prompts of exactly ISL tokens, temperature 0, streaming; n = 8 × users at 128/128, 2 × users at 8K–16K, 1 at 32K+). t/s/u = output tokens per second per user (aggregate t/s ÷ concurrent users; at one user it is the user's decode rate) — the headline metric; TPOT = mean time per output token, TTFT = mean time to first token, MAL = the drafter's mean acceptance length (committed tokens per verify step). The TP=8 profiles ran with the clock pinned at 1200 MHz from the served container of the 2026-10-01 image (7dfae48db1ed), whose decode, verify and prefill programs this image runs unchanged — the profile was named hybrid then; the 2026-10-02 images only rename profiles and drop two. This image's container re-measured spec-decode at 128/128 @1 and natural text @1 within the gate's 5 % (TPOT 12.0 / 9.5 ms, 75.7 / 92.9 t/s, TTFT 171 / 171 ms; logs/points_merged/spec-decode/summary.txt, 2026-10-02 08:10 UTC), as did the two earlier 2026-10-02 images (06:24 UTC: 12.2 / 9.5 ms, 74.0 / 92.8 t/s; 01:45 UTC at 128/128 @1/@4, 8K @1 and natural text @1: TPOT 12.0 / 18.1 / 15.3 / 9.5 ms against 12.3 / 18.2 / 15.1 / 9.5). The 2-, 8- and 16-user cells of the first table were measured on 2026-10-01 from a standalone server of the same code and profile env (scripts/tp8_det_serve.sh, no container; its re-runs of the 1- and 4-user points land within 1.6 % t/s of the container's). p4d4 is the thermally gated v13 sweep of the disaggregated stack (2026-09-29, clock under firmware control), whose stack this image's p4d4 profile reproduces byte for byte (env, argv, plugin, runtime; under the profile's former name until this image); the merged container's gate (2026-09-30) re-measured all seven shared points from a cooled box within 1.5 % TPOT (128/128 @1/4/8/32: 12.1 / 13.8 / 15.2 / 37.8 ms; 8K @1/@4: 14.3 / 15.5; 16K @1: 13.7).

t/s/u by concurrent users, 128/128

<!-- t/s/u = t/s / users. Container points (logs/points_merged_pub3_20261001/<profile>/summary.txt, 2026-10-01; the spec-decode profile's dir is hybrid): hybrid t/s 73.3, 189.1, 307.5, 341.4, 453.3 at 1/4/8/16/32 users; plain @1 42.6, @4 143.6, @32 475.3. Standalone-server points (logs/points_card_fill/<profile>/summary.txt + summary.md, 2026-10-01, scripts/tp8_det_serve.sh with the profile's yaml env, pin 1200, same vllm bench serve recipe, n = 8 x users): hybrid @2 121.1/2; plain @2 80.7/2, @8 237.3/8, @16 352.9/16; sanity re-runs @1/@4 within +0.2..+4.2 % t/s of the container points (summary.md). p4d4: logs/sweeps/v13/pivots.md t/s/u table, row 128 / 128 -->

Bold = the best measured profile at that user count.

Long prompts, one user (t/s/u = t/s)

<!-- cells: t/s (= t/s/u at one user) with TTFT s in parentheses. spec-decode: logs/points_merged_pub3_20261001/hybrid/summary.txt (2026-10-01: 8192 42.7 / 1076 ms, 16384 32.4 / 1987, 32768 21.2 / 4002, 131072 5.6 / 20890); plain: logs/points_merged_pub3_20261001/plain/summary.txt (8192 32.9 / 995; the longer points were not measured for plain); p4d4: logs/sweeps/v13/pivots.md t/s/u and TTFT tables, 1-user column (39.7 / 1409, 28.7 / 2727, 15.8 / 6198, 3.9 / 30505) -->

Cells: t/s/u (TTFT s); 128 output tokens. — = not measured.

Natural text (GSM8K chat prompts), t/s/u

<!-- t/s/u = t/s / users. spec-decode: logs/points_merged_pub3_20261001/hybrid/summary.txt (gsm8k 0/128 points, container, 2026-10-01: 93.1 @1, 249.0/4 @4, MAL 4.94 / 5.10); p4d4 (MTP head): logs/served_spec_mtp/summary.txt ('spec_mtp gsm8k c=1 / c=4' rows: 74.40, 265.11/4; acceptance 3.4-3.6 in its SpecDecoding metrics). plain was not measured on these prompts (no drafter: its random-token TPOT 22.4 / 23.8 ms applies) -->

~115–127 input tokens, 128 output tokens. Cells: t/s/u (MAL).

15-minute 4-user soak of spec-decode on this text (1,536 requests, 2026-10-01): 62.5–63.2 t/s/u in every round (250.0–252.8 t/s, 13.7–13.9 ms/token), acceptance 5.22 over 38.6k verify steps, 0 errors, 0 of 8,568 per-chip clock samples below the pinned 1200 MHz (hottest chip 83 °C, started right after the 131K-token prefill).

Latency at the shared points (TPOT ms / TTFT ms)

<!-- spec-decode: logs/points_merged_pub3_20261001/hybrid/summary.txt (2026-10-01); plain: logs/points_merged_pub3_20261001/plain/summary.txt for 128/128 @1 (22.4 / 163), @4 (23.8 / 538), @32 (28.7 / 4967) and 8192/128 @1 (22.8 / 995), logs/points_card_fill/plain/summary.txt for @8 (24.8 / 1164, standalone server, 2026-10-01); p4d4: logs/sweeps/v13/points (make_tables.py rows) -->

Cells: TPOT ms / TTFT ms; bold = best per row; — = not measured.

p4d4 regimes. Its clock is under firmware control (AICLK 1350 MHz nominal, throttled to 800 MHz at a chip's limits): the ≤ 4-user long-prompt points are bursts started with the prefill chips at idle temperature (82 °C, peaking at 83–92 °C within the point), while at 8+ users with 4K+ prompts the prefill queue runs sustained, the four prefill chips throttle and TTFT is the queue (10.8 s at 8K/8 users, 54 s at 8K/32) with TPOT still 15–20 ms. The decode half tracks the box's thermal state too — on a day the stack idled 5 °C warmer one decode chip was throttled to 800 MHz for a third to three-quarters of a 128/128 point at 8–32 users and the step slowed in proportion (16.7 ms at 8 users, 44–53 at 32), while from a cooled box the same points reproduce the table (15.2 / 37.8 ms) — whereas the TP=8 profiles pin the clock at 1200 MHz and were measured in a single thermal regime.

Accuracy and determinism

<!-- spec-decode (the hybrid-named runs of the same code): logs/points_merged_pub4_20261002/hybrid/summary.txt + gsm8k_eval.log (0.835 +- 0.0263, 0 degenerate, 2026-10-02 image 30b0854de23d), det_probe.log / det_long.log (3 of 9 / 1 of 3, md5 831f5e859d88), prefill_probe.log (60 docs x 3: 0 non-identical, 60/60 == the reference boot); this image's chain re-ran det_probe / det_long (logs/points_merged/spec-decode/). plain: TP8_SPEC_RESULTS.md section 3 (0.835, 2026-09-21 image). p4d4: logs/points_merged_pub4_20261002/pd4x4/summary.txt + doc_compare_pd4x4.log (0.82 +- 0.0272, 200/200 vs the 2026-10-01 run), logs/sweeps/v13/gsm8k.log; this image's chain re-ran its det_probe / det_long (logs/points_merged/p4d4/, md5 765879414ca7, = logs/points_merged/pd4x4/ of the 06:24 image). Concurrency paragraph: logs/HYBRID_T_DETERMINISM.md sections 1/3/4/8 (verify logits == plain decode bitwise at every grid row, 184,320 rows; SDPA-decode cores per (row, head) = min(110, 16B)/B = 16/16/16/13/6/3 at B = 1/2/4/8/16/32, order differs past one 256-token K chunk; c8 wording differences 3-10/200, accuracy 0.835 x4; knob cost 15.8 vs 15.1 ms +5 % at 8K, 22.2 vs 16.0 +39 % at 32K, 12.1 vs 12.3 / 9.4 vs 9.5 below 2K), logs/SERVED_DETERMINISM.md (the knob: GSM8K-200 c8 twice 200/200 and 200/200 vs a separate server, 0.835; logs/points_merged_pub4_20261002/deterministic/); logs/PREFILL_DETERMINISM.md (12/720 -> 0/720 in-process, 0/1080 served prefills) -->

All three profiles are lossless with respect to the plain decode and reproducible at one user. The verify step re-scores the drafts with the full model and commits exactly the plain argmax prefix, whichever drafter proposed it — bitwise on device for every verify grid the ladders use (R ≤ 32 rows), and the DFlash2 and MTP bands produce the same logits as each other and as the plain decode at equal width — so GSM8K measures the model, not the drafter, and a request served alone returns the same stream every time. Prefill is bitwise reproducible on every profile: 0 of 720 in-process and 0 of 1,080 served TP=8 prefills differ, and the 60-prompt × 3 first-token probe at one user is identical per prompt to the reference boot. Under concurrency the TP=8 decode is reproducible in content but not always in wording: the SDPA-decode kernel splits each (row, KV head) over min(110, 16·B)/B cores (16 / 16 / 16 / 13 / 6 / 3 at 1 / 2 / 4 / 8 / 16 / 32 rows) and the bf16 flash accumulation order over 256-token K chunks follows that count, so past ~256 tokens of context the logits of the same (prompt, prefix) differ by 1–3 bf16 ulps between row counts and two runs can pick different tokens where the top-2 are within that distance — 3–10 of 200 GSM8K answers at 8 users differ in wording, never in the final answer, and the accuracy is the same (0.835 in every run). p4d4 (TP=4) never ran the prefill path in question and its GSM8K-200 generations are 200/200 identical across boots and images.

How to serve

pip install tt-model-manager                                   # once; needs docker and the Tenstorrent drivers on a P150x8
tt-model pull  changh95/qwen3.8-27b-p150x8 --with-weights       # optional: image + weights ahead of time (spec-decode: the DFlash2 drafter too, see Use when)

tt-model serve changh95/qwen3.8-27b-p150x8                     # spec-decode (default): TP=8 + DFlash2/MTP speculative decoding
tt-model serve changh95/qwen3.8-27b-p150x8 --profile plain     # TP=8, no drafter
tt-model serve changh95/qwen3.8-27b-p150x8 --profile p4d4      # 4 prefill chips + 4 decode chips
tt-model stop  changh95/qwen3.8-27b-p150x8

--profile spec-decode names the default explicitly. The first serve pulls the image and the model weights (Qwen/Qwen3.8-27B; they are not in the image) into the host's Hugging Face cache unless pull --with-weights did it beforehand. The first boot of a TP=8 profile converts the weights and compiles the 8-chip kernels (11 min on the reference box; 3–4 min warm; spec-decode and plain share the cache); p4d4 converts a separate 4-chip weight cache (18 min cold, ~6 min warm). The server is ready when /health returns 200 (the container logs Application startup complete); GET / reports the mode, chips and drafter.

Environment and provenance

• tt-metal 40e604f2b0b (branch qwen38-pd-disagg, packaged from a pinned detached worktree; the shipped model code is code/ in this repo), vllm-tt-plugin 4ab74cb54ebe3133f689032c6d73bdaf5c7f2c2c (branch changh95/pd-mooncake-connector), vLLM 0.26.0 (empty-target wheel under wheels/), Mooncake transfer engine 0.3.13.post1 (p4d4), Python 3.12 / Ubuntu 22.04; image tt-model/qwen3.8-27b-p150x8:03270fc11d14, tt-model-manager 0.1.0, packaged 2026-10-02. What changed vs the 2026-10-02 06:24 UTC image (2ee500a47257, the same tt-metal 40e604f2b0b): profile pd4x4 renamed p4d4 — name only (the same env, the same QWEN36_SERVE_MODE=pd supervisor path; nothing in the model code or the plugin reads the profile name); that image had itself reduced the profiles from five to three (hybrid → spec-decode, spec-mtp folded into it, deterministic → the serve-env note under Known limits). Model code, wheels and lock are unchanged, so the decode, verify and prefill programs and every number above are those of the previous images (this image's container re-measured spec-decode at 128/128 @1 and natural text @1 within 5 % and reproduced every greedy answer of the previous images: the 200-token answers of spec-decode and p4d4, md5 831f5e859d88 / 765879414ca7, and plain's proof completion byte for byte).
• Weights: Qwen/Qwen3.8-27B at 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 (pulled by tt-model). Drafter (spec-decode): z-lab/Qwen3.8-27B-DFlash2 at 50307d4c4cde6860d4eee73e2547cd786fe8e8a4 — not pulled by tt-model; the container mounts the host cache at /hf and resolves the snapshot offline before the engine starts (a missing snapshot is a start-up error naming the download command; DFLASH2_MODEL=<dir|hub id> overrides).
• One served process for all profiles: the supervisor models/demos/blackhole/qwen36/server/pd_app.py (kind tt-dit-server). QWEN36_SERVE_MODE=mono (spec-decode / plain) launches one TP=8 vllm serve on all eight chips with the vllm-plugin launcher's exact command (--max-num-seqs 32 --block-size 64 --max-num-batched-tokens 262144, the TT additional config, and for spec-decode --speculative-config '{"method":"mtp","num_speculative_tokens":7}' --no-async-scheduling) and forwards every request to it; QWEN36_SERVE_MODE=pd (p4d4) launches the P and D halves with the Mooncake connector and the fan-out proxy, unchanged from the retired changh95/qwen3.8-27b-pd4x4-p150x8 v13.
• TP=8 profile env: QWEN36_MAX_TOKENS_ALL_USERS=525312 QWEN36_FORCE_AICLK_MHZ=1200 QWEN36_DECODE_FUSED_AR=0 QWEN36_DECODE_CHAIN=1 QWEN36_VERIFY_CHAIN=1 QWEN36_DECODE_DRAM_SHARDED=down QWEN36_GDN_HIST_DEVICE_PACK=1 QWEN36_PLAIN_GDN_SLOT_DEVICE_COPY_FORCE=1, plus QWEN36_SPEC_MTP=1 QWEN36_SPEC_ENGINE=mono QWEN36_SPEC_DRAFTER=hybrid QWEN36_SPEC_K=7 QWEN36_SPEC_DFLASH2_MAX_CTX=12288 (spec-decode; the model code still calls its two-drafter mode hybrid) / QWEN36_SPEC_MTP=0 (plain). p4d4 env: QWEN36_PD_PREFILL_CHIPS=0,1,6,7 QWEN36_PD_DECODE_CHIPS=2,3,4,5 QWEN36_PD_PREFILL_MAX_NUM_SEQS=8 QWEN36_PD_DECODE_MAX_NUM_SEQS=32 QWEN36_PD_PREFILL_MAX_TOKENS=525312 QWEN36_PD_DECODE_MAX_TOKENS=1052672 QWEN36_PD_DECODE_BUCKETING=1 QWEN36_SPEC_MTP=1 QWEN36_SPEC_DRAFTER=mtp QWEN36_SPEC_K=3 QWEN36_SPEC_ALLOW_FRACTURED=0.
• Evidence: models/demos/blackhole/qwen36/tests/TP8_SPEC_RESULTS.md (the TP=8 speculative port and its container gates), DFLASH2_RESULTS.md (the block drafter, the 12K context rule, the near-tie analysis) and MTP_SPEC_RESULTS.md in the tt-metal branch; the P/D v13 sweep protocol in the plugin's docs/SPECULATIVE.md.

Known limits

• TP=8 profiles prefill serially with decode. Prompts are prefilled in 2048-token chunks on the same eight chips that decode, so many simultaneous arrivals dominate TTFT (5.4 s at 32 users of 128/128; 2.4 s at 4 users of 8K) and inflate TPOT (36.8 ms at 4 users of 8K). p4d4 exists to remove exactly that coupling.
• p4d4 pays on the lone prompt and under sustained heat. The first token of a single long prompt is 1.3–1.5× slower than TP=8 (1.4 s at 8K, 2.7 s at 16K, 30.5 s at 131K), the 32-user step is 37.5 vs 28.9 ms, and with 8+ users of 4K+ prompts the four prefill chips throttle to 800–875 MHz, making TTFT the prefill queue (54 s at 8K/32 users); the decode half's step tracks the box's thermal state too (regimes above).
• Speculation is for greedy requests, up to 8 users (16 in p4d4). Sampled requests and higher concurrency decode plainly; spec-decode gives up ~3 % of aggregate throughput there to --no-async-scheduling, uses the block drafter only while the longest live context is ≤ 12,288 tokens, and needs the second weights repo (+1 GB/chip).
• Run-to-run wording can differ under concurrency (Accuracy and determinism); bitwise run-to-run reproducibility at any concurrency is available by setting QWEN36_SDPA_DEC_MAX_CORES_PER_HEAD=1 in the serve environment (one core per row and KV head on the decode and verify steps: ~0 cost below 2K tokens of context, +5 % TPOT at 8K, +39 % at 32K; GSM8K-200 at 8 users 200/200 identical twice and against a separate server with this model code) — an environment note, not a profile.
• A profile switch is a restart. The three profiles share one image but not one process: tt-model stop then serve --profile; the TP=8 and p4d4 profiles keep separate kernel and weight caches (1×8 vs 1×4 mesh), so the first switch is another cold boot (11–18 min).