Downloads · 30 days
0
tt-hous/gemma-4-26B-A4B-it_p150_p150x2_p150x4
gemma-4-26B-A4B-it_p150_p150x2_p150x4 is a machine learning model from tt-hous. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Gemma 4 26B A4B (instruction tuned) served on Tenstorrent Blackhole through vLLM, with Gemma 4 tool-call and reasoning parsers. Profiles p150, p150x2, p150x4. Sources: model code from the tt-metal fork housTT/tt-metal…
Downloads · 30 days
0
Access
Public
Updated Oct 23, 2026
Repo size
1.4 GB
Likes
0
Public
Click a slice to open those files.
Other1.4 GB · 99%
From the Hugging Face model README
Gemma 4 26B A4B (instruction tuned) served on Tenstorrent Blackhole through vLLM, with Gemma 4 tool-call and reasoning parsers. Profiles p150, p150x2, p150x4. Sources: model code from the tt-metal fork housTT/tt-metal, branch hous/gemma-4-26b-a4b-it (the commit is in the Provenance table below; the shipped code/ paths are byte-identical to that commit, the sole changes over upstream tt-metal are the models/autoports/google_gemma_4_26b_a4b_it tree and its tests); the vLLM wheel is the author's empty-target build of the tenstorrent/vllm fork at c87df030e, which is also on the housTT/vllm dev branch; the Tenstorrent vLLM plugin is plugins/vllm-tt-plugin of that same repository at the commit linked below. Packaged with the tt-model-manager fork housTT/tt-model-manager, branch fix/v5-serving-capabilities.
Runs on p150 or p150x2 or p150x4 — see the serve profiles below.
Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).
tt-model pull tt-hous/gemma-4-26B-A4B-it_p150_p150x2_p150x4 --with-weights
tt-model serve tt-hous/gemma-4-26B-A4B-it_p150_p150x2_p150x4
pull --with-weights downloads the Docker image and the google/gemma-4-26B-A4B-it weights at 4d7ae4984b7db7de8f8457170b3f1a419ee76d52 (into your HF cache; they are not in the image). serve starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.
Validated on a TT-QuietBox 2 (2x Blackhole P300C, 4 chips) with 1, 2 and 4 chips opened for the p150 / p150x2 / p150x4 profiles. Advertised context is served end to end on every profile. Automatic prefix caching is on: a request that extends an earlier one (multi-turn chat, agent loops) prefills only the new tokens.
One image serves every profile below; pick one with --profile.
| profile | hardware | mesh | max_num_seqs | max_model_len |
|---|---|---|---|---|
p150 | p150 | P150 | 32 | 131072 |
p150x2 (default) | p150x2 | P150x2 | 32 | 262144 |
p150x4 | p150x4 | P150x4 | 32 | 262144 |
All evaluations ran through tt-inference-server against this bundle's serving stack on a TT-QuietBox 2 (P150x2 profile unless noted; source epoch tt-metal 42444b8ec, vLLM fork c87df030e). Scores are on 20% subsets of each task (the release workflow's --limit 0.2), so they are not directly comparable with published full-set figures. No reference (GPU/CPU) run of the same subsets was available, so these are absolute TT numbers without a delta. Recipes are the release workflow's pinned ones.
| task | subset | recipe | result |
|---|---|---|---|
Meta IFEval (meta_ifeval) | 109 prompts | greedy, 0-shot, 2,048 new tokens | 88.74% (original scorer) / 89.25% (corrected scorer; same outputs) — 109/109 completed |
Meta GPQA CoT (meta_gpqa_cot) | 90 questions | greedy, 0-shot, seed 42, 2,048 new tokens | 57/90 exact (63.3%) — 90/90 completed; one document (58) exhausts its 2,048-token budget without an answer (see note) |
| GPQA Diamond, R1 recipe | 40 questions | temperature 1.0, top_p 0.95, top_k 20, 32,768 new tokens | 28/40 exact (70.0%) — 40/40 completed; semantic cycling observed in 2 of the 40 |
| SWE-bench Verified | 5 tasks (matplotlib-25332, sympy-13551, scikit-learn-14629, django-11299, astropy-14096) | mini-swe-agent, temperature 1.0, top_p 0.95, top_k 20, 32,768 max tokens, 250 steps; official swebench 2.1.0 grading | 1/5 resolved (matplotlib). sympy: fixes the target, regresses 2 tests; scikit-learn: clean patch, target not fixed; django: fix visible in the trajectory, empty submitted patch; astropy: step limit after one command repeated 229 times |
| Terminal-Bench 2 | 5 tasks | harbor, same recipe | 2/5 resolved; password-recovery completes with reward 0; caffe-cifar-10 exceeds the 3 h task timeout twice (slow build, not a loop); one further task reward 0 |
| Structured output (guided JSON) | 100 requests per row, 128-token row budget | greedy | json schema 100/100 valid; json-unique (optional UUID field) 0/100 — every response hits the 128-token budget mid-field; xgrammar bench 89/100 (P150x2) / 87/100 (P150x4) — the failures are correct prefixes cut by the budget |
Known quality finding: Meta GPQA document 58 (and, under the R1 recipe, a minority of long generations) enters a repetition cycle and never emits an answer. Every Tenstorrent-side hypothesis was tested and refuted (layer-0 PCC 0.999949 against an independent implementation; identical behaviour on a single chip with no tensor parallelism), so the remaining candidates are cumulative BF16 drift or the checkpoint itself; a reference run would settle it. Tool calling and reasoning parsers are verified live (streaming and non-streaming, n=2, tool-result continuation, cancellation) on all three profiles.
Revision check (2026-09-18, image 3ad9f882120e, tt-metal a3a238e14, batched decode MoE): the same Meta IFEval 109-prompt subset re-run on this image gives a mean of the four IFEval metrics of 93.33% on p150x4 (prompt-level strict 89.91%), 91.93% on p150x2 (strict 88.99%) and 91.33% on p150 (strict 88.07%), against 90.73% (strict 87.16%) for the previous revision served on p150x4. Outputs are not token-identical to the previous revision (BF16 last-bit differences in the batched expert reduction under greedy decoding).
Revision check (2026-09-24, image 8be0c91e6c2a, tt-metal cb1a0351708, plugin f96662a7d; 131k p150 context, prefix caching, top_k 64 on device, KV admission reserve): the same Meta IFEval 109-prompt subset re-run from this image gives exactly the 2026-09-18 figures on every profile and metric (mean 93.33% on p150x4, 91.93% on p150x2, 91.33% on p150; prompt-level strict 89.91 / 88.99 / 88.07%). Feature checks, parser smokes and a 130,969-token needle-recall probe on p150 pass on this image.
Full-set and thinking-mode results on this image (tt-metal cb1a0351708, plugin f96662a7d, image 80602da4b0d3); the 2026-10-08 rows were measured on a second TT-QuietBox 2 (qb2-120-p11t01, firmware bundle 19.15.0) from the published image:
| task | set | profile, date | recipe | result |
|---|---|---|---|---|
GPQA Diamond (r1_gpqa_diamond) | all 198 questions | p150x4, 2026-09-25 to 29 | thinking on (server default), temperature 1.0, top_p 0.95, top_k 64 on the device sampler, 98,304 new tokens, 6 concurrent | 81.31% (161/198, standard error 2.77); Google reports 82.3% under its own harness; 18 responses ran to the budget without an answer |
Meta IFEval (meta_ifeval) | all 541 prompts | p150, 2026-10-08 | greedy, 0-shot, 1,280 new tokens | prompt-level strict 88.91%, loose 90.76%; instruction-level strict 92.21%, loose 93.76% (mean 91.41%). The 109-prompt control reproduced the 2026-09-24 figures exactly |
Meta GPQA CoT (meta_gpqa_cot) | all 448 questions (the GPQA main set; the 90-question subset above is drawn from it, so neither is GPQA Diamond) | p150, 2026-10-08 | greedy, 0-shot, seed 42, 2,048 new tokens, thought channel open | 64.06% (standard error 2.27) |
| MMLU-Pro control | 2% per subject, 247 questions, 5-shot, 512 new tokens | p150, 2026-10-08 | greedy | 33.6% (standard error 2.97). A budget-capped control only; not comparable with the published 82.6% |
| Needle recall | 98,304-token prompt, fact at 50% depth | p150, 2026-10-08 | greedy | recalled, 197 s end to end |
| Prefix cache | 16,384-token shared prefix | p150, 2026-10-08 | greedy | TTFT 24.8 s cold, 179 ms hot (16,384 cached tokens) |
| Sampling recipe cost | 256 new tokens, one user | p150, 2026-10-08 | model-card recipe vs greedy | TPOT 27.9 ms vs 27.5 ms, both on the device sampler; an explicit seed takes the host path at 65.4 ms |
| Meta IFEval control after the prefill change | 109-prompt subset | p150, 2026-10-10 (tt-metal 522cd48fe0) | greedy, 0-shot, 1,280 new tokens | prompt-level strict 88.07%, loose 90.83%; instruction-level strict 92.09%, loose 94.35% (mean 91.33%): identical on all four metrics to the 2026-10-08 control. Last-token logits of a 4,096- and a 32,768-token prefill are bit-identical to the previous revision |
| Meta IFEval after the attention chunk change | 109-prompt subset and all 541 prompts | p150, 2026-10-10 (tt-metal e5a5653720, image 2f6df868eea0) | greedy, 0-shot, 1,280 new tokens | 109 prompts: prompt-level strict 88.07%, loose 90.83%; instruction-level strict 92.09%, loose 93.79% (mean 91.19%; the earlier control was 88.07 / 90.83 / 92.09 / 94.35). All 541 prompts: prompt-level strict 88.17%, loose 89.65%; instruction-level strict 91.61%, loose 92.81% (mean 90.56%, against 88.91 / 90.76 / 92.21 / 93.76, mean 91.41%, on the 2026-10-08 image). Every full-set metric is lower by 0.6 to 1.1 points, each within one standard error; the 128-row chunks change the order of the online softmax, so outputs are not token-identical to the previous revision. Set GEMMA4_OPT_PREFILL_SDPA_Q_CHUNK=32 and _K_CHUNK=32 in the profile env to restore the previous attention numerics |
Sampling note for evaluators: this bundle's on-device sampler covers top_k <= 64, so the checkpoint's generation_config.json recipe (temperature 1.0, top_p 0.95, top_k 64), which vLLM applies to requests that set no sampling parameters, runs on the device path at the greedy rate. Requests with top_k above 64, a seed, penalties or logprobs fall to host sampling (about 2.3x slower per token at one user). The recipes above stay on the device path.
Closed-loop latency sweep of every profile. p150x4 and p150x2 measured 2026-09-24 on a TT-QuietBox 2 (qb2-120-p11t02, firmware bundle 19.13.1); p150 re-measured 2026-10-10 on a second TT-QuietBox 2 (qb2-120-p11t01, firmware bundle 19.15.0) from this image. Each measured on a TT-QuietBox 2 (2x Blackhole P300C, 4 chips) with 1, 2 and 4 chips opened for p150, p150x2 and p150x4, served from this image (2f6df868eea0, tt-metal e5a565372) with tt-model serve. Each row holds Users concurrent streams open until N prompts finish (N = 4 x users up to 4k input, 2 x users up to 16k, users beyond). Prompts are random token ids of exactly ISL tokens; output is pinned to OSL tokens with ignore_eos. TTFT is the time to the first streamed token; TPOT = (E2EL - TTFT) / (OSL - 1); values are means over the row. One unrecorded warm-up request precedes each input length. Sampling is greedy (temperature 0), which stays on the device sampler; requests with top_k unset or above 32 fall to host sampling and run about 4x slower per token. A row whose mean E2EL passes 300 s is shown and fences larger user counts at that input length and beyond; rows predicted to pass 300 s from the single-user prefill rate are skipped the same way. Characterization only, no configured targets.
This revision (2026-10-10) changes the single-chip prefill in two places: each sparse expert matmul now covers 128-token groups instead of 32-token groups (expert weights re-read 2.8x less often, two no-op layout transposes per group gone), and the prefill attention kernel runs 128-row query and key chunks instead of 32 (the online softmax is accumulated in a different order, so outputs are not token-identical to the previous revision; the IFEval control is unchanged). Single-user prefill on p150 rises from about 610 to about 830 tokens/s at 2k to 8k input and from 573 to 761 tokens/s at 32k input (TTFT 57 s to 43 s); decode is unchanged; last-token logits are bit-identical to the previous revision. The p150x2 and p150x4 tables below are the 2026-09-24 measurements, which this revision does not change (those profiles keep 32-token groups).
The previous revision raised the single-chip context to 131,072 tokens (the KV pool is sized from the DRAM left after the weights load), turns on automatic prefix caching on every profile (a request that extends an earlier one prefills only its new tokens; the sweep's random prompts never hit the cache, so these rows show the uncached cost), keeps the scheduler from filling the KV pool to the brim (an admission reserve avoids preempt-and-replay cycles at high concurrency), and samples the checkpoint's default recipe (top_k 64) on the device. Multi-user decode on p150 improves further; single-user latency is unchanged; generated tokens are not token-identical to the previous revision.
Column key: ISL/OSL = input/output tokens; Users = concurrent streams; N = prompts in the row; TTFT = time to first token (ms); TPOT = time per output token (ms); Decode = output tokens/s per user; Output = output tokens/s across users; E2EL = end-to-end latency (s).
p150x4 — (1, 4) mesh, 262,144-token context; 30 of 42 planned rows measured| ISL | OSL | Users | N | TTFT (ms) | TPOT (ms) | Decode (tok/s/user) | Output (tok/s) | E2EL (s) | Status |
|---|---|---|---|---|---|---|---|---|---|
| 128 | 128 | 1 | 4 | 166 | 20.3 | 49.3 | 47 | 2.74 | ok |
| 128 | 128 | 2 | 8 | 258 | 45.8 | 21.8 | 42 | 6.07 | ok |
| 128 | 128 | 4 | 16 | 509 | 47.7 | 21.0 | 78 | 6.56 | ok |
| 128 | 128 | 8 | 32 | 1,099 | 51.8 | 19.4 | 133 | 7.68 | ok |
| 128 | 128 | 16 | 64 | 2,334 | 55.4 | 18.2 | 219 | 9.36 | ok |
| 128 | 128 | 32 | 128 | 3,638 | 69.1 | 14.8 | 330 | 12.42 | ok |
| 1,024 | 256 | 1 | 4 | 1,062 | 21.0 | 47.6 | 40 | 6.42 | ok |
| 1,024 | 256 | 2 | 8 | 1,609 | 49.2 | 20.4 | 36 | 14.14 | ok |
| 1,024 | 256 | 4 | 16 | 2,924 | 54.5 | 18.5 | 61 | 16.82 | ok |
| 1,024 | 256 | 8 | 32 | 5,190 | 65.1 | 15.7 | 94 | 21.78 | ok |
| 1,024 | 256 | 16 | 64 | 9,517 | 85.4 | 12.4 | 131 | 31.29 | ok |
| 1,024 | 256 | 32 | 128 | 17,982 | 121.9 | 9.2 | 167 | 49.07 | ok |
| 4,096 | 256 | 1 | 4 | 4,302 | 21.7 | 46.1 | 26 | 9.83 | ok |
| 4,096 | 256 | 2 | 8 | 6,452 | 57.3 | 17.8 | 24 | 21.08 | ok |
| 4,096 | 256 | 4 | 16 | 10,769 | 74.9 | 14.3 | 34 | 29.87 | ok |
| 4,096 | 256 | 8 | 32 | 19,325 | 111.2 | 10.4 | 43 | 47.68 | ok |
| 4,096 | 256 | 16 | 64 | 36,513 | 185.3 | 6.8 | 49 | 83.75 | ok |
| 4,096 | 256 | 32 | 128 | 61,298 | 361.1 | 3.4 | 53 | 153.38 | ok |
| 16,384 | 256 | 1 | 2 | 17,711 | 22.2 | 45.0 | 11 | 23.38 | ok |
| 16,384 | 256 | 2 | 4 | 26,621 | 90.0 | 13.1 | 10 | 49.58 | ok |
| 16,384 | 256 | 4 | 8 | 44,216 | 160.2 | 8.6 | 12 | 85.08 | ok |
| 16,384 | 256 | 8 | 16 | 79,824 | 301.4 | 5.4 | 13 | 156.69 | ok |
| 16,384 | 256 | 16 | 32 | 150,838 | 580.0 | 3.1 | 14 | 298.74 | ok |
| 16,384 | 256 | 32 | skipped (E2EL > 300 s) (predicted E2EL 572 s at 32 users) | ||||||
| 32,768 | 256 | 1 | 1 | 36,270 | 22.5 | 44.5 | 6 | 42.00 | ok |
| 32,768 | 256 | 2 | 2 | 54,610 | 134.1 | 10.4 | 6 | 88.80 | ok |
| 32,768 | 256 | 4 | 4 | 91,088 | 277.3 | 6.3 | 6 | 161.80 | ok |
| 32,768 | 256 | 8 | 8 | 165,936 | 567.1 | 3.7 | 7 | 310.55 | E2EL > 300 s |
| 32,768 | 256 | 16-32 | skipped (E2EL > 300 s) (E2EL at 8 users was 311 s at ISL 32768) | ||||||
| 65,536 | 256 | 1 | 1 | 76,968 | 23.8 | 42.0 | 3 | 83.03 | ok |
| 65,536 | 256 | 2 | 2 | 115,324 | 226.7 | 7.8 | 3 | 173.13 | ok |
| 65,536 | 256 | 4-32 | skipped (E2EL > 300 s) (predicted E2EL 314 s at 4 users) | ||||||
| 131,072 | 256 | 1 | 1 | 169,011 | 25.3 | 39.5 | 1 | 175.46 | ok |
| 131,072 | 256 | 2-32 | skipped (E2EL > 300 s) (predicted E2EL 345 s at 2 users) |
p150x2 — (1, 2) mesh, 262,144-token context; 28 of 42 planned rows measured| ISL | OSL | Users | N | TTFT (ms) | TPOT (ms) | Decode (tok/s/user) | Output (tok/s) | E2EL (s) | Status |
|---|---|---|---|---|---|---|---|---|---|
| 128 | 128 | 1 | 4 | 167 | 21.6 | 46.2 | 44 | 2.92 | ok |
| 128 | 128 | 2 | 8 | 251 | 65.9 | 15.2 | 30 | 8.62 | ok |
| 128 | 128 | 4 | 16 | 538 | 66.9 | 15.0 | 57 | 9.03 | ok |
| 128 | 128 | 8 | 32 | 1,146 | 67.4 | 14.9 | 105 | 9.71 | ok |
| 128 | 128 | 16 | 64 | 2,351 | 67.8 | 14.8 | 187 | 10.96 | ok |
| 128 | 128 | 32 | 128 | 3,706 | 74.2 | 13.7 | 312 | 13.13 | ok |
| 1,024 | 256 | 1 | 4 | 1,100 | 22.5 | 44.5 | 37 | 6.83 | ok |
| 1,024 | 256 | 2 | 8 | 1,933 | 72.3 | 13.8 | 25 | 20.38 | ok |
| 1,024 | 256 | 4 | 16 | 3,038 | 77.1 | 13.0 | 45 | 22.69 | ok |
| 1,024 | 256 | 8 | 32 | 5,378 | 85.6 | 11.8 | 75 | 27.20 | ok |
| 1,024 | 256 | 16 | 64 | 9,820 | 102.1 | 10.2 | 114 | 35.85 | ok |
| 1,024 | 256 | 32 | 128 | 18,645 | 134.6 | 8.2 | 155 | 52.97 | ok |
| 4,096 | 256 | 1 | 4 | 4,464 | 23.1 | 43.3 | 25 | 10.36 | ok |
| 4,096 | 256 | 2 | 8 | 6,596 | 81.1 | 12.5 | 19 | 27.27 | ok |
| 4,096 | 256 | 4 | 16 | 11,042 | 108.2 | 9.8 | 27 | 38.64 | ok |
| 4,096 | 256 | 8 | 32 | 23,565 | 189.3 | 6.0 | 29 | 71.84 | ok |
| 4,096 | 256 | 16 | 64 | 38,705 | 200.5 | 6.1 | 46 | 89.83 | ok |
| 4,096 | 256 | 32 | 128 | 63,451 | 379.5 | 3.2 | 51 | 160.23 | ok |
| 16,384 | 256 | 1 | 2 | 18,401 | 23.4 | 42.7 | 11 | 24.37 | ok |
| 16,384 | 256 | 2 | 4 | 27,759 | 125.3 | 8.7 | 9 | 59.71 | ok |
| 16,384 | 256 | 4 | 8 | 46,016 | 186.4 | 6.8 | 11 | 93.54 | ok |
| 16,384 | 256 | 8 | 16 | 82,603 | 330.4 | 4.5 | 12 | 166.84 | ok |
| 16,384 | 256 | 16-32 | skipped (E2EL > 300 s) (predicted E2EL 300 s at 16 users) | ||||||
| 32,768 | 256 | 1 | 1 | 39,345 | 23.8 | 42.1 | 6 | 45.40 | ok |
| 32,768 | 256 | 2 | 2 | 57,932 | 160.6 | 8.0 | 5 | 98.88 | ok |
| 32,768 | 256 | 4 | 4 | 97,111 | 317.0 | 5.0 | 6 | 177.93 | ok |
| 32,768 | 256 | 8-32 | skipped (E2EL > 300 s) (predicted E2EL 321 s at 8 users) | ||||||
| 65,536 | 256 | 1 | 1 | 84,812 | 24.6 | 40.6 | 3 | 91.09 | ok |
| 65,536 | 256 | 2 | 2 | 126,796 | 265.2 | 6.2 | 3 | 194.43 | ok |
| 65,536 | 256 | 4-32 | skipped (E2EL > 300 s) (predicted E2EL 346 s at 4 users) | ||||||
| 131,072 | 256 | 1 | 1 | 198,144 | 26.6 | 37.6 | 1 | 204.93 | ok |
| 131,072 | 256 | 2-32 | skipped (E2EL > 300 s) (predicted E2EL 403 s at 2 users) |
p150 — single chip, 131,072-token context; 28 of 36 planned rows measured| ISL | OSL | Users | N | TTFT (ms) | TPOT (ms) | Decode (tok/s/user) | Output (tok/s) | E2EL (s) | Status |
|---|---|---|---|---|---|---|---|---|---|
| 128 | 128 | 1 | 4 | 175 | 27.1 | 36.9 | 35 | 3.62 | ok |
| 128 | 128 | 2 | 8 | 263 | 61.5 | 16.3 | 32 | 8.07 | ok |
| 128 | 128 | 4 | 16 | 552 | 64.1 | 15.6 | 59 | 8.69 | ok |
| 128 | 128 | 8 | 32 | 1,191 | 71.8 | 14.1 | 99 | 10.31 | ok |
| 128 | 128 | 16 | 64 | 2,521 | 72.6 | 13.8 | 174 | 11.74 | ok |
| 128 | 128 | 32 | 128 | 3,921 | 89.7 | 11.3 | 267 | 15.32 | ok |
| 1,024 | 256 | 1 | 4 | 1,101 | 27.8 | 35.9 | 31 | 8.20 | ok |
| 1,024 | 256 | 2 | 8 | 1,660 | 75.5 | 13.3 | 24 | 20.90 | ok |
| 1,024 | 256 | 4 | 16 | 3,034 | 85.6 | 11.8 | 41 | 24.87 | ok |
| 1,024 | 256 | 8 | 32 | 5,351 | 98.7 | 10.3 | 67 | 30.53 | ok |
| 1,024 | 256 | 16 | 64 | 12,704 | 106.5 | 9.7 | 92 | 39.85 | ok |
| 1,024 | 256 | 32 | 128 | 48,481 | 104.7 | 9.8 | 98 | 75.18 | ok |
| 4,096 | 256 | 1 | 4 | 4,375 | 28.4 | 35.2 | 22 | 11.62 | ok |
| 4,096 | 256 | 2 | 8 | 6,531 | 84.5 | 12.0 | 18 | 28.09 | ok |
| 4,096 | 256 | 4 | 16 | 10,893 | 103.5 | 10.0 | 27 | 37.28 | ok |
| 4,096 | 256 | 8 | 32 | 21,122 | 135.2 | 8.1 | 37 | 55.60 | ok |
| 4,096 | 256 | 16 | 64 | 51,965 | 241.6 | 4.4 | 35 | 113.57 | ok |
| 4,096 | 256 | 32 | 128 | 157,102 | 262.1 | 4.0 | 34 | 223.93 | ok |
| 16,384 | 256 | 1 | 2 | 18,006 | 29.1 | 34.4 | 10 | 25.42 | ok |
| 16,384 | 256 | 2 | 4 | 26,968 | 122.4 | 8.9 | 9 | 58.18 | ok |
| 16,384 | 256 | 4 | 8 | 45,022 | 203.6 | 5.9 | 11 | 96.94 | ok |
| 16,384 | 256 | 8 | 16 | 88,687 | 275.2 | 4.3 | 11 | 158.87 | ok |
| 16,384 | 256 | 16 | 32 | 224,625 | 294.6 | 3.9 | 11 | 299.76 | ok |
| 16,384 | 256 | 32 | skipped (E2EL > 300 s) (predicted E2EL 584 s at 32 users) | ||||||
| 32,768 | 256 | 1 | 1 | 37,205 | 30.0 | 33.3 | 6 | 44.85 | ok |
| 32,768 | 256 | 2 | 2 | 55,937 | 174.6 | 6.9 | 5 | 100.47 | ok |
| 32,768 | 256 | 4 | 4 | 95,494 | 322.8 | 5.0 | 6 | 177.80 | ok |
| 32,768 | 256 | 8-32 | skipped (E2EL > 300 s) (predicted E2EL 305 s at 8 users) | ||||||
| 65,536 | 256 | 1 | 1 | 78,899 | 31.7 | 31.5 | 3 | 86.99 | ok |
| 65,536 | 256 | 2 | 2 | 119,163 | 287.2 | 4.9 | 3 | 192.41 | ok |
| 65,536 | 256 | 4-32 | skipped (E2EL > 300 s) (predicted E2EL 324 s at 4 users) |
The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:
| component | built from |
|---|---|
| tt-metal | 4f3c2b13fa9ef41b8d00d8be12db6b0c51a41f35 |
| vLLM | vllm-0.1.dev14200+gc87df030e.empty.gemma4.empty-py3-none-any.whl — a wheel the author built |
| vllm-tt-plugin | f96662a7d239c077e1435b658c0e7d316f9c4799 |
code/ digest | 03d83b851962b859 (sha256, first 16 hex digits) |
| built | 2026-10-23T19:32:37+00:00 by tt-model 0.1.0 |