gemma-4-26b-a4b-it-QB2

Gemma-4-26B-A4B-it — Tenstorrent Blackhole (vLLM bundle)

A tt-kernel vLLM bundle that runs google/gemma-4-26B-A4B-it — a Mixture-of-Experts model (128 experts, top-8, ~4B active params/token) — on 2× Tenstorrent Blackhole p300c boards (4 ASICs ≈ 4× p150a) via the Tenstorrent vLLM plugin. Ships no kernel cache and no weights — only the vLLM registration, the bounded sliding-window KV adapter, and a launch recipe; weights are pulled from the base repo and kernels JIT-compile at first-run warmup.

Serve

tt-kernel serve mando2222/gemma-4-26b-a4b-it-QB2

OpenAI-compatible server on :8000, model id google/gemma-4-26B-A4B-it. Structured tool calling + reasoning channel enabled; single slot (max_num_seqs=1).

Context length — validated 256K

This bundle serves the model's full native 256K (262144) context, and long-range recall is validated there — not just "it fits in DRAM". The unlock is whole-body chunked prefill (GEMMA4_CHUNKED_PREFILL=1, default on): the prompt is prefilled in bounded 16K-token chunks, which (a) keeps every chunk inside a reliable prefill regime and (b) holds peak prefill memory constant regardless of context length, so 256K completes with no OOM. Combined with the bounded sliding-window serving-KV (only the 5 global layers scale with context; the 25 sliding_window=1024 layers use a fixed 1024-token ring), the full 262K-token KV cache fits at max_num_seqs=1.

Measured prose needle-in-haystack recall (5 depths, 10/25/50/75/90%):

At 256K a fact placed deep in the middle is occasionally not retrieved (a long-context "lost-in-the-middle" effect, not corruption — output stays fluent and correct-form). Endpoints and most depths are reliable. If you need strict 5/5 at every depth, cap at 128K.

Configurable knobs (launch env / vllm_bundle/vllm_metadata.json)

See vllm_bundle/README.md for the full knob reference and trade-offs.

Exact stack this was built and validated on

Model / precision

MoE Gemma4ForConditionalGeneration (text-only), 30 layers, 128 experts / top-8, GQA 16/8, head-dim 256, vocab 262144, hybrid attention (5:1 sliding:global, sliding_window=1024), dual RoPE, final_logit_softcapping=30. TP=4 + EP=4 (each chip owns 32 of 128 experts). Precision: BFP8 projection/expert weights + LoFi dense-decode fidelity, BFP8 paged KV, BF16 activations/CCL; on-device greedy sampling; FABRIC_1D; metal-trace decode + gathered chunked eager prefill.

Measured (2× p300c / 4 ASICs)

• Decode: ~30–33 tok/s/user (single user), unaffected by context length.
• Prefill (TTFT): gathered + chunked path — 128K ~41 s, 256K ~95 s. Peak prefill memory is constant in context length (chunk-bounded), so long prompts scale in time but not memory. ~12× faster than the old single-chunk floor.
• Accuracy: AIME24 top-5 1.00 / top-100 1.00; GPQA-Diamond 80% (10-sample); gathered prefill PCC 0.9995 vs the all-ones reference.
• Long-context recall: prose needle 128K 5/5, 256K 4/5 (see above).
• Tool calling / reasoning: structured tool_calls via the gemma4 parser; reasoning routed to reasoning_content with clean content.

Consumer prerequisites

1. 2× Blackhole p300c boards (4 ASICs ≈ 4× p150a), 1×4 mesh, iommu=pt + 1 GB hugepages. A literal 2×2 mesh does not work (fabric routing).
2. tt-metal built with the autoport importable at models.autoports.google_gemma_4_26b_a4b_it (the adapter subclasses its Gemma4ForCausalLM, uses tt/experts/prefill_gathered.py, and the chunked-prefill correctness path in the shared tt_transformers generator + the autoport attention). Currently on autoport/qwen3-32b-bringup, not upstream — build from a tree that includes these.
3. Tenstorrent vLLM fork (dev) + TT plugin.
4. Gated weights: accept the Gemma license and provide an HF token (~52 GB).

Known limitations

• 256K recall is ~4/5 deep in the middle — endpoints/most depths reliable; cap at 128K for strict all-depth recall (see "Context length").
• Long-context TTFT is seconds-to-~1.5 min (256K ~95 s). Use generous client timeouts.
• Single slot (max_num_seqs=1) — a 2nd concurrent request queues (looks like a hang); ≥3 concurrent is not supported (a batched-decode device path is still open).
• All-ones prefill (GEMMA4_PREFILL_GATHERED=0) hangs on prefills >~32K — why gathered is default.
• Chunked prefill off (GEMMA4_CHUNKED_PREFILL=0) degrades/OOMs past ~128K — why chunking is default.

Changelog

2026-07-23 — validated 256K via chunked prefill

• Context 72K → full 256K (262144), recall-validated. Adds the whole-body chunked-prefill driver (default on, 16K chunks): bounds peak prefill memory (constant in context) and keeps each chunk's recall reliable across chunk boundaries. Prose needle recall 128K 5/5, 256K 4/5. TTFT 128K ~41 s, 256K ~95 s. No tt-vllm-plugin edits.
• New knobs: GEMMA4_CHUNKED_PREFILL, GEMMA4_PREFILL_CHUNK_SIZE; MAX_MODEL_LEN default → 262144.

2026-07-23 (earlier) — gathered expert prefill, 72K cap

• Per-token gathered expert prefill (default on): ~5–9× faster TTFT, no large-prefill stall. Cap 72K (single-chunk recall ceiling before chunking).

2026-07-21 — bounded sliding-window KV

• Bounded sliding-window serving-KV adapter (Gemma4BoundedForCausalLM): only the 5 global layers scale with context, so the full 256K KV cache fits at max_num_seqs=1.

License

Weights: google/gemma-4-26B-A4B-it under the Gemma Terms of Use. This bundle contains only Tenstorrent integration metadata + adapter + launch config.