qwen3.8-flash-next

qwen3.8-flash-next on Tenstorrent p300x2

Serves the language model of Qwen/Qwen3.8-Flash-Next by the Qwen team at revision f5d08274. See the original model card for training and evaluation details.

Qwen3.8-Flash-Next, a mixture-of-experts model with 125B parameters (about 6B active per token) plus a 51B n-gram embedding table, served with vLLM as a text-only chat and completion endpoint with tool calling and separated reasoning output. Routed experts run in 4-bit block floating point (BFP4), derived once from the BF16 checkpoint.

Runs on Tenstorrent p300x2 (mesh (4, 1)) — 262,144-token context, up to 1 concurrent sequences.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Chat, reasoning and tool calling on text through an OpenAI-compatible API, one request at a time.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p300x2
• About 360 GB of disk for the checkpoint, plus about 64 GiB for the converted expert cache the first serve writes

Quickstart

tt serve tt-hous/qwen3.8-flash-next

tt serve downloads the Docker image and the Qwen/Qwen3.8-Flash-Next at f5d08274bafd880402bd16f5e3e6c514136ec06c weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve tt-hous/qwen3.8-flash-next.

Capabilities

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "Qwen/Qwen3.8-Flash-Next" — the weights id, not this package's name.

Tool calling is enabled (--tool-call-parser qwen3_xml): a request that passes tools comes back with tool_calls and finish_reason: tool_calls.

Reasoning output is separated (--reasoning_parser qwen3): the thinking text arrives in reasoning, apart from content.

Requests that set no sampling fields get the checkpoint's defaults (temperature 1.0, top_p 0.95, top_k 20). For long reasoning, add a presence_penalty of 1.0 to 1.5. Greedy requests (temperature 0) are sampled on the device. Every request with temperature above 0, penalties, logprobs, a seed, min_p, logit_bias, bad_words, allowed_token_ids or structured output is sampled on the host, about 18 ms per token slower.

Expected performance

Task evals:

GSM8K ran through the served endpoint. The Qwen model card reports no GSM8K figure, and none of the text benchmarks it does report (GPQA Diamond, LiveCodeBench v6, IFBench, HLE, SWE-bench Pro, SWE-bench Multilingual, DeepSWE 1.1, NL2Repo-Bench, Toolathlon Verified, Agents' Last Exam, CoWorkBench, JobBench) has been run against this package.

Token agreement with a BF16 reference of the same checkpoint. This is a teacher-forced comparison of next-token predictions, not a task benchmark:

The reference is a BF16 Hugging Face run (Transformers 5.16) of the same checkpoint revision on the CPU. The reference's tokens are fed in and each next-token prediction is compared. The prompts cover mathematics, code, JSON extraction, Chinese and Japanese text, chat, summarization, tool-call formatting and long documents.

Means over one run per row, measured on 2026 Sep 17 on two P300 boards (four Blackhole chips) with a closed-loop latency client: random-token prompts of exactly ISL tokens, output pinned to OSL tokens with ignore_eos, greedy, one unrecorded warmup request per input length, 4 requests per row up to 4,096 input tokens, 2 at 16,384 and 1 above. Prefill tok/s/u is ISL divided by TTFT.

Limitations

• Text only: requests with image or video content are rejected. Tested on two P300 boards only. No batching above one request: concurrent requests queue.
• No benchmark that the Qwen model card reports has been run against this package; only GSM8K and token agreement with the BF16 reference were measured.
• Not for interactive use with long prompts: TTFT grows by about 2.6 ms per prompt token (about 38 s at 16k, 5.6 min at 131k). Set client timeouts of at least 10 min for 131k prompts and stream the response.
• No prefix caching: every turn of a multi-turn chat re-prefills the whole prompt.
• Decode is about 24 tokens per second per user under 2,048 tokens of context and about 14 above it. Host sampling lowers these to about 17 and 11.
• Chains of thought beyond about 25,000 to 30,000 reasoning tokens can drift and lose the final answer. Cap max_tokens near 32,768 for such prompts and prefer several shorter attempts.
• After prompts above 2,048 tokens, output agrees with the reference less often (93.9% top-1 against 97.0% for short prompts).

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/tt-hous/qwen3.8-flash-next/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: [email protected].

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: