qwen3.8-flash-next_p150x4

qwen3.8-flash-next on Tenstorrent p150x4

Derived from Qwen/Qwen3.8-Flash-Next by the Qwen team. The routed experts are converted to 4-bit block floating point on first start and the dense weights run in 8-bit block floating point; see the original model card for training and evaluation details.

Qwen3.8-Flash-Next, a 125B-parameter mixture-of-experts model with about 6B parameters active per token (48 layers mixing gated delta-net and sparse attention, 512 routed experts, top-10 plus one shared expert, a 51B-parameter n-gram embedding) and a vision tower, served by the model's own OpenAI-compatible chat server with multi-token-prediction drafting. Text and image input, text output.

Updated 2026-11-03: the routed experts now run on tt-metal's expert-rows kernels (tenstorrent/tt-metal#59945, not yet merged; the commit under Provenance). Long prompts prefill 11-20 % faster, and decode is 7 % faster in a back-to-back comparison at c32k. In every comparison we ran, the outputs are token for token the same as the 2026-11-02 revision's: the startup checks of every profile, the sampled speed rows below, the c32k-lanes4 gate, all 1,319 GSM8K answers and the agreement measurement. After the update, the first start of each profile builds that profile's caches again (about 8 GB, 4-5 minutes); the 70 GB expert cache is reused. If you pulled an earlier revision, pull again or serve with --refresh.

The 2026-11-02 revision moved the build to tt-metal main, started from a standard Hugging Face cache (the one before it could not start under tt serve), made 128k the default context and added a 256k profile.

Runs on Tenstorrent p150x4 — see the serving profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Chat, step-by-step reasoning (thinking mode), tool calling and image understanding for one interactive user.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150x4
• About 450 GB of free disk: the weights (360 GB), the converted experts (70 GB), the image (4 GB), the kernel cache (8 GB) and about 8 GB for each context profile you use
• Enough free host RAM to keep the 104 GB n-gram embedding table in the page cache

Quickstart

tt serve sjettTT/qwen3.8-flash-next_p150x4

tt serve downloads the Docker image and the Qwen/Qwen3.8-Flash-Next at f5d08274bafd880402bd16f5e3e6c514136ec06c weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs "event": "ready".

Without tt-cli: tt-model serve sjettTT/qwen3.8-flash-next_p150x4.

The first start converts the routed experts into a 70 GB 4-bit cache and builds the context's caches; later starts are ready in about 4 minutes. The first start of another profile builds that profile's caches (about 8 GB, ready in 4-5 minutes). Thinking is on by default; send "chat_template_kwargs": {"enable_thinking": false} for the non-thinking mode.

Serving profiles

Select a different serving profile using --profile.

Capabilities

• The server speaks the OpenAI chat API: /v1/chat/completions (streaming or not), /v1/models and /health. Pass "model": "Qwen/Qwen3.8-Flash-Next".
• Send the sampling settings. A request that names no sampling field is decoded greedily, and thinking defaults to reasoning_effort medium, which caps the thinking at 16,384 tokens. The original model card's thinking settings are temperature 1.0, top_p 0.95, top_k 20; add "reasoning_effort": "xhigh" (or high) to lift the thinking cap. Non-thinking: temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5.
• Images: image_url parts with data: URLs only (nothing is fetched); detail low, auto or high.
• Tools: OpenAI tools / tool_choice. Reasoning streams as reasoning_content.
• Context: the default profile c128k holds 131,008 tokens of prompt and output. The longest GPQA Diamond thinking answer we have measured at c128k is about 105,000 tokens. c256k holds 262,080 for one user; c32k-lanes4 serves up to four greedy requests at once at 32k.
• Refused with HTTP 400 rather than ignored: response_format other than text, logit_bias, parallel_tool_calls: false, and a prompt plus max_tokens longer than the context. Except on c32k-lanes4, one request decodes at a time; up to four wait, and the next gets HTTP 503 with Retry-After.

Expected performance

161 of the 180 GSM8K misses stopped at the 512-token cap; of the answers that finished, 98.4 % are correct.

GPQA Diamond ran on c128k with lm-eval 0.4.13 (gpqa_diamond_generative_n_shot), sampled at the original card's thinking settings (temperature 1.0, top_p 0.95, top_k 20) with seed 1234 and up to 128,118 output tokens. All 198 answers finished on their own (median 5,650 tokens, longest 104,832). In 18 of them flexible-extract finds no letter; also reading a final \boxed{} or a bare letter gives 176/198 = 88.9 %. It was measured with the 2026-11-02 image, not repeated on this one.

Medians of three timed requests after one warm-up, streaming /v1/chat/completions with ignore_eos and max_tokens = OSL, the prompt natural English text, thinking on (the default), sampled at the original card's thinking settings (temperature 1.0, top_p 0.95, top_k 20, seed 1); prefill tok/s/u = ISL / TTFT and decode tok/s/u = (OSL - 1) / (E2EL - TTFT). Measured 2026-11-03 through this container on 4x p150 in one host (a 1x4 ethernet line), while other work kept the host's load average at 10-36. Decode speed depends on how many drafted tokens the model accepts, which varies with the text, so the rows are not a curve over prompt length. At the same 128-token prompt, c128k decodes about 8 % slower than c32k: a decode pass costs more when more context is allocated. c32k-quietbox has not been measured with this image.

Limitations

• Not for serving many users at once, or for uses that need the full-precision model's exact outputs.
• Except on c32k-lanes4 (greedy only), one request decodes at a time.
• A request without sampling fields decodes greedily, and greedy thinking is prone to repeating itself; send the sampling settings above.
• The routed experts run in 4-bit and the dense weights in 8-bit block floating point, so outputs differ from the BF16 model: 95.2 % top-1 agreement (above).
• No video input; images must be data: URLs.
• Long prompts wait for their first token: about 19 s at 31,000 tokens, 41 s at 63,000, 95 s at 127,000 and 236 s at 255,000. Decode slows to about 50 tokens/s at 127,000 and 44 at 255,000. Agreement with the BF16 model is measured up to 32,704 tokens only.
• The c32k-quietbox profile has not been run with this image. Not tested on a QuietBox 2 (2x p300c).

Licensing

Weights: Qwen Community License 1.0 (the original repository's LICENSE). Port and serving code: Apache-2.0.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/sjettTT/qwen3.8-flash-next_p150x4/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: [email protected].

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: