gpt-oss-20b

gpt-oss-20b on Tenstorrent p150 and p150x2 and p150x4

Serves openai/gpt-oss-20b by OpenAI at revision 6cee5e81. See the original model card for training and evaluation details.

gpt-oss-20b, OpenAI's 21B-parameter mixture-of-experts reasoning model (3.6B active parameters), served with vLLM for chat, reasoning and function calling.

Runs on Tenstorrent p150 or p150x2 or p150x4 — see the serving profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Chat, reasoning and function calling in English through an OpenAI-compatible API.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150 or p150x2 or p150x4
• p150x2 profiles: devices 0 and 1 must be the two ASICs of one P300C board

Quickstart

tt serve tt-hous/gpt-oss-20b

tt serve downloads the Docker image and the openai/gpt-oss-20b at 6cee5e81ee83917806bbde320786a8fb61efebee weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve tt-hous/gpt-oss-20b.

Serving profiles

Select a different serving profile using --profile.

Capabilities

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "openai/gpt-oss-20b" — the weights id, not this package's name.

Tool calling is enabled (--tool-call-parser openai): a request that passes tools comes back with tool_calls and finish_reason: tool_calls.

Pick a profile for the traffic. The -throughput profiles prefill in 32,768-token steps and pause decode for every request while a long prompt prefills: use them for batch work. The p150, p150x2 and p150x4 profiles prefill in 8,192-token steps with 8 decode steps between them: use them for chat, agents and mixed prompt lengths. Pick the largest mesh your box has.

Reasoning comes back separately from the answer, in message.reasoning. max_tokens covers both. For a chat completion with max_tokens of 256 or more, the server keeps clamp(max_tokens // 8, 128, 1024) tokens for the answer, so a request that runs out of budget while reasoning still returns an answer (finish_reason: "length" with non-null content). reasoning_effort (low, medium, high) sets how much the model reasons, and usage.completion_tokens_details.reasoning_tokens reports what it used. Set VLLM_HARMONY_FINAL_RESERVE_TOKENS=-1 in the container to turn the reserve off.

Greedy requests, and sampled requests with top_k of 32 or lower, are sampled on the device. The request fields that move a batch to host sampling are listed under Limitations.

Expected performance

Evals use lm-evaluation-harness 0.4.10.dev0 through the chat API: reasoning_effort medium, greedy, a 4,096-token output budget, seed 42, 32 concurrent requests, chat template applied, on the throughput profile of each mesh. GSM8K is zero-shot. Scores grade the final answer only, and every request returned a final answer on every mesh. The OpenAI model card publishes neither benchmark. The reference column is the frozebench run of openai/gpt-oss-20b with lm-eval, the chat template and a 4,096-token budget on 2025 Oct 17 (IFEval, GSM8K, the lm-eval default gsm8k task).

Means over one run per row on a QuietBox 2 (two P300C boards), against this package's image, with a closed-loop latency client: random-token prompts of exactly ISL tokens sent to /v1/completions and streamed, output pinned to OSL tokens, greedy, 4 requests per user up to 4,096 input tokens, 2 per user up to 16,384 and 1 per user above, one unmeasured warmup request per input length. The rows are selected from a sweep at 1 to 32 concurrent users. The P150 and P150x2 rows were measured at the same time on separate boards. Random prompts never hit the prefix cache. Prefill tok/s/u is ISL divided by TTFT, so above concurrency 1 it includes queueing time. The p150, p150x2 and p150x4 profiles have no speed rows yet.

Limitations

• Speed was measured on the three throughput profiles only; the p150, p150x2 and p150x4 profiles were not benchmarked. Tested on one QuietBox 2 only: four separate P150 boards were not tested, and eight-chip meshes are not offered.
• Not for workloads that need OpenAI's built-in Python or browser tools: none is provided. Function calling uses the OpenAI tool format. The model sometimes emits malformed tool-call JSON, so agent scaffolds need a retry.
• Capacity across all requests is 264,192 tokens of context on P150, 526,336 on P150x2 and 1,054,720 on P150x4. Further requests wait for admission, and the wait is part of TTFT.
• max_tokens covers reasoning and answer. Below 256 tokens, or for a request served without the answer reserve (n above 1, logprobs, ignore_eos, a required tool call), a response can end with finish_reason: "length" and null content.
• One request with min_p, logit_bias, bad_words, allowed_token_ids, min_tokens, or a non-zero temperature with top_k above 32 or unset, moves the whole batch to host sampling and slows decode for every user until it finishes.
• No run of the same prompts against a BF16 reference of the checkpoint was measured.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/tt-hous/gpt-oss-20b/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: [email protected].

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: