gpt-oss-120b

gpt-oss-120b on Tenstorrent p150x4

Serves openai/gpt-oss-120b by OpenAI at revision b5c939de. See the original model card for training and evaluation details.

gpt-oss-120b, OpenAI's 117B-parameter mixture-of-experts reasoning model (5.1B active parameters), served with vLLM for chat, reasoning and function calling.

Runs on Tenstorrent p150x4 — see the serving profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Chat, reasoning, coding and function calling through an OpenAI-compatible API.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150x4
• Four P150 chips: two P300C boards (a QuietBox 2) or four P150 boards

Quickstart

tt serve tt-hous/gpt-oss-120b

tt serve downloads the Docker image and the openai/gpt-oss-120b at b5c939de8f754692c1647ca79fbf85e8c1e70f8a weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve tt-hous/gpt-oss-120b.

Serving profiles

Select a different serving profile using --profile.

Capabilities

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "openai/gpt-oss-120b" — the weights id, not this package's name.

Tool calling is enabled (--tool-call-parser openai): a request that passes tools comes back with tool_calls and finish_reason: tool_calls.

Pick a profile for the traffic. default prefills long prompts in 8,192-token steps and keeps decoding running requests between the steps: use it for chat, agents and mixed prompt lengths. throughput prefills first in 32,768-token steps and pauses decode for every request while a long prompt prefills: use it for batch work. Where the speed table below measures both profiles at the same point, throughput finishes 8 or 32 prompts of 16,384 tokens and 8 prompts of 32,768 tokens 7 to 9% sooner than default, and 32 prompts of 4,096 tokens about 1% sooner.

Reasoning comes back separately from the answer. max_tokens covers both. The server keeps clamp(max_tokens // 8, 128, 1024) tokens of every chat completion's budget for the answer, so a request that runs out of budget while reasoning still returns an answer (finish_reason: "length" with non-null content). reasoning_effort (low, medium, high) sets how much the model reasons, and usage.completion_tokens_details.reasoning_tokens reports what it used.

Sampling: temperature 0, or top_k of 32 or lower, is sampled on the device. The request fields that move a batch to host sampling are listed under Limitations.

Prefix caching is on in both profiles. The prefix-cache table under Expected performance shows what a cache hit saves.

Expected performance

The reference figures are from the OpenAI model card at reasoning effort high without tools, except IFEval. Evals run through the chat API: reasoning_effort high, one sample per item, temperature 1.0, top_p 1.0, top_k 32, output budgets of 65,536 tokens for GPQA Diamond and AIME, 32,768 for HLE and 8,192 for MMLU and MMMLU. IFEval: reasoning_effort medium, greedy, 4,096 tokens, 32 concurrent requests. The last three rows use the tt-inference-server release workflow (lm-evaluation-harness over chat completions, temperature 1.0, seed 42; AIME 2025 and GPQA Diamond at reasoning_effort high with 122,880 output tokens, MMLU at low with 65,536). In the release-workflow AIME 2025 and GPQA Diamond rows, a response whose reasoning used the whole 122,880-token budget returned no answer and counts as a miss.

Means over one run per row on a QuietBox 2 (two P300C boards), against this package's image, with a closed-loop latency client: random-token prompts of exactly ISL tokens, output pinned to OSL tokens, 4 requests per user up to 4,096 input tokens, 2 per user at 16,384 and 1 per user above. The default rows are selected from a sweep at 1 to 32 concurrent users. Random prompts never hit the prefix cache. Prefill tok/s/u is ISL divided by TTFT, so above concurrency 1 it includes queueing time: at most 17 sequences of 16k tokens, 8 of 32k, 4 of 64k or 2 of 128k are resident at once, and later requests wait for admission.

Prefix cache, measured with vllm bench serve (a shared random prefix plus a 512-token unique suffix, 64 output tokens, greedy, one user, 12 requests), median TTFT:

A cache hit still prefills about 2,560 tokens plus the new suffix, at any prefix length. A shared prefix under about 5,000 tokens saves little.

Limitations

• Needs four P150 chips; P150 and P150x2 are not offered. Tested on one QuietBox 2 only.
• Not for workloads that need OpenAI's built-in Python or browser tools: none is provided, so OpenAI's "with tools" results do not apply. A malformed tool-call JSON from the model is passed through as is, and an empty final message ends with finish_reason: "stop", so agent scaffolds need a retry for both.
• Capacity is 296,960 tokens of context across all requests. Further requests wait for admission, and the wait is part of TTFT. Prompts above 2,048 tokens prefill one at a time, and the first prompt of a new length adds a few seconds. 9 to 16 concurrent requests decode at about 50 to 58 ms per token instead of 42.
• One request with min_p, logit_bias, bad_words, allowed_token_ids, min_tokens, structured output, any logprobs, or a non-zero temperature with top_k above 32 or unset, moves the whole batch to host sampling and slows decode for every user until it finishes.
• max_tokens covers reasoning and answer. A hard problem under a small budget gets a short, truncated answer. Requests with n above 1, logprobs, ignore_eos or a required tool call have no answer reserve and can return null content with finish_reason: "length".
• With 8 or more concurrent requests at 16k tokens and up, one request per batch occasionally receives its own prompt streamed back as generated text, in one chunk of up to 725,000 characters. Clients should tolerate an oversized first chunk.
• Under sustained 32-user load with 122,880-token budgets, the server can fail with no free device state slot. Submitting such work in fixed batches of 32 avoids it.

Licensing

Weights: Apache-2.0. Port and serving code: Apache-2.0, from public branches of tt-metal, vLLM and vllm-tt-plugin.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/tt-hous/gpt-oss-120b/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: [email protected].

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: