muse-glimmer-30b

muse-glimmer-30b on Tenstorrent p300x2

Serves the text decoder of meta-models/Muse-Glimmer-30B from the meta-models organization at revision f84ecc3a. See the original model card for training and evaluation details.

Muse-Glimmer-30B, a dense transformer of about 29.6B parameters including its vision encoder, served with vLLM as a text-only endpoint for long-context agentic coding with tool calling and separated reasoning output.

Runs on Tenstorrent p300x2 (mesh P300x2) — 131,072-token context, up to 32 concurrent sequences.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Agentic coding and tool calling on text with long contexts, one request at a time.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p300x2

Quickstart

tt serve tt-hous/muse-glimmer-30b

tt serve downloads the Docker image and the meta-models/Muse-Glimmer-30B at f84ecc3a0ea984a4c04542a84269e3d065350a6e weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve tt-hous/muse-glimmer-30b.

Capabilities

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "meta-models/Muse-Glimmer-30B" — the weights id, not this package's name.

Tool calling is enabled (--tool-call-parser muse_glimmer): a request that passes tools comes back with tool_calls and finish_reason: tool_calls.

Reasoning output is separated (--reasoning_parser muse_glimmer): the thinking text arrives in reasoning_content, apart from content.

The chat template defaults to Reasoning strength: high, so the model reasons before every reply. A short factual question needs about 500 to 700 completion tokens. Put Reasoning strength: low in the system prompt, or send chat_template_kwargs: {"reasoning_strength": "low"}, when a short budget is required. The model card's sampling is temperature 1.0, top_p 0.95, top_k 64.

Prefix caching is on. With eight 4,096-token prefixes each reused by eight requests (vllm bench serve --dataset-name prefix_repetition, one request at a time), median TTFT is 98 ms with caching and 496 ms without, and p99 TTFT is 952 ms with caching and 526 ms without.

Expected performance

Evals ran through tt-inference-server's eval workflow (its lm-eval command and scoring) against this package at 131,072-token context, with the model card's sampling: temperature 1.0, top_p 0.95, top_k 64.

One request at a time on two P300 boards (four Blackhole chips), 512 output tokens, input length swept to the full context. Values are single measurements, or the median of three runs where a point was re-measured. Prefill tok/s/u is ISL divided by TTFT. The last row fills the 131,072-token context.

Limitations

• Text input only: the checkpoint's vision encoder is not served. Tested on two P300 boards only.
• Not for concurrent long-context serving: admission is limited by the KV cache, so serve one request at a time at long context.
• A max_tokens budget that ends inside the reasoning returns the reasoning as reasoning and an empty string as content.
• A request that sends tools with tool_choice: "none" gets no tool calls, but if the model still writes one, a non-streaming response carries its markup in content.
• Through lm-eval's aime25 task, 18 of 30 responses had no extractable answer and one ran to 294,912 characters, so no AIME score is reported.
• Prefix caching raises p99 TTFT on cold prefixes (952 ms against 526 ms without caching).

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/tt-hous/muse-glimmer-30b/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: [email protected].

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: