tt-tnt-1024

tt-tnt-1024

tt-tnt-1024 is a 123M-parameter Llama-3-style model with a 512-token context. It was trained from random initialization on Tenstorrent Blackhole with ttml (tt-train), in three passes:

1. over a ten-source curated corpus that includes a small databricks-dolly-15k question-answer slice;
2. over 2.53B tokens of FineWeb-Edu;
3. over the curated blend again.

It is served through the Tenstorrent vLLM plugin across all four Blackhole chips of a P300x2, as a (1, 4) ring mesh with FABRIC_2D_TORUS_XY. Its chat endpoint uses the plain Question: … Answer: … format it saw in training. It is small on purpose, and useful as an instrument rather than a product. It is more fluent than earlier checkpoints, but not more knowledgeable.

Status: Experimental. An earlier 4-chip output-quality regression didn't reproduce on these weights, but its cause was never identified (see Limitations).

It is the larger sibling of episod/tt-tnt, a project first published as tt-nanollama3.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 6, v6 thin).

At a glance

Intended use

Direct use: a hardware-and-tooling research instrument. Use it for short story continuation, short single questions in the Question: … Answer: … format, and experiments on training, packaging, and serving on Tenstorrent hardware.

Out-of-scope use:

• Anything that needs facts. MMLU sits below chance at every checkpoint, and it answers confidently when wrong.
• A system role. The template drops system messages, because nothing in the training data had one.
• Tool calling. These weights emit no tool calls.
• Long contexts. Prompts longer than 512 tokens are rejected.

Quickstart

uv tool install tenstorrent   # once — the Tenstorrent CLI, `tt`
tt model pull episod/tt-tnt-1024
tt serve episod/tt-tnt-1024

tt model pull installs the bundle into its own venv: Python 3.12, ttnn==0.77.0, empty-target vLLM 0.25.1, and the TT vLLM plugin. It downloads the weights by default (tt has no --with-weights flag). The server listens on port 20000 by default and walks upward if that port is busy.

The first serve does the following:

• initializes the 4-chip fabric (Fabric initialized on 4 devices in the log);
• captures prefill and decode traces;
• converts the weights into a tensor cache inside the install folder.

On a TT-QuietBox 2 the first serve reached a ready endpoint in 64 s, and a restart is faster. The server is ready when the log prints Application startup complete.

Without tt-cli, tt-model alone does the whole job:

tt-model pull  episod/tt-tnt-1024 --with-weights
tt-model serve episod/tt-tnt-1024

Stop it with tt-model stop episod/tt-tnt-1024. The bundle needs four chips:

• If you set TT_METAL_VISIBLE_DEVICES, that choice wins.
• Otherwise, if a chip grant sets TT_VISIBLE_DEVICES, run.sh uses the grant's first 4 devices. It refuses to start if the grant has fewer than 4.
• Otherwise it opens chips 0,1,2,3.

Serve profiles

4 KV heads divide 1, 2, and 4, so the architecture can shard across any of those. Only the 4-chip profile is packaged. On 2026-09-27, one chip (TP1) decoded at the same concurrency-1 speed as four (2.93 against 2.88 ms/token) during tuning. A 1-chip profile isn't packaged.

Use a vLLM TT plugin at or after c127c17. Earlier builds have a decode defect that degrades free-running generation into repetition within a few tokens. The plugin reports version 0.1.0 either way, so a version check can't detect this; the bundle's adapter warns structurally.

Using it

The tokenizer's chat template (dolly_qa, since 2026-09-27) renders a conversation as the exact token sequence of a databricks-dolly-15k document in the pretraining corpus:

• Each closed exchange becomes Question: q Answer: a</s>.
• The last user turn becomes Question: q Answer:, and the model writes the answer.
• System messages are dropped.
• Each line loses trailing spaces and tabs, blank lines are dropped, and lines are joined with a single space.
• Only the last 5 messages are kept.
• Reference text goes in the user message after a blank line, where dolly's context field sat.

curl -s http://localhost:20000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "episod/tt-tnt-1024",
       "messages": [{"role": "user", "content": "What is the capital of France?"}],
       "max_tokens": 48, "temperature": 0}'

On 2026-09-27, this bundle on 4 chips returned the following (greedy; each reply ended on its own with finish_reason: stop):

What is the capital of France? → The capital of France is Paris.

Who wrote Romeo and Juliet? → Juliet is the daughter of King Henry VIII of England.

(after the France exchange as history) What is the capital of Italy? → The capital of Italy is Rome.

The second answer is fluent and wrong, which is typical of this model. Every prompt token vLLM received matched a local render of the template, in 3 of 3 requests (one with a system message and a history).

Why this format. The previous template rendered Q: …\nAnswer:. That sequence occurs zero times in the training tokens: the pipeline encoded the corpus line by line, so newlines were never tokens. The results of the switch:

• The new render matches the training pipeline's own ids on 15,006 of 15,006 consecutive dolly document pairs.
• On 30 fresh questions (CPU), answers end cleanly with </s> 77% of the time, against 7% with the old template.
• The old prompt cost 0.41 nats/token of answer likelihood (paired over 300 documents, t = 14.9).

The proof is in docs/measurements/chat-template-proof.json and scripts/verify_chat_templates.py.

Why the 5-message cap exists. A growing conversation otherwise crashes the vLLM engine outright. That is a generic tt-metal/vLLM defect, not this model: it reproduces identically on stock meta-llama/Llama-3.2-1B-Instruct at the same context size (entry 6 in docs/upstream-tt-metal-asks.md).

• With the guard active at 512 context, 14 growing turns ran with zero crashes. Prompt tokens plateaued at ~102 while the number of messages sent grew to 27.
• Without the guard, the engine crashed hard at turn 3.
• That was measured on the prior (dialogue) weights; the guard is unchanged since.

/v1/completions works for plain story continuation.

Expected performance

Serving throughput and latency (on device, 4 chips, measured 2026-09-27):

Methodology.

• Hardware. A TT-QuietBox 2 (2 × p300c, 4 Blackhole chips), (1, 4) mesh, FABRIC_2D_TORUS_XY.
• Bundle and stack. The 2026-09-27 v6 thin build, identical to this one except for the weights-revision string; ttnn 0.77.0, vLLM 0.25.1.
• Tool. vllm bench serve over streaming /v1/completions, with random-token prompts (seed 0, --ignore-eos) and 2 warmup requests.
• Passes. Each row is the second of two identical passes, so compile time is excluded.
• Sampling. "Default" sends no temperature, so vLLM's default of 1.0 applies. Most chat clients do the same.

How to read these numbers:

• ISL 384 is the longest prompt that fits 128 output tokens in the 512 context. Before 2026-09-27, any prompt longer than 128 tokens killed the server.
• Run-to-run variation is about 10%. The tuning run earlier the same day measured 2.88 ms c1 TPOT and 2,487 tok/s at concurrency 8 with the same settings and bundle contents.
• Higher concurrency. The tuning run reached 7,779 tok/s at concurrency 32 and 7,567 at 64, because the batch is capped at 32 (docs/measurements/serving-tune-2026-09-27.md).
• Sampling costs speed. Default sampling at concurrency 8 cost 2.3× in TPOT here (7.88 against 3.40 ms). A single-pass tuning measurement showed 15.3 ms.

Accuracy (CPU, on these exact weights, sha256 b2958652e9cc3308…):

Accuracy sources and methodology.

• The Stage B column comes from EleutherAI lm-evaluation-harness 0.4.9 on CPU (fp32, a 512-token window, batch 16) through scripts/benchmark_external.py. StoryCloze comes from scripts/eval_storycloze.py (xstory_cloze en, mean-per-token normalization).
• The prior-checkpoint column comes from the same instruments at the same window, as recorded in docs/current_model.json.

How to read the accuracy rows:

• StoryCloze is parity with the prior checkpoint, not a win.
  • Stage A alone significantly regressed it (0.6062 → 0.5725, paired sign test p=0.00504).
  • Stage B significantly reversed that regression (p=6.2e-05 vs Stage A; 166 of 266 discordant pairs favour Stage B).
  • Against the prior checkpoint, the difference isn't significant (p=0.326; 109 of 203 discordant pairs).
• MMLU never moves. It reads 0.2295, then 0.2297, then 0.2292 across a 7.2× increase in training tokens. ARC-Challenge stays at or below chance throughout. The gains are in fluency and commonsense (wikitext, LAMBADA, PIQA, ARC-Easy), not knowledge.
• Stage A's composition and token count moved together. Stage A replaced a share of the curated blend with 100% web-educational text. So "more tokens helped" and "the register shifted" are confounded in the external-benchmark gains.
• These numbers use a 512-token window. They aren't comparable to episod/tt-tnt's 2048-window figures.

On-device agreement with CPU (greedy, 10 prompts × 64 tokens, 2026-09-27; tokens that match CPU fp32 before the first divergence):

TP4 and TP1 are token-identical to each other on 4 of 10 prompts. Every divergence is ordinary bf16 drift into different, fluent English.

Limitations

Context is 512 tokens, enforced.

• max_model_len is 512.
• A 513-token prompt gets a clean HTTP 400 ("maximum context length is 512 tokens"), and the server keeps serving. A 496-token prompt succeeds.
• The published bundle before 2026-09-27 accepted up to 512 tokens. It crashed the whole server on any prompt over 128 tokens, because prefill was padded to a 1024 bucket. The adapter now clamps that bucket to 512.

max_num_seqs is 32, and that is a ceiling of the stack, not a tuning choice.

• tt_transformers 0.77 supports batch sizes 1, 2, 4, 8, 16, and 32 only. With 64 or 128 the server fails to start (ValueError: Batch size 64 not supported). KV-cache capacity was not the limit.

• The only way past 32 is vLLM data parallelism. Four 1-chip replicas (DP4, 4 × 32 sequences) were measured:

  concurrency	DP4	TP4 (shipped)
  1 (TPOT)	3.25 ms (13% slower)	2.88 ms
  8	1,018 tok/s	2,487 tok/s
  32	3,713 tok/s	7,779 tok/s
  128	12,366 tok/s	not measured

• DP4 only pays off at concurrency 128 and above, so it isn't shipped (docs/measurements/serving-tune-2026-09-27.md).

4-chip output quality: a documented regression that did not reproduce.

• The earlier record. A controlled A/B on the prior (dialogue) weights produced invented non-words on 4 chips (FABRIC_2D_TORUS_XY), such as "Tryburg", "Alexandary", and "Higheriq". The same prompt and sampling settings on 2 chips gave rough but recognizable English (docs/serving-with-tt-kernel.md §8).
  • A stale output buffer behind a live trace was the working hypothesis. It was never confirmed (entry 7 in docs/upstream-tt-metal-asks.md).
• The 2026-09-27 comparison. The regression did not reproduce on the current Stage B weights.
  • 1-chip vs 4-chip, greedy, 10 prompts: fluent English on both topologies, with no invented non-words.
  • 4 chips agreed with CPU slightly longer than 1 chip did (table above).
  • The three chat answers above, served on 4 chips, are also fluent English.
• What this does and doesn't show. Ten prompts show that 4 chips are not measurably worse than 1 chip here. They can't rule the regression out.
  • The two comparisons differ in weights, in comparison topology (2 chips then, 1 chip now), and in decoding (then, a sampled short-generation task from scripts/story_tools.py; now, greedy decoding).
  • Nothing has explained the original observation, so it isn't declared fixed.

The tokenizer is pinned; the on-device weights load from main.

• The manifest pins weights.revision to the commit that added the dolly_qa chat template, and vLLM loads the tokenizer and template from that commit.
• The tt_transformers model code loads the model config and weights from episod/tt-tnt-1024 without a revision, so it reads main. The two are identical today.
• On 2026-09-25 these weights replaced the dialogue checkpoint on main without a bundle rebuild. Any future weight upload would again change what installs serve, and it will come with a repackage and a changelog entry.
• The tensor cache is keyed by commit, so stale weights are never served. A card-only edit triggers one re-conversion on the next serve.

It repeats. Greedy decoding often falls into a repetition loop after the first sentence, though the new template lets short answers stop on </s>.

• 2.5B+ tokens of additional pretraining (Stage A) didn't fix it.
• On the prior checkpoint, the 4-gram repeat rate was measurably worse than tt-tnt-1024a's, at 3.32× the seed floor:

• Nine of ten behavioural signals came back NOT INTERPRETABLE against the project's 0.1944-nat seed-only noise floor.
• That comparison was measured on the prior (dialogue) checkpoint and is kept for the record (docs/measurements/evaluation-tt-tnt-1024a-vs-tt-tnt-1024-dialogue.md).

It isn't instruction-tuned beyond a 2% slice of databricks-dolly-15k.

• It has no system role. That role is structurally absent from anything this regimen could produce, not merely unimplemented.
• It is more fluent than earlier checkpoints and not more knowledgeable.
• Treat it as an artifact of a hardware-and-tooling project, not as an assistant.

Tool calling isn't supported on these weights. They emit no tool calls. A separate, unpublished continued-training checkpoint does (see History).

Risks and safety considerations

• Confident wrong answers. It answers in the shape of an answer whether or not it knows. On 2026-09-27 it said Juliet "is the daughter of King Henry VIII of England". Don't rely on it for facts.
• 4-chip numerical path. The historical invented-word regression (see Limitations) is unexplained. It is a TT-specific serving-path deviation, not a property of the weights. Watch for non-words in long free-running output.
• Repetition loops under greedy decoding. Use sampling for open-ended text, and accept the throughput cost shown above.
• Unbounded conversations crash the engine without the shipped chat-template guard. Don't replace the template with one that renders the full history.

Licensing

• Weights and this project's code: Apache-2.0.
• Stage B / curated blend (400M-token budget, shipped only as a recipe):
  • 46% of the blend is share-alike, under two mutually incompatible copyleft terms.
  • TinyStories is CDLA-Sharing-1.0.
  • Wikipedia is CC-BY-SA-3.0.
  • The databricks-dolly-15k dialogue slice (2%, rendered as plain Question: … Answer: … prose with no role markers) is CC-BY-SA-3.0.
  • Per-source terms are in docs/corpus_licensing.md.
• Stage A: 2,529,270,500 tokens of HuggingFaceFW/fineweb-edu (sample-10BT, ODC-By 1.0, pinned revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9).
  • It was fetched directly rather than through the project's corpus-blend registry, so it isn't part of the episod/tt-tnt-corpus recipe.
  • By a wide margin, it is the majority of what these weights have read.
  • See docs/current_model.json's corpus.note.

Training

Training ran as 4-chip DDP across both p300c cards of a TT-QuietBox 2. The final validation loss is 2.5373 (Stage B, real held-out loss over the full validation split). The ten-source blend is episod/tt-tnt's nine sources plus the dolly slice.

History: experiments on the prior checkpoint

⚠️ Everything in this section was measured against the prior (2026-08-20 to 2026-08-29 dialogue) checkpoint, not against the currently published Stage A/B weights. It records this model's lineage and makes no claims about the current weights.

Routing by physical die address is nearly free. Tokens can be routed to experts by where they live on the harvested 11×10 Tensix grid rather than by a learned gate:

• Freezing the gate to that geography costs only 0.0118 nats against a freely learned gate (|t| 5.1, 14/15 signs).
• Source-characteristic tokens occupy measurably distinct die regions: cell purity 0.546 against a 0.231 permutation floor.
• Seeding the gate from the die map and then letting it move buys nothing measurable (+0.0044, signs 8+/7−), even though the seeding works as a classifier (61.2% region recovery against a 10% chance floor).

Sparse routing (Mixture of Enthusiasts) beats dense from scratch. Both arms trained one epoch from init, paired on seed 5489:

• Validation was 2.8098 for MoE against 2.8748 for dense (mean delta +0.0481, |t| 7.3, 20 of 22 signs).
• It replicated at seed 8191 (+0.0354, |t| 4.5, 19/22). Pooled, the gain is +0.0417 over 44 points, with the same late-separating trajectory in both runs.
• Read it as the ordinary MoE bargain: 3.62× total parameters at 0.989× active compute. It isn't evidence about die-region routing.

Tool calling works structurally, and only structurally. A continued-training run teaches four tools (factual_response, witty_response, absurdist_response, misunderstood_question). They are emitted as <tool_call> blocks, which vLLM's hermes parser turns into structured tool_calls:

• Unseen questions score higher than seen ones (78% vs 72%).
• 3/5 chat requests came back as genuine structured tool_calls end-to-end through the server.
• The content inside the calls is poor ("The capital of France is the capital of France"; Portugal answered "Madrid").
• That checkpoint is not published. See docs/measurements/tool-calling-stage{1,2}.json. Its Q: …\nAnswer: format was the chat template this repo shipped until 2026-09-27.

A five-slot think-block can be learned, but it doesn't help yet. The fine-tuned model emits offer / accept / add / stakes / handback blocks:

• Blocks are well formed in 98% of generations (784/800; the control produces them 0% of the time).
• Substituting another story's block changes 100% of continuations.
• It moves none of the four failure-mode scores at α = 0.01. The block is context the model conditions on, not an instruction it obeys.
• An earlier pass reported 0% adherence from a run in which all 17 RMSNorm gammas were frozen (stochastic_rounding defaults off on the SFT path). With the gammas free, 0% became 98%.
• Close reading is in episod-log.md, 2026-08-21.

The reach dial (2026-08-23/24) works, is small, and is paused. Forcing a reach slot to near / mid / far moves the realised semantic distance of the add word monotonically, scene-paired over 826 held-out scenes:

• About 53% of the raw effect is word frequency and ~47% survives the frequency control.
• A control arm never shown a reach slot shows nothing.
• The pre-declared EUREKA criterion failed on one gate: add slot-hit shortfall 0.0896, a near-side dip.
• Three explanations were eliminated:
  • More training doesn't help: at 9000 steps, +0.060392 vs +0.060438.
  • The arms aren't undertrained: the adherence gate got worse at 9000 steps.
  • The vocabulary isn't fixable by filtering: a validated content-word filter made the slot more concentrated.
• The constraint is the corpus. TinyStories has 13,777 distinct words, and the top 1,000 cover 90.9% of tokens.
• Everything re-derives from docs/measurements/reach-dial.json through scripts/eval_reach.py --rescore-from, with no model, tokenizer, or device.

Changelog

Related packages

• episod/tt-tnt is the 22M sibling in the same from-scratch family. It is a different model, not a hardware variant: hidden 384, 3 KV heads, 2048 context, no FineWeb-Edu, and no dialogue data, so its chat template is plain continuation. It serves on one chip. Neither package supersedes the other.
• episod/tt-tnt-corpus is the curated blend recipe. Stage A's FineWeb-Edu isn't part of it.

Feedback

• Questions or problems with this package: open a discussion at https://huggingface.co/episod/tt-tnt-1024/discussions. That is the one channel that reaches the bundle's author.
• A problem with the tt tooling itself: use tt report issue. It opens a prefilled issue against tenstorrent/tt-cli, not this package.
• Product feedback: send it to [email protected].

The full build log and every measurement are at tsingletaryTT/tt-tnt.

Provenance