tt-waypoint

tt-waypoint

Waypoint-1.5-1B is Overworld's interactive "world model". It is an autoregressive causal diffusion transformer that generates video one frame at a time, steered by mouse and scroll input.

This package runs it on Tenstorrent Blackhole via TTNN:

• What it is: a from-scratch bring-up. Nothing in tt-metal's models.tt_dit covers this architecture.
• How you use it: a stateful, session-based HTTP API served from a small ASGI server.
• What it needs: one Blackhole chip (profile P150, a 1x1 mesh). All hardware verification so far ran on one chip of a P300c board.
• Maturity: experimental. Frames are coherent, but it runs roughly 1,400x below real time, and no performance work has been done yet.

Packaged and published with tt-model-manager as a v6 thin bundle (manifest schema 6). It is a pip/venv install, not a container image.

At a glance

Intended use

Direct use: interactive demos and research into world-model inference on Tenstorrent hardware. Seed a session with an image, then step it forward one frame at a time with directional (mouse) and zoom (scroll) controls.

Out-of-scope use:

• Real-time or interactive-rate play. It runs at about 0.04 FPS against the upstream 60 FPS target.
• Multiple concurrent users. There is one active session per server process.
• Keyboard or button control. The 256-wide button vector has no published semantics, so it is always sent as zeros.
• Text-prompted generation. This checkpoint has prompt_conditioning: null.
• Multi-chip meshes.

Quickstart

uv tool install tenstorrent   # once: the Tenstorrent CLI, `tt`
tt model pull episod/tt-waypoint
tt serve episod/tt-waypoint

tt model pull downloads two things:

• This bundle: a small venv-install recipe. It is install.sh/run.sh plus two wheels; ttnn, torch and the base HTTP stack come from an index at install time.
• The Overworld/Waypoint-1.5-1B weights: revision 391f928, into the bundle's own HF cache. Weights come down by default, and tt has no --with-weights flag. The pull fetches the whole upstream repo, about 11.4 GB. The server reads only three files from it.

serve starts the model's own HTTP server on port 20000, or the next free port if 20000 is busy. It is ready when it logs Application startup complete.

Measured on a fresh install of this bundle (P300c QuietBox, 2026-09-27):

Without tt-cli, tt-model alone does the whole job:

tt-model pull  episod/tt-waypoint --with-weights
tt-model serve episod/tt-waypoint

Serve profiles

No multi-chip profile exists. SUPPORTED_MESH_SHAPES is {(1, 1)}, and the server refuses any other WAYPOINT_MESH_SHAPE.

Using it

This is not an OpenAI-compatible API. /v1/models exists only as a stub for tooling. The real interface is a stateful session: seed once from an image, then step.

# Seed a session from a real starting image (base64-encoded PNG/JPEG; resized to 512x1024):
curl -s localhost:20000/v1/sessions -H 'Content-Type: application/json' \
  -d "{\"image_b64\": \"$(base64 -w0 my_photo.png)\"}"
# -> {"session_id": "...", "frame_index": 1, "frame_b64": "<base64 PNG>"}

# Step the session forward one frame, steered by direction/zoom:
curl -s localhost:20000/v1/sessions/<session_id>/step \
  -H 'Content-Type: application/json' -d '{"direction": "forward", "zoom": 0.0}'
# -> {"session_id": "...", "frame_index": 2, "frame_b64": "<base64 PNG>"}

# End the session:
curl -s -X DELETE localhost:20000/v1/sessions/<session_id>

• direction: one of forward, back, left, right, forward_left, forward_right, back_left, back_right. These map to the model's [dx, dy] mouse input.
• zoom: the scroll scalar. Only its sign has a documented meaning.
• Seeding: the API has no RNG seed parameter. Per-step noise is drawn inside the server.
• Errors:
  • 503: the model is still starting.
  • 404: the session ID is unknown, or a newer POST /v1/sessions superseded it.
  • 409: you stepped with no active session.
  • 400: the request was bad.

The GitHub repo also has a local Gradio UI (app.py), which is a pure HTTP client for this same server.

Expected performance

Methodology: on one Blackhole chip of a P300c (1x1 mesh, ttnn==0.78.0 from PyPI, eager execution, batch 1, one session), a session seeded from ref_activations/seed_frame_ref.png was stepped with the default controls (forward, zoom 0), timing host HTTP wall-clock per POST .../step after one untimed warmup step: N=15 on the published bundle 1dd34b8 and N=3 on this bundle revision (weights 391f928), both on 2026-09-27.

How to read the accuracy numbers:

• Why a trajectory match is the wrong test. This model is iterative and self-referential: each frame's denoising runs on this port's own previous output. Any per-step difference, including ordinary bf16 rounding, therefore makes the generated trajectory diverge from any single reference run, the way two chaotic systems do. Comparing generated-frame pixels against one reference trajectory gives correlations as low as 0.05-0.24. The bring-up investigation found that this is the wrong bar, not a defect.
• What the claim rests on: correctness of each individual step, correctness of each component, and visual inspection of real-image sessions.
• What it does not claim: a bit-exact trajectory match.

The full investigation is in BRINGUP_LOG.md.

Limitations

• Far from real time. It takes about 25 s per frame (0.04 FPS) against the model's 60 FPS target. No tracing, kernel fusion, batching across sigma steps or quantization has been attempted.
• One session per process. The KV-cache state lives on the same objects as the loaded weights. Starting a new session silently ends the previous one, and a stale ID returns 404.
• Single chip only. There is no multi-chip or tensor-parallel path. Every hardware run used one chip of a P300c board. The p150 profile states the requirement (one chip); it has not been tested on a physical P150 board.
• Mouse and scroll control only. The 256-wide button input is always zero, because no control-scheme mapping has been published. zoom has sign-only semantics.
• The 24-layer correlation (0.963) is below the 0.99 bar. The gap was diagnosed as bf16 compounding over a deep residual stack, with no localized logic bug. The in-repo test_full_model.py assert is left failing on purpose rather than weakened.
• Resolution is fixed at 512x1024. Upstream's GPU numbers are at 720p, so the performance comparison above is not a controlled one.
• The pull over-fetches. The server loads only transformer/config.json, transformer/diffusion_pytorch_model.safetensors and vae/diffusion_pytorch_model.safetensors, at the pinned revision. The bundle's weight pull has no file filter, so pull --with-weights also downloads about 4 GB the server never reads: the root model.safetensors and assets/.

Risks and safety considerations

• Drift over long sessions. Per-step bf16 noise accumulates. How many frames stay usable has not been measured. In a 16-frame forward session from the reference seed:
  • The view settled into a close-up grass texture by frame 4.
  • It lost contrast by frame 16: pixel standard deviation 42-50 on frames 1-8, 14 on frame 16.
• Out-of-distribution seeds give noise. A synthetic or random-static seed image produces unstructured output. The upstream reference behaves the same way under that input. Seed with a real, natural image.
• Numerical deviation from the reference. ttnn SDPA has no fp32 mode. This port uses bf16 with fp32 accumulation (fp32_dest_acc_en, HiFi4). Outputs are plausible but will not match GPU output pixel for pixel.
• Generated video is synthetic. It is not a faithful simulation of any real place.

Licensing

• Weights: Overworld/Waypoint-1.5-1B, Apache-2.0.
• Port and serving code: the waypoint_ttnn wheel, Apache-2.0 (repo LICENSE).
• Vendored tt-metal code: tt_waypoint_models_closure, which is models/common/lightweightmodule.py from tt-metal v0.78.0. Apache-2.0.

Changelog

Related packages

None. This is the only Tenstorrent port of Waypoint-1.5-1B.

Feedback

• Questions or problems with this package: open a discussion at https://huggingface.co/episod/tt-waypoint/discussions. That is the one channel that reaches the bundle's author.
• Problems with the tt tooling itself: run tt report issue. It collects your environment and opens a prefilled issue against tenstorrent/tt-cli; it does not reach this package's author.
• Product feedback: [email protected].

Provenance

The served-path code is split into two small wheels built for this bundle, both in wheels/: