tt-animatediff

tt-animatediff

This is a port, not a checkpoint. This repository ships no model weights. It is an implementation of AnimateDiff that runs the Stable Diffusion 1.4 UNet and VAE on a single Tenstorrent Blackhole chip through TTNN. Motion coherence comes from cross-frame temporal attention (temporal_alpha), which approximates the AnimateDiff MotionAdapter rather than running it. The package produces 512×512 text-to-video GIFs over a small HTTP API, and it also works as a diffusers custom pipeline with a CPU fallback. The weights come from their upstream repositories when you first pull or generate. This is a community bring-up and is not a Tenstorrent-supported product.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 6, "thin").

GitHub · Docs · How it was built

At a glance

Intended use

Direct use: Short text-to-video clips (8-16 frames, 512×512) on a single Blackhole chip. Use it for demos and experiments, or as a reference for bringing a diffusion model up on TTNN.

Out-of-scope use:

• Resolutions other than 512×512. The UNet is compiled once at a 64×64 latent.
• Multi-chip serving. The server refuses a multi-chip mesh.
• Concurrent throughput. Requests are serialized on one device.
• Anything prohibited by the RAIL-M use restrictions on the weights.
• The full AnimateDiff MotionAdapter is not on the served path. It is only available through the GitHub repo's CLI (--motion-adapter, Phase 3).

Quickstart

uv tool install tenstorrent   # once — the Tenstorrent CLI, `tt`
tt model pull episod/tt-animatediff
tt serve episod/tt-animatediff

tt model pull downloads the bundle and the SD 1.4 weights (CompVis/stable-diffusion-v1-4, pinned to revision 133a221) into the bundle's own HF cache (<install>/.hf). Weights come down by default for a bundle; tt has no --with-weights flag. The server needs only about 4 GB of SD 1.4 (unet/, vae/, text_encoder/, tokenizer/), but the prefetch has no file filter and takes the whole repo, about 22 GB. If disk is tight, run tt-model pull without --with-weights: the server then downloads just the 4 GB it reads, at the same pinned revision, on first start.

Install builds a per-model venv from tt-model-requirements.txt: ttnn==0.77.0 from PyPI plus the two wheels in wheels/. tt-model serve listens on port 20000, or the next free port. (run.sh run by hand defaults to 8000.) The device is opened and the pipeline warmed inside the ASGI lifespan, before readiness is reported. The server is ready when uvicorn logs Application startup complete.

Measured on a fresh install with an empty HF cache (one P300c chip, 2026-09-27):

• Launch to ready: 153 s. This includes the 4 GB weight download and the TTNN UNet build.
• First request: about 155 s (8 frames x 8 steps). Most of it is one-time kernel compile.
• After a warm restart: ready in about 10 s, and the first request takes about 13 s.

Without tt-cli — tt-model alone does the whole job:

tt-model pull  episod/tt-animatediff --with-weights
tt-model serve episod/tt-animatediff

Serve profiles

One chip is the model's limit, not a conservative default. A four-chip serve failed in the lifespan with Can't convert a tensor distributed on MeshShape([1, 4]) mesh to row-major logical tensor, because the SD demo's tensors are not sharded. The server now refuses a multi-chip mesh up front.

Using it

This is not an OpenAI-compatible chat API. /v1/models exists for tooling. The real endpoint is OpenAI-shaped /v1/videos/generations, with the same field names tt-inference-server's media server uses.

curl -s http://localhost:20000/v1/videos/generations \
  -H 'Content-Type: application/json' \
  -d '{"prompt": "a swirling nebula, teal and gold, cinematic",
       "num_frames": 8, "num_inference_steps": 25, "seed": 42}' \
  | jq -r '.data[0].b64_json' | base64 -d > out.gif

Request fields, with server defaults:

The response is {"created": <unix time>, "data": [{"b64_json": "<GIF>"}]}. There is one denoise loop at a time. Concurrent requests queue behind a device lock.

As a diffusers pipeline

The same repo also loads as a custom diffusers pipeline. trust_remote_code=True is required, which means loading executes pipeline.py from this repo:

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "episod/tt-animatediff",
    custom_pipeline="episod/tt-animatediff",
    trust_remote_code=True,
)

# mode="auto": Blackhole if the ttnn runtime is importable, CPU otherwise.
frames = pipe("a swirling nebula, teal and gold, cinematic").frames
frames[0].save("out.gif", save_all=True, append_images=frames[1:], duration=125, loop=0)

print(pipe.resolved_mode)  # "blackhole" or "cpu" — what actually ran

Loading is offline-safe and opens no device. Nothing is fetched until you call the pipeline.

model_index.json's base_model, motion_adapter and lightning_repo fields are declarative metadata, not configurable inputs. They record the upstream weights the pipeline resolves, but __call__ never reads them. Passing a different value at from_pretrained() time is accepted and changes nothing.

On a Blackhole box, mode="blackhole" requires the hardware instead of falling back. A missing runtime is then an error rather than a silent 100× slowdown:

frames = pipe("a swirling nebula", mode="blackhole", num_frames=8, num_steps=25).frames

On any machine without hardware, the distilled 4-step Lightning weights make CPU tolerable:

frames = pipe(
    "a swirling nebula", mode="cpu", use_lightning=True, lightning_steps=4,
    num_frames=4, guidance_scale=1.0,
).frames

The CPU path uses the full AnimateDiff MotionAdapter (guoyww/animatediff-motion-adapter-v1-5-2), which is why it is listed in base_model. The tt-model bundle never fetches it. The delegate package is imported if installed, and otherwise fetched from this repo. To install it explicitly:

pip install 'animatediff-ttnn @ git+https://github.com/tenstorrent/tt-animatediff'

Implementation phases

The HTTP server and mode="blackhole" both run Phase 2.5. mode="cpu" runs Phase 1. Phase 3 is reachable only through the GitHub repo's CLI (--motion-adapter).

Expected performance

This bundle, served by tt-model serve:

Methodology:

• Setup: one Blackhole chip of a P300c (mesh 1x1, ttnn==0.77.0 from PyPI), prompt "a lighthouse in a storm, cinematic", 512×512, guidance 7.5, temporal_alpha 0.35, seeds 1-3.
• Timing: end-to-end HTTP wall time of POST /v1/videos/generations, including base64 GIF encoding. One untimed 8×8 warmup came first, and N=3 requests were run per configuration.
• Runs: the published bundle 6ccfdce and this bundle revision (weights 133a221), both on 2026-09-27.

Earlier development measurements (tt-metal v0.77.0 host environment, not this bundle):

• Latency model: it fits 2.84 s + 0.571 s × steps at 8 frames.
• Concurrency: three concurrent requests took 1.016× the serial time. Requests are serialized by design.
• Supersedes: these numbers replace an older claim of ~12.5 s/frame, which was 6.4× too slow.

Accuracy vs reference: not yet measured. No PCC, CLIP or SSIM comparison exists between the served Phase 2.5 output and a diffusers SD 1.4 reference. The only PCC figures on record cover kernel-level TT-Lang tests and a 4-chip determinism check, not the served output. The byte-identical result above is a regression check between two bundle revisions, not an accuracy measurement.

Limitations

• One chip, 512×512 only. Other sizes are rejected at the request edge. A multi-chip mesh is refused at startup.
• P150 is declared but not measured. Every Blackhole number above comes from one P300C chip.
• pull --with-weights over-fetches. The whole SD 1.4 repo (~22 GB) comes down, although serving reads ~4 GB.
• Requests are serialized. No multi-user throughput exists, and none should be expected from this design.
• Phase 2.5 approximates AnimateDiff. Temporal attention runs on 4-channel noise predictions on CPU, not on the 320-dimensional UNet features the MotionAdapter uses. The full adapter (Phase 3) is CLI-only and much slower (~52 s/frame, not re-measured).
• The thin bundle installs pytest. tt-metal v0.77.0's models/common/utility_functions.py imports it at module scope. The fix (tt-metal fdcb5cf) is in v0.79.0 but not in v0.77.0.
• The diffusers TTNN path needs a local build. It requires a Blackhole board and a local tt-metal build. Most users of the diffusers pipeline will only exercise the CPU path.
• Multi-chip frame counts. On multi-chip boards (CLI mesh frame sharding), the frame count must be a multiple of the chip count. 8 works on 1, 2 and 4 chips.
• No distilled LCM weights. The LCM distillation track is closed: all four runs failed, and no distilled weights ship here. use_lightning=True on CPU uses ByteDance's published checkpoint.
• mode="sim" is slow. The ttsim virtual Blackhole is bit-exact but 10-100× slower per op. It needs the simulator binary: pass sim_so="/path/to/libttsim_bh.so", or leave it unset only if the binary is at ~/sim/libttsim_bh.so.

Risks and safety considerations

• Output comes from SD 1.4 and inherits its known biases and failure modes. The RAIL-M use restrictions apply to anything you generate.
• Motion is an approximation. Frames can drift or flicker in ways a full AnimateDiff run would not.
• The numerical deviation from the diffusers reference on Blackhole (bf16 TTNN UNet) has not been measured.
• The diffusers pipeline needs trust_remote_code=True, which executes this repo's pipeline.py.

Licensing

Read this before redistributing output. The code in this repository is Apache-2.0. The weights it downloads at runtime are not, and this repository cannot grant you their terms:

Using this package means accepting the RAIL-M use restrictions on the weights it fetches, even though this repository does not carry them. The MotionAdapter's publisher does not state its terms. If that matters to your use, resolve it with the upstream author rather than inferring permission from this repo's Apache-2.0 header.

A note on the model tree

The base_model entries make the Hub show stable-diffusion-v1-4 and the AnimateDiff MotionAdapter in this repo's model tree. Read that as "this code resolves those weights at runtime". It does not mean the weights were fine-tuned into a new checkpoint. Nothing here is trained, and the repo ships no weights of its own. The Hub has no relation type for a reimplementation on different hardware, which is what this is.

Changelog

Related packages

• episod/tt-animatediff-demo is the public demo Space (CPU).
• The GitHub repo also carries a v5.1 container manifest (tt_model_package.yaml). It has not been published. Its repo: field currently names this repo; publishing it there would replace this bundle.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/episod/tt-animatediff/discussions. That is the one channel that reaches the bundle's author. Code issues can also go to tenstorrent/tt-animatediff.

For a problem with the tt tooling itself, use tt report issue. It collects your environment and opens a prefilled issue against tenstorrent/tt-cli, so it does not reach this package's author. Send product feedback to [email protected].

Provenance