gaze-lle-p150

gaze-lle-p150

Gaze-LLE (gazelle_dinov2_vitb14_inout: DINOv2 ViT-B/14 backbone + gaze decoder) port on one Tenstorrent Blackhole p150a. Weights: changh95/gaze-lle-weights · Paper: arXiv:2412.09586 · Upstream code: fkryan/gazelle · Port: changh95/tt-Gaze-LLE

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart (Python)

Prerequisite: a Python environment with a built tt-metal / ttnn at tt-metal 8b98410e730 with patches/tt-metal-eth-dispatch.patch applied.

hf download changh95/gaze-lle-p150 --exclude "image/*" --local-dir gaze-lle-p150 && cd gaze-lle-p150
pip install -e code/                  # the Python API only
pip install -e "code/[server,test]"   # also the HTTP server (fastapi, uvicorn, pyspng, orjson) and the tests (pytest, httpx)

from gaze_lle import GazeLLE, visualize

with GazeLLE.from_pretrained() as model:                  # device 0: weights from the HF cache, traces captured
    result = model("media/source_1.png",                  # path, PIL image, numpy or torch array
                   [[116, 28, 176, 104], [269, 37, 301, 81], [87, 57, 107, 76]],  # one head box per person
                   bbox_format="pixels")                  # default "normalized" (fractions of W/H, upstream)

print(result.gaze_xy, result.inout)    # (N, 2) gaze targets in pixels, (N,) P(target in frame)
result.heatmap                          # torch (N, 64, 64) heatmaps over the whole image
visualize("media/source_1.png", result).save("gaze.png")

• The relative path media/source_1.png is correct when the current directory is the model repo root (gaze-lle-p150). From a different directory, use the full path.
• from_pretrained downloads the pinned weights changh95/gaze-lle-weights to the HF cache. It opens the chip with dispatch on ETH cores, 1 command queue and the 12×10 compute grid (the p150 configuration).
• It captures the metal traces for 1, 2, 3, 4, 6, 8 and 10 heads and runs 2 warm-up forwards. A cold start takes approximately 1 minute.
• from_pretrained() prepares all head counts from 1 to 10 and all input types (array, PIL image, PNG or JPEG file, float array, batch). Thus, the first call is as fast as the next calls. The start-up time is approximately 5 s (approximately 1 min on the first run, which compiles the kernels).
• To prepare more head counts, use warmup_variants="all" or model.warmup(heads=[11]). Use max_heads of 11 or less.
• Without the ETH-dispatch patch, the model uses the default dispatch and shows a RuntimeWarning. The results are identical, but the calls are slower. On a p150, this fallback has an 11×10 grid.
• The with block releases the traces, the weights and the chip at the end. You can also call model.close().
• The output is identical to the HTTP /predict response for the same image and boxes (tested bit for bit).

Throughput helpers:

• model.predict_batch(images, bboxes, num_workers=2) runs many images. Host threads decode and resize the next images while the device runs. A list of images in model(...) does the same.
• A (448, 448, 3) uint8 array goes to the device without the host resize. The "Python model() call" row below measures this input.
• gaze_lle.open_device() and from_pretrained(device=...) let you share one open chip with other code.
• One device call runs at a time, so you can call the model from more than one thread.

code/PYTHON.md gives the full API reference. code/examples/quickstart.py runs the snippet on the demo image and writes gaze_quickstart.png and gaze_quickstart.json.

Serving (HTTP)

tt-model pull  changh95/gaze-lle-p150 --with-weights
tt-model serve changh95/gaze-lle-p150      # or, with tt-cli: tt serve changh95/gaze-lle-p150
printf '{"image":"%s","bboxes":[[116,28,176,104],[269,37,301,81],[87,57,107,76]]}' "$(base64 -w0 media/source_1.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/gaze-lle-p150       # tt-cli

• Weights changh95/gaze-lle-weights go to your HF cache; the image does not contain them.
• Serves on port 20000 (or the next free port); ready when the log says Application startup complete.
• POST /predict: image (base64 PNG/JPEG), bboxes (1–10 head boxes [x1,y1,x2,y2]); optional bbox_format (pixels | normalized), heatmap_format (png | npz | json | none).
• GET /health, GET /info.
• The HTTP default for bbox_format is pixels. The Python default is normalized.

{"model": "gaze-lle-p150", "image_size": [500, 334], "input_size": [448, 448], "num_heads": 3,
 "heatmap_size": [64, 64], "heatmap_format": "png",
 "predictions": [
   {"bbox": [116.0, 28.0, 176.0, 104.0], "bbox_normalized": [0.232, 0.083832, 0.352, 0.311377], "inout": 0.025,
    "gaze_target": {"x": 289.1, "y": 172.2, "x_norm": 0.578, "y_norm": 0.516, "heatmap_row": 33, "heatmap_col": 37, "score": 0.373},
    "heatmap_png": "..."}, "..."],
 "timing_ms": {"inference": 6.38, "total": 14.77}}

• inout, gaze_target.x/y and score are the same values as the Python inout, gaze_xy and score. x = col/64 · W (upstream convention).
• heatmap_png is a base64 8-bit grayscale PNG (uint8(round(p·255))); npz returns one base64 .npz (heatmap float32 (N,64,64), inout float32 (N,)) in heatmap_npz.

Demo

GazeFollow test-set scenes; boxes come from RetinaFace, each same-coloured arrow/× is that person's predicted gaze target.

Demo & Performances

Warm, batch 1, media/source_1.png resized to 448×448, pretrained weights, 1 / 3 / 10 head boxes, scripts/profile_e2e.py, 2 runs of 200 iterations (median / min). Measured on 2026-10-05 with dispatch on ETH cores, 1 command queue and a 12×10 compute grid (the p150 configuration).

The Python row ran on a busy host (load average 9–19). The bare model call in the same loop had the same median excess, so this excess is host noise. The API adds 0.02–0.10 ms (median) to the bare model call. The Python minimums are below the published medians.

The measurement hardware is a Blackhole chip in the p150 configuration: dispatch on ETH cores, 1 command queue and a 12×10 compute grid. The 2026-10-03 chip has an x1 PCIe link. On an x8 link, the H2D takes 0.050 ms. Accuracy on the full GazeFollow test set (4,782 images, served uint8 path, 1 head): AUC 0.9543, Avg L2 0.1126, Min L2 0.0505 (gates: AUC ≥ 0.9530, Avg L2 ≤ 0.1130; torch fp32 reference 0.9543 / 0.1103 / 0.0491). Details: VERIFICATION_2026-10-03.md (the section "p150 ETH-dispatch compliance (2026-10-05)" gives the 2026-10-05 run).

RTX 5090 reference measurements (2026-09-14) are unchanged. They use the port's torch reference in PyTorch 2.11, batch 1, with H2D/D2H included in the "incl. H2D" columns. The GPU gets a float 448×448 tensor. Our values are the mean of the two 2026-10-05 run medians. Full table: GPU_COMPARISON.md.

The device lead comes from the metal traces. They run the whole forward with fused custom kernels on all 120 compute cores. The linears write and read the SDPA head layout directly and add the residual in the same kernel. Part of the end-to-end lead comes from the input. This build uploads 602 KB of uint8 pixels and normalises them on the device. With the same float input as the GPU, the lead over the fastest GPU row is 1.04× (3.604 vs 3.754 ms, 2026-10-03 run). The previous release was 1.52× slower than that GPU row.

Caveats

• Does not scale to multiple p150a in a mesh configuration. The current build uses a 12x10 compute grid of Tensix cores. To get this grid, the dispatch functions move from 10 Tensix cores to ETH cores (patches/tt-metal-eth-dispatch.patch). Thus, this build assumes that you do not need chip-to-chip ethernet communication.
• dispatch="worker" (Python) or GAZE_LLE_DISPATCH=worker (HTTP server) is an opt-in mode for a Galaxy chip only. It puts dispatch on Tensix cores, so a p150 gets an 11×10 grid. The published numbers do not use it.
• Every image is squashed to 448×448; one image per request, 1–10 head boxes. All heads share one backbone pass; the gaze decoder runs all heads in one batched, metal-traced pass (head-count buckets 1,2,3,4,6,8,10 are captured at boot, so N=5 runs the 6-head trace).
• bf16/bfp8 on device. The full-set Avg L2 is 0.1126. This is 0.0023 above the torch fp32 reference and 0.0007 above the previous release (0.1119). The margin to the 0.1130 gate is 0.0004. Some optimizations change numerics (for example, the LoFi decoder linears at 1 head). OPT_REPORT.md lists each change.
• Heatmap PCC vs torch on media/source_1.png is 0.9951 (1 head), 0.9825–0.9950 (3 heads) and 0.9861 (10 heads). The argmax can move on very small or off-face boxes. The float-input path is not the served path, and one of its 3 heads has a lower PCC (0.8904).
• TT_FUSED=0 in the container env restores the original eager per-head path.
• Only the vitb14 checkpoint is ported; hardware: p150 only.
• Not an OpenAI-compatible API; GET /v1/models is a stub so the tt-model ready card does not 404.
• Validated on tt-metal v0.78.0-dev20260820 (main 8b98410e730) with patches/tt-metal-eth-dispatch.patch.
• p150a power was not measured, so no efficiency comparison is made.

Licensing

• Weights: changh95/gaze-lle-weights mirrors the two upstream checkpoints unchanged — Gaze-LLE decoder MIT, DINOv2 ViT-B/14 backbone Apache-2.0.
• Port and serving code (code/): from changh95/tt-Gaze-LLE; source files carry SPDX-License-Identifier: Apache-2.0 headers and the repo is published under the upstream MIT terms.

Provenance

These are the exact sources the container image was built from. code/ has since been updated (2026-10-03 optimized build; see OPT_REPORT.md) and is newer than the image. The 2026-10-04 update adds the gaze_lle Python package (code/pyproject.toml, code/PYTHON.md). It does not change the device code or the numerics. The 2026-10-05 update makes the HTTP server, the benchmark and the test scripts in code/ open the chip with dispatch on ETH cores, 1 command queue and the 12×10 grid by default (before, the server used Tensix dispatch). The outputs are bit-identical. tt-model serve runs the image's code until the image is rebuilt. tt-model.yaml and SERVING.md still describe the image: