egovlm-360m-egopro

EgoVLM-360M-EgoPro

A small vision-language model that looks at one frame of first-person (egocentric) task video and says what the person is doing.

Q: What is the person doing right now?
A: Push the pink chair under the blue table.

It was built from components and trained end-to-end on a laptop (Apple M3 Pro, 57 minutes) on a deliberately tiny slice of EgoPro — 140 episodes, about 23 minutes of video, roughly 0.004 % of that dataset. It is a teaching artefact and a baseline, not a production model. Everything about how it was built is written up in the repository linked below.

Architecture

The standard late-fusion VLM recipe, with only the middle piece trained from scratch:

A 224 px frame becomes 196 patch features; pixel-shuffle merges each 2×2 block into one token (49 tokens of 3072-d), and the MLP projects them into the language model's embedding space. The prompt contains 49 <image> placeholders whose embeddings are overwritten by those visual tokens, so the language model attends over pixels and words in one sequence.

Only the connector and the LoRA adapters are in this repo (16.4 M parameters, 66 MB). The vision tower and the language model are downloaded from their own repositories at load time; nothing is redistributed here.

Usage

pip install torch transformers peft safetensors pillow huggingface_hub

from huggingface_hub import hf_hub_download
import sys, os
sys.path.append(os.path.dirname(hf_hub_download("RastogiAkshat/egovlm-360m-egopro", "modeling_egovlm.py")))
from modeling_egovlm import EgoVLM

model = EgoVLM.from_pretrained("RastogiAkshat/egovlm-360m-egopro").eval()

print(model.ask("frame.jpg", "What is the person doing right now?"))
print(model.ask("frame.jpg", "What task is being performed?"))
print(model.ask("frame.jpg", "Where is this taking place?"))

Runs on CPU; .to("cuda") or .to("mps") for a GPU. About 2 GB of memory in float32.

The model was trained on six question types: the current sub-action, the task name, an episode summary, the task instruction, the setting, and which camera the frame came from. It answers best in that phrasing.

Training

Data. 140 EgoPro episodes across 80 tasks → 5,486 frames sampled at 2 fps from the head camera and the right wrist camera, resized to 448 px. Labels come from the dataset's own time-stamped subtask annotations, so each frame has a ground-truth answer for "what is happening right now". Split by episode (4,622 train / 864 validation), so validation episodes were never seen.

Two stages, the LLaVA/PaliGemma recipe:

Stage 2 is best after one epoch and then overfits slightly — 4.6 k frames is a small set for adapting a language model — so the released checkpoint is the best-by-validation one, not the last.

Batch 16 effective, AdamW, cosine schedule, bf16 off (fp32 throughout), Apple MPS.

Evaluation

150 frames from held-out episodes, three questions each:

Read these honestly:

• Sub-action captions are usable. About a third of predictions are exact or near-exact ("Push the pink chair under the blue table", "Close the drawer"). Most failures name the right kind of action on the wrong object — small objects in cluttered kitchens and offices, seen in a single 224 px frame.
• Place is easy — scene type is a global property of the image.
• Task exact-match is a harsh metric here. Many of the 78 task names are near-duplicates ("Align Chairs", "Align Chairs in a Row", "Align Chair Orientations"), and one frame from a 6-second clip often cannot distinguish them. The model's guesses are consistently in the right category.

During evaluation the model was sometimes right where the label was wrong — on one drying-rack frame the annotation says plastic box and the model said glass cup with a handle, which the frame supports. Any large annotated corpus has some of this, and it caps what these scores can show.

Limitations

• One frame, no motion. A hand closing on a cup looks identical whether it is being picked up or put down. Most "wrong action" errors are this. Feeding a few frames spanning ~1 s is the obvious next step.
• 224 px. Small objects (a spice jar, a sticker) occupy a handful of pixels.
• Tiny training set. 23 minutes of video from one dataset; domains are homes, offices, and light logistics/factory scenes.
• English only, single-turn, no grounding outputs (no boxes or points), no action outputs.
• Not evaluated for, and not suitable for, safety-critical use or any decision about a person.

Where this goes

Replacing the text output with the next half-second of hand trajectory — which EgoPro provides as 21-joint hand poses in every episode — turns this into a vision-language-action model, the class of model behind GR00T and π-series robot policies. The encoder, connector and splicing stay exactly as they are.

Reproduce it

Everything — the data builder, the model, the two-stage trainer, evaluation, and a tutorial written for someone who last touched ML a few years ago — is at https://github.com/AkshatRastogi-1nC0re/vlm-lab.

git clone https://github.com/AkshatRastogi-1nC0re/vlm-lab && cd vlm-lab
pip install -r requirements.txt && hf auth login
./scripts/download_subset.sh && python -m vlm.build_dataset --fps 2
python -m vlm.train --stage 1 --epochs 2 --out checkpoints/stage1
python -m vlm.train --stage 2 --init checkpoints/stage1 --epochs 3 --out checkpoints/stage2

License and data

The connector and LoRA weights and the code in this repository are released under Apache-2.0.

Trained on LightwheelAI/EgoPro, part of EgoSuite-Open100K, whose licence (commercial-training-no-resale-v1.0) permits academic and commercial training while restricting resale of the data. No dataset video, frames, or annotations are included in this repository. Review that licence for your own use. Base model licences: SigLIP2 and SmolLM2 are Apache-2.0.

Citation

@misc{rastogi2026egovlm,
  author = {Rastogi, Akshat},
  title  = {EgoVLM-360M-EgoPro: a laptop-scale vision-language model for egocentric task video},
  year   = {2026},
  url    = {https://huggingface.co/RastogiAkshat/egovlm-360m-egopro}
}