egolongqa-2b-distill-adapter

EgoLongQA 2B — distilled junior perception module (LoRA)

🏆 LoRA adapter for Qwen/Qwen3.5-2B, from Team Ambient's entry to the EgoLongQA track of the Wearable AI Challenge @ ECCV 2026 — 1st place, small division.

What it does

Given ~100 uniformly sampled frames spanning a ten-minute egocentric video, plus a four-option question, the model answers in one greedy forward pass — no retrieval, no tool calls, no ensemble. It emits a timestamped description, then a structured answer:

<video_description>…</video_description>
<answer><choice>C</choice><reason>…</reason><citation>…</citation></answer>

This is the perception step of a tool-using agentic pipeline, distilled out of it. The agent's senior orchestrator — which retrieves clips and revises over multiple turns — is discarded, not imitated: its value is the multi-turn loop, which no single forward pass can express. Collapsing the pipeline this way takes per-sample latency from ~393 s to ~32 s.

Usage

import torch, json
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
from PIL import Image

BASE = "Qwen/Qwen3.5-2B"
proc  = AutoProcessor.from_pretrained(BASE)
model = PeftModel.from_pretrained(
    AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="cuda"),
    "infinitylogesh/egolongqa-2b-distill-adapter").eval()

query = ("Provide the detailed video description and answer to the question now: "
         f"Question:{question} {mcq_options}")
msgs = [{"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]},
        {"role": "user",   "content": [{"type": "image"} for _ in frames]
                                      + [{"type": "text", "text": query}]}]
text = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
batch = proc(text=[text], images=[Image.open(p).convert("RGB") for p in frames],
             return_tensors="pt", images_kwargs={"max_pixels": 331776, "min_pixels": 3136}
             ).to(model.device)
out = model.generate(**batch, max_new_tokens=8192, do_sample=False)

Three settings are not free parameters:

• max_pixels=331776 at inference, though the adapter was trained at 50176. Training cheap and inferring high is worth +5.8 points; the adapter learns the task, not the resolution.
• max_new_tokens=8192. At the common default of 768, 3–9 % of completions are cut mid-description, never emit <answer>, and score wrong regardless of what the model concluded.
• 100 frames. A sweep at 100/200/400 showed 400 frames costing ~4.4 points at this scale.

SYSTEM_PROMPT is prompts/junior_system.txt in the code repo — 2000 bytes, md5 f342eb86cceb9ee18c9fba0ceae9f013. It carries two trailing spaces; a copy with them stripped changed 10 of 70 held-out answers.

Frames must be extracted the same way the traces were, or the model sees different moments than it was trained on:

ffmpeg -i VIDEO -vf "fps=min(2,100/DURATION),scale=768:768:force_original_aspect_ratio=decrease" \
  -q:v 3 frames/%03d.jpg      # take the first 100

Fitting under 2B

Merged, this is 2.2132 B — over the divisional limit, and the embedding table is 23 % of it (248,320 rows × 2048, weight-tied). Pruning the vocabulary to 143,469 rows yields 1.9985 B with a measured logit difference of exactly 0 on retained rows and byte-identical generations.

The prune is reproducible from this adapter: merging and pruning with the keep-set in the code repo regenerates the submitted model.safetensors bit-for-bit (md5 896c434c402450f57315daee869bc68e).

The output side is not lossless. Input-side pruning has a byte fallback; the output side has none — if the argmax lands on a removed row the model emits a different word. Capture the keep-set on the GPU that will serve the model.

Training

Distilled from 2,936 teacher traces over 605 videos (median 5 distinct descriptions per question), harvested from agentic runs and filtered to keep only passes whose <choice> matched ground truth — so the student imitates teacher parity, not teacher average. All teachers are open-weight (gemma-4-31b-it, qwen3.5-122b-a10b, Qwen3.6-27B/35B).

LoRA r=16 α=32, 1 epoch (epoch 2 was worse in all five runs), cosine schedule, batch size 1 × 8 accumulation, 3× token weight on the <choice> span, max_pixels=50176, bf16.

• Code: https://github.com/ambient-intelligence-hq/egolongqa-2b
• Data: infinitylogesh/egolongqa-synth-annotations

Citation

@techreport{umapathi2026egolongqa,
  title  = {Ambient @ EgoLongQA 2026: Distilling Perception, Not Orchestration,
            into a Sub-2B Model},
  author = {Umapathi, Logesh Kumar},
  year   = {2026},
  institution = {Team Ambient},
  note   = {Wearable AI Challenge @ ECCV 2026, EgoLongQA track}
}