Downloads · 30 days
0
trajgaze-merge-ckpts
RANSAC123/trajgaze-merge-ckpts
trajgaze-merge-ckpts is a machine learning model from RANSAC123. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Stage-1 trajectory encoder + Stage-2 LoRA-finetuned Qwen2.5-VL-7B-Instruct checkpoints for the TrajGazeMerge project — a trajectory-aware visual token selection method that compresses egocentric visual tokens 10× (mer…
Access
Public
Updated May 28, 2026
Repo size
142 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pth134 GB · 95%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- May 28, 2026
- Updated
- May 28, 2026
- SHA
- 9c73f786
Model card
From the Hugging Face model README
TrajGazeMerge — checkpoints
Stage-1 trajectory encoder + Stage-2 LoRA-finetuned Qwen2.5-VL-7B-Instruct checkpoints for the TrajGazeMerge project — a trajectory-aware visual token selection method that compresses egocentric visual tokens 10× (merge_ratio = 0.9) while keeping the kept tokens task-relevant.
Code: https://github.com/Jaewoo97/trajgaze (branch cf-mask-augmented-training).
Paper draft: docs/NeurIPS_2026_Gaze_hand_Trajectory_Merging_for_Efficient_Egocentric_Video_Understanding.pdf in the source repo.
Best result (one sentence)
TAS-only is best in both 2-dataset (StreamGaze + EgoGazeVQA) and 3-dataset (+ HD-EPIC) setups. Adding ATR / CGM losses hurts mean accuracy; counterfactual-mask diagnosis (see eval_results/) shows these extensions do not change LLM language-prior usage on EgoGazeVQA. See the code repo's docs/paper_narrative_v3.md for the mechanism-by-role reading.
| Setup | Method | StreamGaze | EgoGazeVQA | HD-EPIC | mean |
|---|---|---|---|---|---|
| A (2 ds) | TAS-only ★ | 67.49 | 57.77 | — | 62.63 |
| A | TAS+ATR+CGM (FULL) | 61.98 | 59.40 | — | 60.69 |
| A | CGM-only | 62.74 | 55.92 | — | 59.33 |
| B (3 ds) | TAS-only-hdepic ★ | 63.69 | 55.92 | 50.12 | 56.57 |
| B | TAS+ATR-hdepic | 60.65 | 54.76 | 50.66 | 55.35 |
| B | CF-1 (cf-mask aug, early-stopped) | — | — | — | ~54.5 |
| B | CF-3 (cf-mask + shuf aug, early-stopped) | — | — | — | ~50.0 |
Files
stage1/ — Trajectory encoder (Stage-1 pretrained)
| File | Size | Role |
|---|---|---|
stage1/E1_combined_AB_TAS.pth | 142 MB | A + B + TAS encoder (Stage-1 supervised). Required as --stage1-ckpt for every Stage-2 / cf-mask run. |
stage2/ — LoRA-finetuned Stage-2 (Qwen-VL + TAS encoder fine-tune)
LoRA: r=16, α=32, target_modules=[q_proj, k_proj, v_proj, o_proj]. Each .pth contains lora_state, encoder_state, and (when applicable) atr_state.
| File | Size | Stage-2 loss | Setup | Notes |
|---|---|---|---|---|
stage2/E1_combined_TAS_only.pth ★ | 16 GB | CE only | A | Headline (Setup A best). |
stage2/E1_combined_TAS_ATR.pth | 16 GB | CE + ATR | A | Ablation. |
stage2/E1_combined_TAS_ATR_CGM.pth (FULL) | 16 GB | CE + ATR + CGM | A | Ablation; EgoGazeVQA mask_kept sign-flip but StreamGaze regress. |
stage2/E1_combined_CGM_only.pth | 16 GB | CE + CGM | A | Ablation; trained from earlier non-TAS Stage-1. |
stage2/E1_combined_TASonly_hdepic.pth ★ | 16 GB | CE only | B | Setup B best (+ HD-EPIC). |
stage2/E1_combined_TAS_ATR_hdepic.pth | 16 GB | CE + ATR | B | Setup B ablation row. |
stage2/E1_combined_cf1_hdepic.pth | 16 GB | CE + cf-mask margin | B | Direction A (CF-1). Training stopped early at step ~6500. |
stage2/E1_combined_cf3_hdepic.pth | 16 GB | CE + cf-mask + shuf margins | B | Direction A (CF-3). Training stopped early at step ~6500. |
eval_results/ — counterfactual-mask summaries + per-sample parquets
Used to reproduce the 4×2 cf-mask matrix in docs/visual_grounding_diagnosis_v2.md. Each ckpt has per-variant *_mask_summary.json + *_per_sample.parquet for {streamgaze, egovqa}.
How to use
- Clone the code:
git clone https://github.com/Jaewoo97/trajgaze && cd trajgaze && git checkout cf-mask-augmented-training. - Place the Stage-1 ckpt at
TrajGaze_v2/checkpoints/E1_combined_AB_TAS/best.pth. - Place each Stage-2 ckpt at
TrajGazeMerge/checkpoints/<run-name>/best.pthmatching the file name above (drop the.pthsuffix in the dir name; the file inside is alwaysbest.pth). - Run cf-mask diagnostic (no training needed for inference):
PYTHONPATH=. python -m TrajGazeMerge.eval.counterfactual_mask_eval \ --stage1-ckpt TrajGaze_v2/checkpoints/E1_combined_AB_TAS/best.pth \ --lora-ckpt TrajGazeMerge/checkpoints/E1_combined_TAS_only/best.pth \ --val-dataset streamgaze --tag my_run
See repo README.md for full training commands (including CF-1, CF-3 reproduce commands in §4d).
- License
- apache-2.0
- Created
- May 28, 2026
- Updated
- May 28, 2026