Downloads · 30 days
3
14% of all-time downloads
bi199797/VR-egodex-qwen36-27b-lora-caption-boundary
VR-egodex-qwen36-27b-lora-caption-boundary is a machine learning model from bi199797. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as other.
LoRA adapter for Qwen/Qwen3.6-27B (qwen35 architecture, hybrid GatedDeltaNet + full-attention), fine-tuned to re-time an already-known ordered list of action segments: given the video plus the segments' text (no timin…
Downloads · 30 days
3
14% of all-time downloads
All-time downloads
21
Public
Repo size
821 MB
Likes
0
Public
Click a slice to open those files.
.safetensors801 MB · 98%
From the Hugging Face model README
LoRA adapter for Qwen/Qwen3.6-27B (qwen3_5 architecture,
hybrid GatedDeltaNet + full-attention), fine-tuned to re-time an already-known ordered list of
action segments: given the video plus the segments' text (no timing), it predicts the correct
MM:SS.d - MM:SS.d for each, in order.
This is not an error-correction model trained on (correct caption, correct timestamp) pairs --
feeding the true timestamp back in as input would let it learn to just copy it through without
looking at the video. Instead its training target strips timestamps out of the input entirely
(stage_2_2_timestamp_only_with_caption/scripts/build_caption_conditioned_manifest.py), so there
is nothing to copy and the only way to answer is real visual grounding. It is the second stage of
a two-stage pipeline: a captioning model (e.g.
bi199797/VR-egodex-narration-qwen36-27b-lora-vision12-lr1e5)
supplies the what, this model supplies the when -- conditioning on the given text takes the
in-loop timestamp score from 0.6048 (the
video-only boundary sibling,
no text at all) to 0.6868 on the same eval sample.
Training code: VR-finetune-VLM, stage_2_2_timestamp_only_with_caption/.
| Base weights | bf16 (full precision, not quantized) |
| Adapter | LoRA, rsLoRA scaling |
lora_r | 32 |
lora_alpha | 64 |
lora_dropout | 0.05 |
| Target modules | language-model attention/MLP only (q/k/v/o_proj, gate/up/down_proj) -- no vision-tower adaptation |
| Trainable params | ~200.3M / 27.6B (0.73%) |
| Data | 19,672 EgoDex episodes (stage_2_2_timestamp_only_with_caption/data/manifests/train.jsonl) -- video + the episode's own ordered, timestamp-free action-segment text folded into the prompt as episode context, target is the ground-truth MM:SS.d - MM:SS.d for each listed segment |
| Prompt | stage1/prompts/boundary_with_caption/v1.txt -- "find the correct start and end time for each listed action segment" |
| Epochs | 2 (1,230 steps) |
| Batch size | 1 x 16 grad-accum = 16 effective |
| Learning rate | 1e-4, cosine schedule, warmup 3% |
| Weight decay | 0.01 |
| Sampling | fps=2.0, max_frames=128, max_pixels=272384/frame |
| Max sequence length | 40,960 |
| Precision | bf16 |
| Cluster | 2x NVIDIA H100 80GB (device_map=auto) |
timestamp axis: cue-boundary IoU only, no text)In-loop eval logged during training (stage2/train.py's periodic callback, 24-episode sample
of stage_2_2_timestamp_only_with_caption/data/manifests/val.jsonl) -- not yet the full
300/549-episode generation sweep the narration cards report.
| step | timestamp_score | eval_loss |
|---|---|---|
| 100 | 0.6664 | -- |
| 600 | 0.6587 | -- |
| 1000 | 0.6903 | -- |
| 1100 (peak) | 0.6983 | -- |
| 1230 (final) | 0.6868 | 0.1259 |
Consistently 0.65-0.70 across the whole run -- well above the video-only boundary sibling's 0.52-0.60 band on the same schedule, which is the intended effect of conditioning on the given caption text rather than localizing blind.
Not yet measured end to end: whether re-timing a predicted caption (from a captioning
model that gets some segments wrong) holds up as well as re-timing the ground-truth segments
this checkpoint was trained and evaluated against here. A two-stage pipeline run
(stage_2_2_timestamp_only_with_caption/scripts/run_pipeline_deakin.sbatch) chaining this
checkpoint after a captioning model's own predictions, scored on the full test/val splits, is
in progress.
This model expects the video plus the ordered segment text folded into the prompt as episode context -- it is not a plain video-to-timestamp model (see the boundary sibling for that).
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor
base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "bi199797/VR-egodex-qwen36-27b-lora-caption-boundary")
processor = AutoProcessor.from_pretrained("bi199797/VR-egodex-qwen36-27b-lora-caption-boundary")
Build the prompt with the system prompt at stage1/prompts/boundary_with_caption/v1.txt, plus
episode context ending in:
## Predicted action segments (in order; find the correct start/end time for each)
1. [left hand] reach toward the small object | [right hand] rest on the table | [ego] stay still
2. [left hand] grasp and lift the small object | [right hand] hold the container steady | [ego] stay still
...
The model returns exactly one MM:SS.d - MM:SS.d line per listed segment, in the same order,
first line starting at 00:00.0, last line ending at the video's final frame.