qwen3.5-9b-4by4-consistency-merged

Qwen3.5-9B — Four-Head Temporal Ordering (LoRA-merged)

Qwen/Qwen3.5-9B with our LoRA adapter (r=16, alpha=32, targets q,k,v,o,gate,up,down_proj) merged into the base weights (PeftModel.merge_and_unload()). This is the backbone for our SNU AI Challenge submission: given four shuffled video frames and a one-sentence description, predict their chronological order.

The vision encoder is unmodified from the base model (it was frozen during training — no LoRA was applied to it, so its weights here are byte-identical to Qwen/Qwen3.5-9B).

This repo is the backbone only — you also need the rank heads

This model was trained with four independent rank-classification heads read from the last non-padding prompt hidden state (see our GitHub repo for the full source: code/modeling.py, class FourIndependentRankHeads). Those heads are not part of the standard AutoModelForImageTextToText weights — they are a small separate checkpoint bundled in this repo as direct_head.pt (~200 KB), together with direct_objective.json (answer-encoding metadata) and ablation_manifest.json / train_args.json (exact training configuration, ablation preset consistency).

Loading this repo with plain AutoModelForImageTextToText.from_pretrained(...) gives you a fully functional causal LM + vision backbone, but not an ordering predictor by itself — you additionally need to load direct_head.pt onto the last hidden state as documented in our GitHub repo's code/modeling.py (load_head, four_head_rank_logits) and decode by scoring the 24 valid permutations in closed form (rank_logits_to_permutation_scores).

Full training/inference code, the LoRA-adapter (unmerged) version, and the paper describing the method: see the linked GitHub repository.

Why a merged copy, in addition to the LoRA adapter

We also publish the un-merged LoRA adapter (58 MB) in our GitHub repo. This merged copy exists because folding the adapter into the base weights removes the extra low-rank matmul branch PEFT adds at every one of the 7 adapted projections in every decoder layer, which matters most at batch size 1 (our inference setting): with no batching to amortize the extra kernel launches, that per-layer LoRA branch is a proportionally larger tax than in typical large-batch benchmarks. Our 24-view test-time augmentation runs the language model forward once per permutation, so removing that per-call overhead compounds over the 24 calls. We have not published a controlled side-by-side latency number for this specific model/hardware pair; treat the merged copy as an inference-convenience artifact (also: no peft dependency needed to load it), not as a benchmarked speedup claim.

Training data / method summary

• Task: SNU AI Challenge frame-ordering (4 frames -> permutation of 1..4)
• Backbone: Qwen3.5-9B, vision encoder frozen throughout
• Adaptation: LoRA r=16/alpha=32/dropout=0, merged here
• Head: 4 independent per-slot rank classifiers over the last prompt hidden state (LayerNorm -> Dropout -> Linear), decoded by closed-form scoring of all 24 valid permutations (always a valid output, no autoregressive decode)
• Losses: rank cross-entropy + induced 24-way permutation CE + constrained- position NLL + pairwise BCE + soft permutation-distance CE + expected Hamming/Kendall distance + two-view permutation-consistency divergence (ablation preset consistency; see ablation_manifest.json)
• Validation (954 held-out rows, seed 42): 58.5% exact match, 71.8% position accuracy, 83.2% pairwise accuracy (see eval_metrics.json)

License

Follows the base model's license (Qwen/Qwen3.5-9B).