tars-opa-dpo-llava15-7b

TARS trained on OPA-DPO preference pairs (LLaVA-1.5-7B)

The TARS DPO method trained on OPA-DPO's released preference pairs, then evaluated on AMBER with OPA-DPO's evaluation pipeline.

This is a cross-method study: the training method is TARS, the data is OPA-DPO's release. It is not OPA-DPO, and it is not the TARS paper's own released model.

Results — AMBER generative task

1004/1004 AMBER images, mean response length 121.4 words (median 99).

Training: 228/228 steps, 3 epochs, global batch 64, train_loss 0.7346, final-step loss 0.4395, reward margins ~0.92, reward accuracy 1.0. 7 h 19 m on 2× NVIDIA B200.

Note on an earlier, incorrect number. A first evaluation reported CHAIR 2.9 / Cover 37.1 / Hal 8.9 / Cog 0.9. Those values are invalid — a prompt-slice bug silently truncated the opening of every response (see "Evaluation-path defects" below). The invalid artifacts are kept in results/ as *_TRUNCATED_INVALID.* for transparency. Use only the table above.

Contents

./                              final trained model (directly loadable)
checkpoints/checkpoint-76/      epoch 1 weights
checkpoints/checkpoint-153/     epoch 2 weights
checkpoints/checkpoint-228/     epoch 3 weights (== final)
results/answers.jsonl           1004 AMBER responses (valid)
results/metrics.log             AMBER scores (valid)
results/*_TRUNCATED_INVALID.*   superseded, buggy-decode artifacts
results/RUN_NOTES.md            full run write-up
patches/                        the code fixes this run required
logs/                           all SLURM logs (incl. the failed attempts)
scripts/                        the sbatch scripts used
dataset_manifest.json           data provenance + SHA-256
checkpoints/checkpoint-*/global_step*/
                                DeepSpeed ZeRO-3 optimizer states (245 GB)

The global_step* optimizer states are included (~82 GB per checkpoint). They are only useful for resuming training, and resuming requires the same DeepSpeed ZeRO-3 topology used here (2 ranks). To simply load or evaluate the model, ignore them and use the safetensors weights.

Data

Built from OPA-DPO's released LLaVA-1.5-7B rollouts (subsets 1 and 2):

• chosen = AI_pseudo_response (GPT-4V correction)
• rejected = original_generate_response (original LLaVA-1.5-7B)

616 rollout files → 4,928 raw pairs → 4,907 after OPA-DPO's released filters (19 empty_report, 2 terminal_repetition, 0 empty_correction; 309 identical pairs retained, as the official builder does). Parquet SHA-256 42210f0348669ca1a2896e3741e5f682f7ea6ccee67892da632b3de87b20afb0.

The preference data itself is not redistributed here — it belongs to the OPA-DPO authors. AMBER images likewise belong to the AMBER authors (COCO-derived).

Reproduction notes / deviations

Unchanged from the TARS release: the pinned commit b2e62881bb39dd28a0d12b81d5bcbdf033f7fa9d, the TARS loss, scripts/zero3.json (including CPU offload), and every published hyperparameter (3 epochs, lr 5e-7, cosine, warmup 0.03, max_grad_norm 20.0, model_max_length 2048, bf16, gradient checkpointing, use_image_type diffusion, diffusion_step 500, tok_beta 0.1), plus transformers==4.41.2 / accelerate==0.30.1 / peft==0.12.0.

Forced changes, all hardware- or packaging-driven:

1. torch 2.3.0 → 2.7.1+cu128 — the GPUs available were Blackwell (sm_100); torch 2.3.0 has no sm_100 kernels.
2. FlashAttention-2 → PyTorch SDPA — FA2 ships no sm_100 kernels. SDPA is exact attention, so the math is unchanged; only the kernel differs.
3. deepspeed 0.14.2 → 0.15.4 — 0.14.2 cannot import under torch 2.7.1. 0.15.4 was chosen over 0.16.x because 0.16 added DeepSpeedEngine.no_sync(), which asserts under ZeRO-3 and breaks accelerate 0.30.1's accumulation path; 0.15.4 has no such method, reproducing 0.14.2's behaviour.
4. transformers one-line int() cast for DeepSpeed's "auto" stage3_prefetch_bucket_size (a float in 4.41.2, rejected by pydantic v2); backports the fix upstream shipped in 4.45.
5. 2 GPUs × grad-accum 32 instead of 8 × 8, preserving global batch 64.

Evaluation-path defects

Evaluating a TARS-trained model with OPA-DPO's AMBER scripts hits three incompatibilities. All three are in patches/:

1. image_processor is None — LLaVA's load_pretrained_model only builds a vision tower when 'llava' in model_name.lower(). Keep llava in the checkpoint directory name.
2. generate() signature mismatch — OPA-DPO passes input_ids=; TARS names that parameter inputs, so it lands in **kwargs and inputs stays None.
3. Prompt-slice truncation — OPA-DPO decodes output_ids[:, input_token_len:], assuming generate() returns prompt + completion. TARS returns only new tokens, so this discarded the start of every response. This fails silently and produces plausible-looking metrics; it is the bug behind the invalid numbers noted above.

Usage

from llava.model.language_model.llava_llama import LlavaLlamaForCausalLM
from transformers import AutoTokenizer

# NOTE: the local directory name must contain "llava" for LLaVA's loader
model = LlavaLlamaForCausalLM.from_pretrained(
    "ToiTenBao/tars-opa-dpo-llava15-7b", torch_dtype="bfloat16",
    attn_implementation="sdpa",
)
tok = AutoTokenizer.from_pretrained("ToiTenBao/tars-opa-dpo-llava15-7b", use_fast=False)

Citation

Please cite the original TARS, OPA-DPO, LLaVA and AMBER works. This repository only combines them; it introduces no new method.