c3-3b-vanilla-stage1

C3 — Vanilla · Stage 1 (align) · Qwen2.5-VL-3B

Stage 1 align output for the C3 (QK-norm injection) experiment. Trains only the LLaVA-style mm_projector against a frozen LLM + frozen vision encoder.

What's in this repo

• mm_projector.bin — trained projector weights (no QK-norm — vanilla Qwen2.5 attention).
• Qwen2.5-VL tokenizer / image processor / video processor configs.
• effective_config.json — the merged YAML used at train time.

How to use

This repo contains the Stage 1 deliverable only. To run Stage 2 instruction tuning on top of it, point --pretrain-projector at this folder (mirroring the c3_vanilla_stage2.yaml recipe in minsik-choi126/mllm-model-merging).

torchrun --nproc_per_node=N -m training.train.cli \
    --config training/configs/3b/c3_vanilla_stage2.yaml \
    --pretrain-projector ckpts/c3/3b/vanilla_align

Resuming from an in-progress Stage 1

If you need DeepSpeed optimizer / RNG state to resume training, see the -ckpt variant: CHOI0126/c3-3b-vanilla-stage1-ckpt.

Training recipe

• Base: Qwen/Qwen2.5-VL-3B-Instruct (LM swapped to a from-scratch composed init — see build_init_from_pretrained in the repo).
• Data: liuhaotian/LLaVA-Pretrain (BLIP/LAION/CC/SBU 558K caption pretrain).
• Resolution: 336² (max_pixels: 112896).
• Effective batch: 128 (per-device bs 32 × 2 GPU × ga 2).
• Optimizer: AdamW, lr 1.0e-4, constant_with_warmup (warmup 3%).
• Hardware: 2× A6000-48GB.