hunmin-vlm-0.12

Hunmin-VLM 0.12

A Vision-Language Model built on GLM-4V-9B with a custom projector and LoRA fine-tuning using ORPO (Odds Ratio Preference Optimization).

Model Details

• Base Model: THUDM/glm-4v-9b (GLM-4.7-Flash)
• Training Method: 4-stage progressive training (projector → LoRA SFT → LoRA ORPO)
• Image Tokens: 288 per image
• Hardware: 1 node × 8 NVIDIA B200 GPUs
• Framework: PyTorch + HuggingFace Transformers ≥5.0.0rc0

Training Stages

This model was trained in 4 progressive stages. Each stage is available as a separate branch.

Stage Details

• Stage 0–1 (Projector): Only the vision-language projector is trained. The LLM backbone and vision encoder are frozen. Output: projector_state.pt (~17MB).
• Stage 2 (LoRA SFT): The projector is frozen. LoRA adapters are applied to the LLM for supervised fine-tuning on VLM conversation data. The full merged model is saved (~76GB).
• Stage 3 (LoRA ORPO): Same as stage 2, but trained with ORPO preference optimization. Text ORPO pairs are mixed in at 10% ratio. Evaluation every 100 steps with early stopping (patience=5).

Training Data

• VLM pairs: mncai/orpo-vlm-pairs-full — multimodal preference pairs
• Text pairs: mncai/orpo-text-pairs-full — text-only preference pairs (10% mix in stage 3)

Training Logs (W&B)

Full training metrics are publicly available:

• Project: ttagu99/hunmin-vlm-0.12

Branch Usage

# Final model (stage3 ORPO)
git clone https://huggingface.co/mncai/hunmin-vlm-0.12

# Specific stage
git clone -b stage2 https://huggingface.co/mncai/hunmin-vlm-0.12

Or with Python:

from huggingface_hub import snapshot_download

# Final model
snapshot_download("mncai/hunmin-vlm-0.12")

# Specific stage
snapshot_download("mncai/hunmin-vlm-0.12", revision="stage1")

Included Training Files

License

Please contact mncai for licensing information.