glm4v-sft-lora

GLM-4.6V SFT LoRA

這是基於 GLM-4.6V 108B MoE 模型進行 SFT (Supervised Fine-Tuning) 訓練的 LoRA adapter。

模型資訊

• Base Model: GLM-4.6V 108B MoE (128 experts, 8 active)
• Training Method: QLoRA (4-bit quantization + LoRA)
• LoRA Rank: 64
• LoRA Alpha: 128
• Training Steps: 600
• Max Sequence Length: 4096

訓練配置

• Hardware: 4x NVIDIA H100 80GB
• Precision: BF16
• Optimizer: AdamW
• Learning Rate: 2e-5
• Batch Size: 1 per GPU
• Gradient Accumulation: 8

使用方式

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# 載入基礎模型
base_model = AutoModelForCausalLM.from_pretrained(
    "your-base-model-path",
    device_map="auto",
    torch_dtype=torch.bfloat16
)

# 載入 LoRA adapter
model = PeftModel.from_pretrained(base_model, "HUNGTZE/glm4v-sft-lora")

# 載入 tokenizer
tokenizer = AutoTokenizer.from_pretrained("HUNGTZE/glm4v-sft-lora")

Framework Versions

• PEFT: 0.18.0
• Transformers: 4.x
• PyTorch: 2.x