110611065_DL_kaggle-1

NYCU Kaggle1 Qwen2.5-72B QLoRA Adapter

This repository contains the PEFT LoRA adapter trained for a Kaggle multiple-choice answer-only SFT task.

Model

• Base model: Qwen/Qwen2.5-72B-Instruct
• Fine-tuning method: 4-bit NF4 QLoRA
• LoRA rank: 64
• LoRA alpha: 128
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Training objective: answer-only SFT, where only the final A/B/C/D answer token contributes to loss

Training Run

qwen25_72b_qlora_answer_only_sft_20260510_175911

Training config and logs are available in the GitHub repository under:

artifacts/training_logs/qwen25_72b_qlora_answer_only_sft_20260510_175911/

Files

The main adapter file is:

adapter_model.safetensors
size: 3,368,703,728 bytes
sha256: CE03837B223AA12B583CE79A767E6A6FEECE062D15C1632449D8A12E85210146

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

base_model = "Qwen/Qwen2.5-72B-Instruct"
adapter = "Atziluth98061/110611065_DL_kaggle-1"

tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
    ),
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, adapter)

For reproducible inference, use the project scripts in the GitHub repository.