Qwen2.5-7B-Instruct-CommonSense170K-LoRA

Qwen2.5-7B-Instruct + Commonsense170K — LoRA

This repository contains the LoRA adapter produced by supervised fine-tuning Qwen2.5-7B-Instruct on Commonsense170K. It is the source adapter for the corresponding post-hoc Spectral Surgery HNS 8+2 experiment.

Training

• Base model: Qwen/Qwen2.5-7B-Instruct
• Dataset: Commonsense170K, 170,420 examples
• Epochs: 2 (10,652 optimizer steps)
• Sequence length: 2,048
• Global batch size: 32
• Learning rate: 2e-4, cosine schedule, warmup ratio 0.10
• LoRA rank/alpha/dropout: 16 / 32 / 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Precision: bf16
• Seed: 42
• Chat template: non-thinking

Evaluation

Greedy evaluation on the eight-task commonsense suite, using the tokenizer chat template in non-thinking mode and at most 8 generated tokens.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(
    base,
    "tianzl66/Qwen2.5-7B-Instruct-CommonSense170K-LoRA",
)

Files

• adapter_model.safetensors / adapter_config.json: final PEFT adapter
• run_args.json, run_config.json, training_args.json: training configuration
• eval-commonsense8/: aggregate metrics and per-example LoRA predictions
• comparison-summary.json / .tsv: Base, LoRA, and HNS comparison