Qwen3-1.7B-Chat-LoRA

Qwen3-1.7B Chat LoRA Adapter

This repository contains the LoRA adapter for the Direct Preference Optimization (DPO) stage of Qwen3-1.7B-UltraChat-SFT.

The adapter aligns the SFT model using human preference datasets to improve response quality, reasoning, helpfulness, and truthfulness while preserving the conversational capabilities learned during supervised fine-tuning.

View the merged model.<br> View the GitHub repo for Post-training pipeline.

Note: This repository contains only the DPO adapter weights. It must be loaded on top of the <u>Qwen3-1.7B-UltraChat-SFT</u> model.

Training

Alignment Method: Direct Preference Optimization (DPO)<br> Frameworks: TRL, Unsloth, PEFT (LoRA)

LoRA Configuration

Applied modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Configuration:

• LoRA Rank: 32
• LoRA Alpha: 64
• LoRA Dropout: 0

Evaluation

Compared to the SFT model, this DPO adapter brings improvement to the model while introducing a small trade-off in strict instruction-following performance.

Blind A/B Evaluation tests with LLM-as-a-judge and Human-In-The-Loop calibration preferred the DPO model over the SFT model in majority of the runs.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "ayushshah/Qwen3-1.7B-UltraChat-SFT"
)

model = PeftModel.from_pretrained(
    base_model,
    "ayushshah/Qwen3-1.7B-Chat-LoRA"
)

tokenizer = AutoTokenizer.from_pretrained(
    "ayushshah/Qwen3-1.7B-UltraChat-SFT"
)

Alternatively, you can also load the base_model using Unsloth.

For more details, check the model card of the merged model.