Qwen3-4B-DPO-Final-Day3

Qwen3-4B-Instruct DPO Finetuned (Day 3 Final)

This model is a fine-tuned version of unsloth/Qwen3-4B-Instruct-2507 using Direct Preference Optimization (DPO) via the Unsloth library.

This repository contains the full-merged 16-bit weights. No adapter loading is required.

⚠️ Note on Chat Template / URL

The repository URL and training configuration contain qwen-2.5. This was selected solely for technical compatibility (as Qwen 3 shares the same ChatML format), and does NOT indicate the use of the Qwen 2.5 base model. The base model used is strictly Qwen3-4B-Instruct-2507.

Training Configuration

• Compliance Note: The training base model strictly follows the competition rule §6.6.
• Base Model: unsloth/Qwen3-4B-Instruct-2507 (Authorized Model)
• Method: DPO (Direct Preference Optimization)
• Dataset: Custom Tier-Strict++ Dataset
• Epochs: 3
• Learning Rate: 5e-6
• Max Length: 2048
• Hardware: NVIDIA A100 (Google Colab)

Usage

Since this is a merged model, you can use it directly with transformers.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Note: The URL contains '2.5' due to initial template settings, but the model is Qwen3.
model_id = "beachcities/Qwen2.5-3B-DPO-Final-Day3" 

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Test inference
prompt = "指示: 次の質問に論理的に答えてください。\\n質問: AIの未来についてどう思いますか？"
messages = [{"role": "user", "content": prompt}]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))