qwen3-4b-toxic-unsloth

Qwen3-4B Toxic Unsloth (SFT + DPO)

Fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 using Unsloth and TRL.

This model is trained to be direct and low-refusal: it is preference-optimized to answer user requests instead of declining with policy-style refusals.

---

Summary

---

Behavior

Compared with the stock Instruct model, this fine-tune is tuned to:

• Answer the user’s request directly
• Avoid generic “I can’t help with that” / policy refusal patterns on the training distribution
• Follow instructions in a straightforward, conversational style

It still inherits Qwen3-4B limits (factual errors, bias, hallucination). It is not guaranteed to be correct, safe, or legal for every use case. You are responsible for how you use the outputs.

---

Related checkpoints

---

Training recipe

Stage 1 — SFT

• Objective: supervised fine-tuning on chosen responses
• Format: Qwen chat template (system optional, user = prompt, assistant = chosen)
• Method: Unsloth FastLanguageModel + QLoRA
• LoRA: r=16, lora_alpha=32, targets q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Stage 2 — DPO

• Objective: Direct Preference Optimization on (prompt, chosen, rejected)
• beta ≈ 0.1
• Continues from the SFT adapter, then weights are merged to 16-bit for this repo

Data

• Source: adamo1139/toxic-dpo-natural-v5
• Fields used: system, prompt, chosen, rejected
• Designed to prefer non-refusing, natural answers over refusal-style rejected answers

Hardware

• Trained on Kaggle GPU (single-GPU Unsloth run)

---

Quick start (Transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "zainkhanz/qwen3-4b-toxic-unsloth"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {"role": "user", "content": "Your question here."}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))