sift-1b-dpo

Sift-1B-DPO (Direct Preference Optimization — Golden Release)

<!-- Quick Summary -->

Sift-1B-DPO is the flagship 1.5B Small Language Model (SLM) adapter trained via Direct Preference Optimization (DPO) on top of the SFT checkpoint. It achieves 100% Tool Selection Accuracy and 95.2% overall benchmark score, gently steering the model away from parameter hallucinations, schema mismatches, and markdown formatting fluff.

• Developed by: Sanatan Singh
• Model Type: PEFT / QLoRA Adapter (Direct Preference Optimization Stage)
• Base Model: unsloth/Qwen2.5-1.5B-Instruct-bnb-4bit (trained via SanatanSinghVishen/sift-1b-sft)
• Golden Checkpoint: checkpoint-750
• Language(s): English (en)
• License: MIT
• Repository: GitHub — SanatanSinghVishen/Sift-1B
• GGUF Export: SanatanSinghVishen/sift-1b-gguf

---

Model Details

Overview

While Supervised Fine-Tuning (SFT) teaches the model how to generate JSON, models often output unnecessary prose, hallucinated key-value pairs, or slightly invalid data types.

Sift-1B-DPO represents Stage 2 in the Sift-1B pipeline:

• Chosen Responses ($y_w$): Strict, minimal, schema-compliant ChatML JSON tool calls.
• Rejected Responses ($y_l$): Markdown code block wrapping (json ... ), conversational intros ("Here is your JSON:"), missing required arguments, or wrong tool choices.

Through DPO alignment with $\beta = 0.1$, Sift-1B-DPO delivers deterministic, instant local tool calling.

---

Benchmark Results & Checkpoint Comparison

Across comprehensive evaluation tests (50 multi-domain benchmark queries), DPO Checkpoint-750 emerged as the golden checkpoint:

---

How to Get Started with the Model

Run Sift-1B-DPO using Hugging Face transformers and peft:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "unsloth/Qwen2.5-1.5B-Instruct-bnb-4bit"
adapter_name = "SanatanSinghVishen/sift-1b-dpo"

# 1. Load Tokenizer & Base Model
tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# 2. Load Golden DPO Adapter
model = PeftModel.from_pretrained(base_model, adapter_name)
model.eval()

# 3. Formulate Input Query
messages = [
    {
        "role": "system",
        "content": "You are a function calling agent. Output only valid JSON tool calls."
    },
    {
        "role": "user",
        "content": "Book a ride to JFK airport arriving by 5:00 PM."
    }
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

# 4. Generate Clean JSON Response
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=128, temperature=0.1)

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)