Downloads · 30 days
12
14% of all-time downloads
sarthaksenapati/qwen1.5b-hinglish-dpo
qwen1.5b-hinglish-dpo is a text generation model from sarthaksenapati. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
A DPO-aligned LoRA adapter that turns Qwen2.5-1.5B-Base into a Hindi–English (romanized Hinglish) chat assistant. This is the preference-optimization (DPO) stage that sits on top of an SFT model; it makes the assistan…
Downloads · 30 days
12
14% of all-time downloads
All-time downloads
86
Public
Repo size
159 MB
Likes
0
Public
Click a slice to open those files.
.safetensors73.9 MB · 87%
From the Hugging Face model README
A DPO-aligned LoRA adapter that turns Qwen2.5-1.5B-Base into a Hindi–English (romanized Hinglish) chat assistant. This is the preference-optimization (DPO) stage that sits on top of an SFT model; it makes the assistant's answers more preferred while preserving general capability.
This adapter is the final stage of a two-step post-training pipeline (SFT → DPO) built end-to-end on rented GPUs for a few dollars. The base model was first supervised-fine-tuned into a Hinglish assistant (qwen1.5b-hinglish-sft-v2); this DPO stage then optimizes that SFT model on on-policy preference pairs using a from-scratch Direct Preference Optimization loss (verified against TRL's formula and analytic invariants).
The DPO LoRA was trained on top of the merged SFT model, so the SFT model is its reference policy. Toggling this adapter on/off recovers the DPO policy vs the SFT reference.
sarthaksenapati/qwen1.5b-hinglish-sft-v2 (itself fine-tuned from Qwen/Qwen2.5-1.5B)Low-stakes conversational assistance in Hinglish: casual Q&A, suggestions, explanations, and everyday instruction-following in romanized Hindi–English.
A starting point for further preference optimization (IPO/KTO), or as a small, cheap Hinglish chat model for experimentation and as a portfolio/teaching artifact for the SFT → DPO pipeline.
Not safety-aligned. Do not use for high-stakes advice (medical, legal, financial), factual lookup where correctness matters, or any safety-critical or adversarial setting. The model can produce confident but wrong or dubious claims.
Use only for low-stakes Hinglish interaction. Do not rely on factual correctness. Keep a human in the loop for anything consequential.
This DPO adapter was trained on top of the merged SFT model, so you must apply the SFT adapter first, merge it, then load this DPO adapter:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
BASE = "Qwen/Qwen2.5-1.5B"
SFT = "sarthaksenapati/qwen1.5b-hinglish-sft-v2"
DPO = "sarthaksenapati/qwen1.5b-hinglish-dpo"
CHATML = (
"{% for m in messages %}"
"{{ '<|im_start|>' + m['role'] + '\n' + m['content'] + '<|im_end|>' + '\n' }}"
"{% endfor %}"
"{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
)
tok = AutoTokenizer.from_pretrained(DPO)
tok.chat_template = CHATML
im_end = tok.convert_tokens_to_ids("<|im_end|>")
base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
base = PeftModel.from_pretrained(base, SFT).merge_and_unload() # 1) merge SFT (the reference)
model = PeftModel.from_pretrained(base, DPO).eval() # 2) apply DPO adapter
msgs = [{"role": "user", "content": "Mujhe weekend ke liye ek easy dinner recipe batao."}]
text = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
ids = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**ids, max_new_tokens=256, do_sample=False,
repetition_penalty=1.2, no_repeat_ngram_size=3, eos_token_id=im_end)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))
sarthaksenapati/khichdi-pref — 1,497 on-policy preference pairs {prompt, chosen, rejected}. Built by sampling 4 completions per held-out prompt from the SFT model (temperature 0.7, top-p 0.9, with foreign-script token suppression), then ranking them with an independent judge (DeepSeek, shuffled to cancel position bias) into best (chosen) and worst (rejected). Pairs with no clear quality gap were dropped.
Direct Preference Optimization with a from-scratch sigmoid DPO loss (L = −log σ(β · [(logπθ(y_w) − logπref(y_w)) − (logπθ(y_l) − logπref(y_l))])), verified numerically against TRL's formula and analytic invariants (loss = log 2 at init, correct gradient signs). One model serves as both policy and reference: the SFT adapter is merged into the base (= reference), and a fresh LoRA adapter is added for DPO; disable_adapter() recovers the reference.
Prompt+response tokenized with a ChatML template; loss is masked to the response tokens (plus the stop token) only.
Adapter ≈ 74 MB. ~279 steps in roughly 25–40 minutes on a single RTX 4090. First-step loss = 0.6931 (log 2), confirming the policy equals the reference at initialization.
300 held-out prompts from sarthaksenapati/khichdi-sft (eval_prompts split), never seen in SFT or preference generation.
DPO produces a clean, statistically significant preference gain over its SFT reference while preserving general capability and answer length — i.e. better-preferred answers with no measurable over-optimization. (An earlier 2-epoch run reached only 60.9% with a CI touching chance; diagnosing under-training and retraining at 3 epochs / lr 1e-5 produced this result, verified safe by the flat MMLU and length canary.)
Qwen2.5-1.5B decoder-only transformer + LoRA adapter; objective = sigmoid DPO (Bradley–Terry preference loss) with the SFT model as the frozen reference.
If you use this model, please cite the repository:
@misc{khichdi2026,
title = {Khichdi: post-training Qwen2.5-1.5B into a Hinglish assistant (SFT -> DPO)},
author = {Senapati, Sarthak},
year = {2026},
url = {https://github.com/sarthaksenapati/khichdi-hinglish-llm}
}
Sarthak Senapati
Via the GitHub repository: https://github.com/sarthaksenapati/khichdi-hinglish-llm