Qwen3.5-0.8B-SFT-DPO

Qwen3.5-0.8B-SFT-DPO (NLP Project Phase 2)

This is a fine-tuned version of Qwen/Qwen3.5-0.8B-Base. It was trained using a complete Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) pipeline using LoRA adapters. The model is capable of instruction-following and conversational interactions utilizing the ChatML prompt format.

Model Details

Model Description

This model was developed as part of Assignment 04 for the "NLP with Deep Learning" course at the Institute of Business Administration. The objective was to transform a raw base model (Qwen3.5-0.8B-Base) into a human-aligned, instruction-following assistant.

The model was trained in two stages:

1. SFT: Trained on a 5k subset of alpaca-cleaned to learn the ChatML conversation format.
2. DPO: Trained on a 2k subset of ultrafeedback_binarized to align the model with human preferences, encouraging more thorough and structured responses.

• Developed by: Muddasir Javed, Qamar Raza, Ramail Khan
• Institution: Institute of Business Administration (Instructor: Sajjad Haider)
• Model type: Causal Language Model (Decoder-only Transformer)
• Language(s) (NLP): English
• License: Apache 2.0
• Finetuned from model: Qwen/Qwen3.5-0.8B-Base

Model Sources

• Repository: GitHub Pipeline Codebase
• Base Model: Qwen3.5 0.8B

Uses

Direct Use

The model is intended to be used for general instruction-following tasks, including writing Python/JavaScript code, answering science and history questions, and creative writing. It must be prompted using the ChatML format (<|im_start|>user...\n<|im_start|>assistant...).

Out-of-Scope Use

Because this is a highly compact (0.8 billion parameter) model fine-tuned on a limited dataset subset, it is not suitable for high-stakes, real-world deployment.

• It should not be used as a calculator (it exhibits arithmetic hallucination).
• It should not be used as a factual knowledge base without verification (it sometimes hallucinates historical facts and populations).

Bias, Risks, and Limitations

Based on comprehensive evaluation, the model exhibits the following known limitations:

• Verbosity and Truncation: The model occasionally enters repetitive loops (e.g., repeating a city's population multiple times) or generates overly verbose answers, which may cause it to hit generation token limits before concluding.
• Arithmetic Errors: While the model understands mathematical formulas and steps (e.g., calculating mean/median/mode), it frequently fails at basic addition arithmetic during the final step.
• Format Hallucination: In specific coding tasks (e.g., JavaScript async API calls), the model may learn the API surface but hallucinate the proper asynchronous structure.

How to Get Started with the Model

Use the code below to get started with the model. Note that the apply_chat_template function automatically handles the required ChatML formatting.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sparx3/Qwen3.5-0.8B-SFT-DPO"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

prompt = "Write a Python function that takes a list of integers and returns the sum of all even numbers in the list. Include an example usage."
messages = [
    {"role": "user", "content": prompt}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

• SFT Stage: 5,000 samples randomly selected from yahma/alpaca-cleaned (instruction-response pairs).
• DPO Stage: 2,000 samples randomly selected from trl-lib/ultrafeedback_binarized (chosen/rejected pairs).

Training Procedure

The model was trained using Parameter-Efficient Fine-Tuning (PEFT) with LoRA. The SFT weights were merged into the base model prior to initiating the DPO stage.

Training Hyperparameters

Best SFT Configuration (Trial 4):

• LoRA Rank: 32 (Alpha: 64)
• Target Modules: q_proj, v_proj
• Learning Rate: 5e-5
• Batch Size: 2 (Effective 8 via gradient accumulation)
• Epochs: 3
• Precision: bfloat16

Best DPO Configuration (Trial 3):

• LoRA Rank: 16 (Alpha: 32)
• Target Modules: q_proj, v_proj
• Learning Rate: 1e-5
• Beta: 0.1
• Batch Size: 4
• Epochs: 2
• Precision: bfloat16

Evaluation

Testing Data, Factors & Metrics

The model was evaluated against a custom, fixed test set of 10 highly diverse prompts generated by DeepSeek-v4-flash, spanning coding, science, math, history, and creative writing.

Metrics:

• BERTScore (F1): Primary metric (robust to paraphrasing, evaluated via distilbert-base-uncased).
• BLEU: Secondary metric (measures n-gram precision).

Results

The pipeline demonstrated monotonic improvement across stages, confirming that preference optimization successfully added semantic quality beyond instruction tuning.

Reward Accuracy: The best DPO model achieved a 60% reward accuracy during evaluation with a reward margin of 0.086, confirming genuine preference learning.

Environmental Impact

Training was conducted efficiently using serverless cloud infrastructure on Modal.com.

• Hardware Type: NVIDIA B200 (Blackwell architecture)
• Hours used: ~2.15 hours total for all 10 trials combined (SFT: 95 mins, DPO: 34 mins).
• Cloud Provider: Modal.com
• Compute Region: US

Citation

APA: Javed, M., Raza, Q., & Khan, R. (2024). Instruction Tuning and Preference Fine-Tuning of Qwen3.5-0.8B-Base. Institute of Business Administration.