Llama-3.2-1B-SFT-DPOMix-Adapters

LLaMA 3.2-1B — SFT LoRA Adapters

These are the Stage 1 LoRA adapters from a complete SFT → DPO fine-tuning pipeline trained on argilla/dpo-mix-7k.

These are adapter weights only — not a standalone model. You need the base model + these adapters together to run inference.

What are LoRA Adapters?

Instead of saving a full 2.5 GB fine-tuned model, LoRA saves only the small "diff" (~50 MB) from the base model. You load the base model once and plug in these adapters on top. This makes storage and sharing much more efficient.

How to use

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_id = "meta-llama/Llama-3.2-1B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype = torch.float16,
    device_map  = "auto",
)

# Load SFT adapters on top of base model
model = PeftModel.from_pretrained(
    base_model,
    "pranav6905/Llama-3.2-1B-SFT-DPOMix-Adapters"
)

prompt = (
    "<|begin_of_text|>"
    "<|start_header_id|>user<|end_header_id|>\n"
    "Explain what machine learning is.\n<|eot_id|>"
    "<|start_header_id|>assistant<|end_header_id|>\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Next Stage

These adapters were used as the starting point for DPO alignment. See the final aligned model: pranav6905/llama-1b-sft-dpo-final