falcon-7b-simple-dpo-lora-lablebox

falcon-7b-simple-dpo-lora-lablebox

This model is a fine-tuned version of tiiuae/falcon-7b-instruct using Direct Preference Optimization (Simple DPO).

Model Description

• Training Method: Direct Preference Optimization (Simple DPO)
• Base Model: Falcon-7B-Instruct
• Parameter Count: 6.92B (base model)
• LoRA Parameters: 0.0085% trainable
• Hardware: Apple Silicon Mac (128GB RAM)
• Framework: PyTorch with MPS backend

Training Results

• Runtime: ~30 minutes
• Steps: 200
• Loss Reduction: 88.3%
• Benchmark Quality Score: 0.90/1.00

Training Configuration

LoRA Configuration

• Rank (r): 2
• Alpha: 4
• Target Modules: query_key_value
• Dropout: 0.1

Training Parameters

• Learning Rate: 5e-5
• Gradient Accumulation: 8 steps
• Mixed Precision: FP16
• Scheduler: Cosine Annealing

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "tiiuae/falcon-7b-instruct",
    trust_remote_code=True,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "falcon-7b-simple-dpo-lora-lablebox")

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("falcon-7b-simple-dpo-lora-lablebox")

# Generate text
prompt = "What is machine learning?"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

Training Details

This model was trained as part of the Lablebox Take Home Assignment, demonstrating gradient-based training of large language models on consumer hardware.

Framework versions

• Transformers 4.44.2
• PyTorch 2.5.0.dev20240912
• PEFT 0.13.0
• Datasets 3.0.0
• Tokenizers 0.19.1