gpt2-nebulamesh-dpo

GPT-2 Fine-Tuning: Domain → Instruction → DPO

This project demonstrates an end-to-end LLM fine-tuning workflow using GPT-2 (124M parameters).

The goal of this project is to understand how a pretrained language model can be adapted through multiple training stages:

Pretrained GPT-2
      ↓
Non-Instruction Fine-Tuning
      ↓
Instruction Fine-Tuning
      ↓
Preference Fine-Tuning (DPO)
      ↓
Final Model

🚀 Project Overview

I used GPT-2 (124M) as the base model and created a small fictional technical domain called NebulaMesh.

The project demonstrates:

1. Non-instruction/domain fine-tuning
2. Instruction fine-tuning
3. Preference fine-tuning using DPO
4. Model evaluation through text generation
5. Saving and publishing the final model to Hugging Face Hub

Note: NebulaMesh is a fictional domain created for learning and experimentation. The dataset is intentionally small and is not intended for production use.

---

🧠 1. Base Model

The starting model is:

GPT-2 — 124M parameters

The pretrained GPT-2 model already has general language knowledge.

We first tested the pretrained model with a simple prompt and then adapted it to the fictional NebulaMesh domain.

---

📚 2. Non-Instruction Fine-Tuning

The first training stage used plain domain-specific text.

Example:

NebulaMesh is a fictional distributed computing protocol
for edge devices.

NebulaMesh nodes communicate using the NMX protocol.

MeshPulse sends a heartbeat every 10 seconds.

The objective is next-token prediction.

The model learns patterns from the NebulaMesh text by:

Input text
   ↓
Forward Pass
   ↓
Calculate Loss
   ↓
Backpropagation
   ↓
Update Model Weights

This stage adapts the pretrained GPT-2 model toward the NebulaMesh domain.

Output:

./gpt2-solidity

---

🤖 3. Instruction Fine-Tuning

Next, the domain model was fine-tuned using instruction/response examples.

Example:

### Instruction:
What protocol does NebulaMesh use?

### Response:
NebulaMesh uses the NMX protocol.

The purpose of this stage is to teach the model to respond to instructions rather than simply continue domain text.

The training format was:

Instruction → Response

Output:

./gpt2-solidity-instruct

---

🎯 4. Preference Fine-Tuning with DPO

The final training stage used Direct Preference Optimization (DPO).

Instead of providing only one answer, each training example contains:

• Prompt
• Chosen response
• Rejected response

Example:

Prompt:
What protocol does NebulaMesh use?

Chosen:
NebulaMesh uses the NMX protocol.

Rejected:
NebulaMesh uses the NebulaMesh protocol repeatedly.

The model learns to prefer the chosen response over the rejected response.

Conceptually:

                  Prompt
                    │
          ┌─────────┴─────────┐
          ↓                   ↓
       Chosen              Rejected
        👍                    👎
          └─────────┬─────────┘
                    ↓
                   DPO
                    ↓
        Preference-aligned model

Final model:

./gpt2-nebulamesh-dpo

---

🔄 Complete Training Pipeline

                    GPT-2
                  124M Params
                      │
                      ▼
          Non-Instruction Fine-Tuning
                      │
                      ▼
              NebulaMesh Domain
                      │
                      ▼
             Instruction Fine-Tuning
                      │
                      ▼
              Instruction Following
                      │
                      ▼
             DPO Preference Training
                      │
                      ▼
                Final Model
                      │
                      ▼
             Hugging Face Hub

---

🛠️ Technologies Used

• Python
• PyTorch
• Hugging Face Transformers
• Hugging Face Datasets
• Hugging Face TRL
• Direct Preference Optimization (DPO)
• Hugging Face Hub

---

📦 Installation

pip install torch transformers datasets accelerate trl huggingface_hub

---

▶️ Training

Domain Fine-Tuning

Train GPT-2 on the NebulaMesh domain dataset.

GPT-2
  ↓
Domain Dataset
  ↓
Full Fine-Tuning

Instruction Fine-Tuning

Start from the domain-fine-tuned model:

./gpt2-solidity

and train using instruction/response examples.

DPO

Start from the instruction-fine-tuned model:

./gpt2-solidity-instruct

and train using preference data containing:

prompt
chosen
rejected

---

🧪 Testing

The final model can be loaded using:

from transformers import GPT2Tokenizer, GPT2LMHeadModel

model_path = "./gpt2-nebulamesh-dpo"

tokenizer = GPT2Tokenizer.from_pretrained(model_path)
model = GPT2LMHeadModel.from_pretrained(model_path)

tokenizer.pad_token = tokenizer.eos_token
model.config.pad_token_id = tokenizer.pad_token_id

Example:

question = "What protocol does NebulaMesh use?"

prompt = (
    "### Instruction:\n"
    + question
    + "\n\n"
    "### Response:\n"
)

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=False,
    pad_token_id=tokenizer.pad_token_id,
)

print(
    tokenizer.decode(
        outputs[0],
        skip_special_tokens=True
    )
)

Expected response:

NebulaMesh uses the NMX protocol.

Because the dataset is intentionally small, the model may still produce repetitive or imperfect responses.

---

⚠️ Limitations

This project is primarily an educational experiment.

The dataset is small and uses a fictional domain. Therefore:

• The model may memorize training examples.
• The model may generate repetitive text.
• The model may produce incorrect responses.
• The model is not suitable for production use.
• DPO preferences are based on a small manually created dataset.

A larger, higher-quality dataset would be required for meaningful real-world performance.

---

📖 What I Learned

Through this project, I explored:

Full Fine-Tuning

Updating the pretrained model's trainable weights during training.

GPT-2
 ↓
Train model weights
 ↓
Fine-tuned model

Instruction Fine-Tuning

Teaching a model to follow instructions and produce responses.

Instruction
     ↓
Response

Preference Fine-Tuning

Teaching a model to prefer better responses over less desirable responses.

Prompt
 ↓
Chosen vs Rejected
 ↓
DPO
 ↓
Preferred behavior

Hugging Face Hub

The final model was uploaded to Hugging Face Hub so it can be shared and loaded later.

---

🎓 Learning Progression

This project helped me understand the progression:

Pretraining
     ↓
Domain Fine-Tuning
     ↓
Instruction Fine-Tuning
     ↓
Preference Fine-Tuning

The next step is to explore Parameter-Efficient Fine-Tuning (PEFT) techniques such as:

• LoRA
• QLoRA

These approaches allow large language models to be adapted while training far fewer parameters than full fine-tuning.

---

👤 Purpose

This repository was created as a hands-on learning project to understand the mechanics of modern LLM fine-tuning using Hugging Face tools.

The implementation and experiments are for educational purposes.