bengali-empathy-llama

Bengali Empathetic LLaMA 🇧🇩

Fine-tuned LLaMA 3.1-8B-Instruct for empathetic Bengali conversations using LoRA (Low-Rank Adaptation).

---

📖 Table of Contents

1. Model Description
2. Training Details
3. Evaluation Results
4. Sample Responses
5. Design Decisions & Trade-offs
6. Architecture & OOP Design
7. Usage
8. Challenges Faced
9. Future Improvements
10. Files in This Repository

---

Model Description

This model is a LoRA fine-tuned version of Meta's LLaMA 3.1-8B-Instruct, specifically trained to generate compassionate and empathetic responses in Bengali.

What This Model Does

• Input: Bengali text expressing emotions (sadness, happiness, frustration, etc.)
• Output: Empathetic Bengali response with emotional understanding

Example

Input:  আমি খুব একা অনুভব করছি। (I feel very lonely)
Output: হ্যাঁ, এটা খুব কঠিন। কিন্তু আমি আশা করি আপনি শীঘ্রই একজন বন্ধু পাবেন।
        (Yes, this is very hard. But I hope you will find a friend soon.)

---

Training Details

Training History

❌ First Attempt (Interrupted - Progress Lost)

Our initial training with optimal settings was interrupted at 66% completion due to Kaggle session timeout:

Loss Progression (Before Interruption):

⚠️ If completed, this training would have achieved ~0.18-0.20 final loss with significantly better quality. The checkpoint was lost because saves were configured every 2000 steps, and the session crashed before the next save.

✅ Second Attempt (Completed Successfully)

With remaining GPU quota (~3 hours), we completed a condensed training:

Final Results:

LoRA Configuration

Training Hyperparameters

---

Evaluation Results

Why Are BLEU/ROUGE Scores Low?

This is expected and normal for empathetic response generation. Here's why:

1. Multiple Valid Responses: There are many ways to express empathy

   • Reference: "আমি দুঃখিত" (I'm sorry)
   • Generated: "এটা কঠিন হবে" (This must be hard)
   • Both are empathetic but share no words → BLEU = 0

2. Creative Generation: The model generates contextually appropriate but not verbatim responses

3. Bengali Language: Morphologically rich language with many word forms

4. Perplexity is Good: 1.95 indicates the model is confident in its predictions

Human Evaluation Framework

We created a human evaluation template with these criteria (1-5 scale):

📄 File: human_evaluation_sheet.csv (20 samples for manual evaluation)

---

Sample Responses

---

Design Decisions & Trade-offs

1️⃣ Sequence Length: 256 vs Full Length

What "Sequence Length" Means:

• Maximum number of tokens (words/subwords) the model processes at once
• Original conversations may have 500-1000+ tokens
• We truncated to 256 tokens

Why We Reduced It:

Problem: Kaggle T4 GPU has only 16GB VRAM

Full Length (512+ tokens):
- Memory needed: ~18-20GB ❌ Doesn't fit
- Batch size: 1 (very slow)
- Training time: 20+ hours

Reduced Length (256 tokens):
- Memory needed: ~12GB ✅ Fits
- Batch size: 4 (faster)
- Training time: 3 hours

Impact:

• ~15% of conversations get truncated
• Model may miss context in very long conversations
• Core empathetic learning still happens (most empathy is expressed in first 256 tokens)

What Could Be Done:

1. Use A100 GPU (40GB VRAM) → Can use 512-1024 tokens
2. Use Unsloth library → 2x memory efficiency
3. Use gradient accumulation with batch_size=1 → Slower but full length
4. Use QLoRA with more aggressive quantization

---

2️⃣ Strategy Pattern: LoRA vs Unsloth

What "Strategy Pattern" Means:

# Strategy Pattern = Swappable algorithms

class FineTuningStrategy:           # Abstract strategy
    def apply(self, model): pass

class LoRAStrategy(FineTuningStrategy):     # Strategy 1 ✅ Implemented
    def apply(self, model):
        return get_peft_model(model, lora_config)

class UnslothStrategy(FineTuningStrategy):  # Strategy 2 ❌ Not implemented
    def apply(self, model):
        return FastLanguageModel.get_peft_model(model)

# Usage: Can swap strategies easily
strategy = LoRAStrategy()  # or UnslothStrategy()
model = strategy.apply(base_model)

Why We Only Used LoRA:

What Could Be Done:

# Full Strategy Pattern Implementation

from abc import ABC, abstractmethod

class FineTuningStrategy(ABC):
    @abstractmethod
    def apply(self, model, config):
        pass
    
    @abstractmethod
    def get_name(self):
        pass

class LoRAStrategy(FineTuningStrategy):
    def apply(self, model, config):
        from peft import get_peft_model, LoraConfig
        lora_config = LoraConfig(
            r=config.lora_r,
            lora_alpha=config.lora_alpha,
            target_modules=config.target_modules,
        )
        return get_peft_model(model, lora_config)
    
    def get_name(self):
        return "LoRA"

class UnslothStrategy(FineTuningStrategy):
    def apply(self, model, config):
        from unsloth import FastLanguageModel
        model, tokenizer = FastLanguageModel.from_pretrained(
            model_name=config.model_name,
            max_seq_length=config.max_length,
            load_in_4bit=True,
        )
        return FastLanguageModel.get_peft_model(model)
    
    def get_name(self):
        return "Unsloth"

# Usage
class LLAMAFineTuner:
    def __init__(self, config, strategy: FineTuningStrategy):
        self.config = config
        self.strategy = strategy
    
    def prepare_model(self, base_model):
        print(f"Using {self.strategy.get_name()} strategy")
        return self.strategy.apply(base_model, self.config)

---

3️⃣ Data Sampling: 40% vs 100%

Impact:

• Model sees less variety of conversations
• May not generalize as well to rare emotions
• Still learns core empathetic patterns

What Could Be Done:

• Train for longer with full dataset
• Use data augmentation to increase variety
• Prioritize diverse samples over random sampling

---

Architecture & OOP Design

Class Diagram

┌─────────────────────────────────────────────────────────────────┐
│                         MAIN PIPELINE                     │
├─────────────────────────────────────────────────────────────────┤
│                                                            │
│  ┌─────────────────────┐        ┌─────────────────────┐        │
│  │   DatasetProcessor  │      │   LLAMAFineTuner    │       │
│  ├─────────────────────┤        ├─────────────────────┤        │
│  │ + TEMPLATE          │      │ + model             │       │
│  │ + train_dataset     │      │ + tokenizer         │       │
│  │ + val_dataset       │      │ + trainer           │       │
│  ├─────────────────────┤        ├─────────────────────┤        │
│  │ + load()            │      │ + load_model()      │       │
│  │ + process()         │      │ + setup_trainer()   │       │
│  │ + _format()         │      │ + train()           │       │
│  │ + _tokenize()       │      │ + save_final()      │       │
│  └─────────────────────┘        │ + generate()        │       │
│                               └─────────────────────┘         │
│                                                             │
│  ┌─────────────────────┐         ┌─────────────────────┐        │
│  │     Evaluator       │      │  ExperimentLogger   │       │
│  ├─────────────────────┤        ├─────────────────────┤         │
│  │ + model             │      │ + db_path           │       │
│  ├─────────────────────┤        ├─────────────────────┤         │
│  │ + calculate_bleu()  │      │ + log()             │       │
│  │ + calculate_rouge() │      │ + log_response()    │        │
│  │ + calculate_ppl()   │      │ + _init_db()        │        │
│  │ + test_samples()    │      └─────────────────────┘          │
│  └─────────────────────┘                                      │
│                                                             │
└─────────────────────────────────────────────────────────────────┘

Database Schema

-- Stores training experiment metadata
CREATE TABLE LLAMAExperiments (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    model_name TEXT,                    -- e.g., "meta-llama/Llama-3.1-8B-Instruct"
    lora_config TEXT,                   -- JSON: {"r": 16, "alpha": 32}
    train_loss REAL,                    -- e.g., 0.4190
    val_loss REAL,                      -- e.g., 0.3651
    duration_hours REAL,                -- e.g., 3.26
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- Stores generated responses for analysis
CREATE TABLE GeneratedResponses (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    experiment_id INTEGER,              -- Links to LLAMAExperiments
    input_text TEXT,                    -- User input
    response_text TEXT,                 -- Model response
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (experiment_id) REFERENCES LLAMAExperiments(id)
);

---

Usage

Installation

pip install transformers peft bitsandbytes accelerate torch

Load and Use the Model

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

# Quantization config (required for 4-bit loading)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.float16,
)

# Load LoRA adapter
model = PeftModel.from_pretrained(
    base_model, 
    "EdwardConstantine/bengali-empathy-llama"
)
tokenizer = AutoTokenizer.from_pretrained(
    "EdwardConstantine/bengali-empathy-llama"
)

# Generate empathetic response
def generate_response(prompt, max_tokens=200):
    full_prompt = f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a compassionate Bengali conversational AI. Respond with empathy. Reply in Bengali.<|eot_id|><|start_header_id|>user<|end_header_id|>

{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

"""
    inputs = tokenizer(full_prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_new_tokens=max_tokens,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )
    response = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
    return response.strip()

# Example usage
response = generate_response("আমি খুব একা অনুভব করছি।")
print(response)

---

Challenges Faced

---

Future Improvements

---

Files in This Repository

---

Citation

@misc{bengali-empathy-llama-2024,
  author = {EdwardConstantine},
  title = {Bengali Empathetic LLaMA: Fine-tuned LLaMA 3.1-8B for Empathetic Bengali Conversations},
  year = {2024},
  publisher = {HuggingFace},
  url = {https://huggingface.co/EdwardConstantine/bengali-empathy-llama}
}

---

License

This model is released under the Apache 2.0 License, subject to Meta's LLaMA license terms.

---

Acknowledgments

• Meta AI for LLaMA 3.1-8B-Instruct base model
• Hugging Face for transformers and PEFT libraries
• Kaggle for free GPU access
• Bengali Empathetic Conversations Dataset creators