gemma-2b-empathy-lora

---

language:

• en library_name: peft base_model: google/gemma-2-2b-it tags:
• lora
• peft
• empathy
• mental-health
• conversational
• chat

---

Gemma-2B Empathy LoRA Adapter

Overview

This repository contains a LoRA (Low-Rank Adaptation) adapter fine-tuned on top of the google/gemma-2-2b-it base model to improve empathetic and emotionally supportive conversational responses.

The adapter modifies the response style and tone of the base model without altering its underlying knowledge or reasoning capabilities.

This model is intended as a supportive conversational assistant, not as a replacement for professional mental health care.

---

Base Model

• Base model: google/gemma-2-2b-it
• Model type: Instruction-tuned large language model
• Fine-tuning method: Parameter-Efficient Fine-Tuning (PEFT) using LoRA

Only the LoRA adapter weights are provided in this repository.
The base model must be loaded separately.

---

Fine-Tuning Method

This adapter was trained using LoRA (Low-Rank Adaptation) via the PEFT framework.

Key characteristics:

• Base model parameters are frozen
• Approximately 0.12% of total parameters are trainable
• LoRA layers are applied to attention projection layers
• Training focuses on behavioral adaptation, not factual learning

This approach allows efficient fine-tuning on limited compute while preserving base model performance.

---

Datasets Used

Two publicly available datasets were combined to balance emotional sensitivity and professional tone:

1. Empathetic Dialogues

• Source: https://huggingface.co/datasets/Estwld/empathetic_dialogues_llm
• Used to learn emotional mirroring, conversational flow, and response grounding

2. CounselChat

• Source: https://huggingface.co/datasets/nbertagnolli/counsel-chat
• Used to learn calm, supportive, and non-judgmental response framing from mental health professionals

Approximately 12,000 processed dialogue pairs were used for training.

No dataset files are redistributed in this repository.

---

Training Configuration (Summary)

• Fine-tuning type: Supervised Fine-Tuning (SFT)
• Epochs: 2
• Learning rate: 2e-4
• Effective batch size: 16 (via gradient accumulation)
• Optimizer: AdamW
• Quantization: 4-bit loading for memory efficiency
• Hardware: Single GPU (Tesla T4 class)

Training converged stably without divergence or overfitting.

---

Intended Use

This model is intended for:

• Empathetic conversational agents
• Mental health support tools (non-clinical)
• Reflection, validation, and emotional acknowledgment use cases
• Educational or research purposes related to affective AI

---

Not Intended For

This model must NOT be used for:

• Medical or psychological diagnosis
• Crisis intervention or suicide prevention
• Emergency response or clinical decision-making
• Replacing licensed mental health professionals

Applications should implement external safety checks and escalation logic.

---

Safety & Limitations

• The model may generate incomplete or overly verbose responses
• It does not have real understanding of human emotions
• It may fail in high-risk or crisis scenarios
• Safety, moderation, and escalation must be handled at the application level

Users are responsible for deploying this model responsibly.

---

Usage Example

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b-it")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b-it")

model = PeftModel.from_pretrained(
    base_model,
    "<your-username>/<your-repo-name>"
)

model.eval()