tinyllama-trl-gguf

🦙 TinyLlama GGUF - Quantized Model

Model Description

This is a GGUF quantized version of TinyLlama (1.1B parameters) fine-tuned using Unsloth and TRL with LoRA adapters. The model has been optimized for efficient CPU inference with minimal memory footprint.

Key Features:

• GGUF Format: Optimized for llama.cpp and CPU inference
• Quantized: Reduced memory usage without significant quality loss
• Fine-tuned: Custom trained on specific dataset for improved performance
• Efficient: Runs on CPU, Raspberry Pi, and mobile devices

Model Details:

• Base Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Fine-tuning Method: LoRA (Low-Rank Adaptation) with Unsloth optimizations
• Format: GGUF (GGML Universal Format)
• Quantization: Q4_K_M (4-bit quantization)
• Parameters: 1.1 Billion
• Context Length: 2048 tokens

🛠️ Training Details

Base Model

Fine-tuning Configuration

LoRA Parameters

LORA_R       = 16        # Rank of LoRA matrices
LORA_ALPHA   = 32        # Scaling factor (alpha/r = 2.0)
LORA_DROPOUT = 0.05      # Dropout for regularization
TARGET_MODULES = [       # Layers where LoRA is applied
    "q_proj",            # Query projection
    "k_proj",            # Key projection  
    "v_proj",            # Value projection
    "o_proj",            # Output projection
    "gate_proj",         # Gate projection (MLP)
    "up_proj",           # Up projection (MLP)
    "down_proj"          # Down projection (MLP)
]

## 🚀 Usage

### Option 1: Using llama-cpp-python (Recommended)

```python
# Install llama-cpp-python
# pip install llama-cpp-python

from llama_cpp import Llama

# Load the model
model_path = "arif-butt/tinyllama-trl-gguf"  # or local path to .gguf file
llm = Llama(
    model_path=model_path,
    n_ctx=2048,              # Context length
    n_threads=4,             # Number of CPU threads
    n_gpu_layers=0,          # Set >0 for GPU offloading
    verbose=False,
)

# Simple prompt
prompt = "Q: Name all the courses Arif butt teach?\nA:"

# Generate response
output = llm(
    prompt,
    max_tokens=100,          # Maximum tokens to generate
    temperature=0.2,         # Lower = more deterministic
    top_p=0.95,              # Nucleus sampling
    repeat_penalty=1.1,      # Penalize repetition
    stop=["Q:", "\nQ:"],     # Stop sequences
)

print(output["choices"][0]["text"])



# ── Simple Chat Interface for GGUF Model ───────────────────────────────────

from llama_cpp import Llama
import sys

class TinyLlamaChat:
    def __init__(self, model_path="arif-butt/tinyllama-trl-gguf"):
        """Initialize the GGUF model"""
        print("Loading model...")
        self.llm = Llama(
            model_path=model_path,
            n_ctx=2048,
            n_threads=4,
            verbose=False,
        )
        print("✅ Model loaded!")
    
    def generate(self, prompt, max_tokens=100, temperature=0.7):
        """Generate response for a single prompt"""
        output = self.llm(
            prompt,
            max_tokens=max_tokens,
            temperature=temperature,
            top_p=0.95,
            repeat_penalty=1.1,
            stop=["Q:", "\nQ:", "User:", "\nUser:", "Human:"],
        )
        return output["choices"][0]["text"]
    
    def chat(self):
        """Interactive chat mode"""
        print("\n💬 Chat Mode (type 'quit' to exit)")
        print("-" * 50)
        
        while True:
            user_input = input("\n👤 You: ")
            if user_input.lower() in ['quit', 'exit', 'q']:
                break
            
            prompt = f"Q: {user_input}\nA:"
            response = self.generate(prompt, temperature=0.7)
            print(f"🤖 Assistant: {response}")

# Use the chat interface
if __name__ == "__main__":
    chat = TinyLlamaChat()
    chat.chat()