tinyllama-peft-gguf

🦙 TinyLlama PEFT GGUF - Quantized LoRA Model

Model Description

This is a GGUF quantized version of TinyLlama (1.1B parameters) fine-tuned using PEFT (Parameter-Efficient Fine-Tuning) with LoRA adapters. The model has been optimized for efficient CPU inference with minimal memory footprint.

Key Features:

• GGUF Format: Optimized for llama.cpp and CPU inference
• Quantized: Reduced memory usage without significant quality loss
• PEFT Fine-tuned: LoRA adapters trained on custom dataset
• Efficient: Runs on CPU, Raspberry Pi, and edge devices

Model Details:

• Base Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Fine-tuning Method: LoRA (Low-Rank Adaptation) with PEFT
• Format: GGUF (GGML Universal Format)
• Quantization: Q4_K_M (4-bit quantization)
• Parameters: 1.1 Billion
• Context Length: 2048 tokens

🚀 Usage

Option 1: Using llama-cpp-python (Recommended)

# Install llama-cpp-python
# pip install llama-cpp-python

from llama_cpp import Llama

# Load the model
model_path = "arif-butt/tinyllama-peft-gguf"  # or local path to .gguf file
llm = Llama(
    model_path=model_path,
    n_ctx=2048,              # Context length
    n_threads=4,             # Number of CPU threads
    n_gpu_layers=0,          # Set >0 for GPU offloading
    verbose=False,
)

# Simple prompt
prompt = "Q: Name all the courses Arif butt teach?\nA:"

# Generate response
output = llm(
    prompt,
    max_tokens=100,          # Maximum tokens to generate
    temperature=0.2,         # Lower = more deterministic
    top_p=0.95,              # Nucleus sampling
    repeat_penalty=1.1,      # Penalize repetition
    stop=["Q:", "\nQ:"],     # Stop sequences
)

print(output["choices"][0]["text"])

Option 2: Using Transformers with llama-cpp
from transformers import AutoTokenizer
from llama_cpp import Llama

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")

# Load GGUF model
model_path = "arif-butt/tinyllama-peft-gguf"
llm = Llama(model_path=model_path, n_ctx=2048)

def generate_response(prompt, max_tokens=100, temperature=0.7):
    """Generate response using the GGUF model"""
    output = llm(
        prompt,
        max_tokens=max_tokens,
        temperature=temperature,
        stop=["Q:", "\nQ:", "User:", "\nUser:"],
    )
    return output["choices"][0]["text"]

# Test
prompt = "Q: What is machine learning?\nA:"
response = generate_response(prompt)
print(f"Response: {response}")

Option 3: Using llama.cpp CLI
# Download the model
wget https://huggingface.co/arif-butt/tinyllama-peft-gguf/resolve/main/tinyllama-peft-q4_k_m.gguf

# Run inference
./main -m tinyllama-peft-q4_k_m.gguf \
       -p "Q: Name all the courses Arif butt teach?\nA:" \
       -n 100 \
       -t 4 \
       --temp 0.2 \
       --top_p 0.95