mistral-7b-qlora

Mistral-7B UltraChat QLoRA Adapter

Model Description

This repository contains a LoRA adapter fine-tuned from Mistral-7B using the UltraChat instruction dataset.

The model was trained using QLoRA (4-bit quantization + LoRA adapters) to enable efficient fine-tuning on limited GPU resources such as Kaggle GPUs.

This adapter can be loaded on top of the base model to generate instruction-following responses.

---

Model Details

• Developer: Hemanth Naravi
• Model type: Instruction-tuned causal language model adapter
• Base model: mistralai/Mistral-7B-v0.1
• Fine-tuning method: QLoRA (LoRA + 4-bit quantization)
• Language: English
• License: Apache-2.0

---

Training Data

The model was trained using the UltraChat-200k dataset.

UltraChat contains multi-turn conversations between users and assistants designed to improve instruction-following abilities of large language models.

Dataset link:

https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k

For this experiment, a subset of the dataset was used.

---

Training Procedure

Fine-tuning Method

QLoRA (Quantized LoRA)

Training Hyperparameters

---

Hardware

Training was performed using:

• GPU: NVIDIA Tesla T4 (Kaggle)
• Framework: PyTorch

Libraries used:

• Transformers
• PEFT
• BitsAndBytes
• Datasets

---

How to Use the Model

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
import torch

model_name = "mistralai/Mistral-7B-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True
)

base_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    "Hemanth-5204/mistral-7b-qlora"
)

prompt = "Explain self attention in transformers."

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=200
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

for continous chat use this

import torch

chat_history = ""

print("Type 'exit' to stop\n")

while True:
    
    user_input = input("User: ")
    
    if user_input.lower() == "exit":
        break

    # Add user message to history
    chat_history += f"\nUser: {user_input}\nAssistant:"

    inputs = tokenizer(chat_history, return_tensors="pt").to(model.device)

    outputs = model.generate(
        **inputs,
        max_new_tokens=150,
        temperature=0.3,
        top_p=0.9,
        repetition_penalty=1.2,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

    response = tokenizer.decode(outputs[0], skip_special_tokens=True)

    # Extract only assistant reply
    reply = response.split("Assistant:")[-1].strip()

    print(f"\nAssistant: {reply}\n")

    # Add assistant response to history
    chat_history += " " + reply

---

Intended Uses

This model can be used for:

• Instruction following
• Question answering
• Educational tasks
• Conversational AI experiments
• Research on parameter-efficient fine-tuning

---

Limitations

• The model was trained on a small subset of the dataset
• May produce hallucinated or incorrect information
• Not suitable for medical, legal, or critical decision making

---

Risks and Bias

As with most large language models, the model may reproduce biases present in the training data.

Users should carefully review outputs before using them in sensitive contexts.

---

Citation

If you use this model, please cite:

• UltraChat Dataset
• Mistral-7B Base Model

---

Author

Hemanth Naravi