mistral-7b-dolly-qlora

Mistral-7B Dolly-15K QLoRA Adapter

This repository contains a QLoRA adapter trained on top of mistralai/Mistral-7B-v0.3 using the databricks/databricks-dolly-15k instruction dataset.

The base model was loaded using 4-bit NF4 quantization and remained frozen. Only the LoRA adapter parameters were optimized.

Training configuration

The adapter targeted the q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj linear layers.

Dataset preparation

Eight exact duplicates were removed. Repeated complete inputs were assigned only to training, producing zero instruction-plus-context overlap among the training, validation, and test splits.

Examples longer than 1,024 tokens were filtered instead of silently truncated.

Held-out test results

These results use all 1,446 untouched test examples.

• Test-loss reduction: 16.74%
• Perplexity reduction: 39.87%
• Token-accuracy improvement: 3.4698 percentage points

Generation evaluation

Generation was evaluated on a reproducible balanced sample of 80 held-out examples, containing 10 examples from every Dolly category.

QLoRA achieved a higher BERTScore on 70/80 examples.

Loading the adapter

import torch

from peft import PeftModel
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig
)

base_model_id = "mistralai/Mistral-7B-v0.3"
adapter_id = "shoron07/mistral-7b-dolly-qlora"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(adapter_id)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    revision="caa1feb0e54d415e2df31207e5f4e273e33509b1",
    quantization_config=quantization_config,
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    adapter_id
)

prompt = (
    "### Instruction:\n"
    "Explain why the sky appears blue.\n\n"
    "### Response:\n"
)

inputs = tokenizer(
    prompt,
    return_tensors="pt"
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=200,
        do_sample=False
    )

print(
    tokenizer.decode(
        output[0],
        skip_special_tokens=True
    )
)

Limitations

• The adapter was trained for one epoch on Dolly-15K.
• Dataset answers can contain outdated or incorrect information.
• ROUGE and BERTScore do not directly measure factual correctness.
• Generation metrics used a balanced 80-example test sample.
• The model has not undergone dedicated safety evaluation.
• Generated information should be independently verified.