GIIS-Mini-LoRA-adapter

Model Card for granite-30b-sft-citation-lora

This model is a fine-tuned version of ibm-granite/granite-4.1-30b. It has been trained using TRL.

Quick start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

MODEL_ID = "Prience91/GIIS-Mini-LoRA"
BASE_ID = "ibm-granite/granite-4.1-30b"

tokenizer = AutoTokenizer.from_pretrained(BASE_ID)

# 1. Reconstruct the format used during dataset creation
system_prompt = "You are a helpful assistant. Answer the question based on the provided documents using citations."

# Format documents exactly as they appeared in your dataset
documents_context = """
Doc(id=1): Climate change is causing regional shifts in vegetation communities and altering species distributions across altitude limits.
Doc(id=2): Rising global temperatures increase the frequency of extreme droughts and wildfire risks in forest ecosystems.
"""

user_query = "What are the main impacts of climate change on local biodiversity?"

# 2. Construct the full prompt matching your SFT template
messages = [{"role": "system", "content": system_prompt},
           {"role": "user", "content": f"Context:\n{documents_context}\n\nQuestion: {user_query}"}]

prompt = tokenizer.apply_chat_template(messages,
                                       tokenize=False,
                                       add_generation_prompt=True)

# 3. Load Model with 4-bit Quantization
bnb_config = BitsAndBytesConfig(load_in_4bit=True,
                                bnb_4bit_quant_type="nf4",
                                bnb_4bit_compute_dtype=torch.bfloat16)

base_model = AutoModelForCausalLM.from_pretrained(BASE_ID,
                                                  quantization_config=bnb_config,
                                                  dtype=torch.bfloat16,
                                                  device_map="auto")

model = PeftModel.from_pretrained(base_model, MODEL_ID)

# 4. Generate with Greedy Decoding (do_sample=False prevents sampling loops)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(**inputs,
                             max_new_tokens=256,
                             do_sample=False,  
                             pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,
                             eos_token_id=tokenizer.eos_token_id)

response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print("--- MODEL OUTPUT ---")
print(response)

Training procedure

<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>

This model was trained with SFT.

Framework versions

• TRL: 1.9.1
• Transformers: 5.14.1
• Pytorch: 2.11.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citations

Cite TRL as:

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}