RoLlama3.1-8b-Instruct-DPO-4Bit-BB

Model Card for 4-bit RoLlama3.1-8b-Instruct-DPO

Built from RoLlama3.1-8b-Instruct-DPO, quantized to 4-bit.

This variant of RoLlama3.1-8b-Instruct-DPO provides a reduced footprint through 4-bit quantization, aimed at enabling usage on resource-constrained GPUs while preserving a high fraction of the model’s capabilities.

Model Details

Comparison to 16 bit

It loooks that the effects of the quantization are minimal :

Model Description

• Developed by: OpenLLM-Ro
• Language(s): Romanian
• License: cc-by-nc-4.0
• Quantized from model: RoLlama3.1-8b-Instruct-DPO
• Quantization: 4-bit

Quantization reduces model size and improves inference speed but can lead to small drops in performance. Below is a comprehensive table of the main benchmarks comparing the original full-precision version with the new 4-bit variant.

How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "OpenLLM-Ro/RoLlama3.1-8b-Instruct-DPO-4bit"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, device_map="auto")

instruction = "Ce jocuri de societate pot juca cu prietenii mei?"
chat = [
    {"role": "system", "content": "Ești un asistent folositor, respectuos și onest. Încearcă să ajuți cât mai mult prin informațiile oferite, excluzând răspunsuri toxice, rasiste, sexiste, periculoase și ilegale."},
    {"role": "user", "content": instruction},
]
prompt = tokenizer.apply_chat_template(chat, tokenize=False, system_message="")

inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt").to("cuda")
outputs = model.generate(input_ids=inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))