qwen2.5-7B-Instruct-dUO-finetuned-20260618-100steps

Qwen2.5-7B-Instruct LoRA — DOUE/BOE/BOPA Administrative Title Summarization

This is a LoRA adapter fine-tuned from Qwen2.5-7B-Instruct for generating summarization titles for administrative and legal texts published in Spanish/European official journals: DOUE (EU Official Journal), BOE (Spanish State Gazette), and BOPA (Asturias Regional Gazette).

Model Details

Model Description

A lightweight LoRA fine-tune on Qwen2.5-7B-Instruct that produces brief, clear, formal titles ready for publication from official administrative text while preserving essential meaning and eliminating redundancies.

• Developed by: Diego Gonzalez Suarez
• Model type: Qwen2ForCausalLM — decoder-only causal language model
• Fine-tuning method: LoRA (Low-Rank Adaptation) with 4-bit base model quantization
• Language(s): Spanish
• License: Apache 2.0
• Base model: Qwen/Qwen2.5-7B-Instruct
• Finetuned from model: Qwen/Qwen2.5-7B-Instruct

Base Model Architecture

LoRA Configuration

All 7 linear projection layers in attention and FFN blocks were targeted. The base model was loaded in 4-bit (NF4, double quantization) via bitsandbytes.

Uses

Direct Use

Generate titles from official administrative texts published in DOUE, BOE, and BOPA. The model takes a Spanish administrative/legal text as input and outputs a concise, formal title.

System Prompt

The model was trained with a system prompt instructing it to summarize administrative texts into formal titles while preserving essential meaning and eliminating unnecessary details and redundancies.

Training Details

Training Data

Custom dataset of approximately 200 examples (~86k tokens) sourced from:

• DOUE — Diario Oficial de la Unión Europea (EU Official Journal)
• BOE — Boletín Oficial del Estado (Spanish Official State Bulletin)
• BOPA — Boletín Oficial del Principado de Asturias (Official Bulletin of Asturias)

Training Procedure

Training Results

How to Get Started with the Model

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model in 4-bit
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype="float16",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "diegogs1451/qwen2.5-7B-Instruct-dUO")

messages = [
    {"role": "system", "content": "Eres un asistente experto en resumir textos oficiales administrativos y jurídicos del DOUE (Diario Oficial de la Unión Europea), del BOE (Boletín Oficial del Estado) y del BOPA (Boletín Oficial del Principado de Asturias) en títulos breves, claros, formales y listos para publicación. Conserva el significado administrativo esencial eliminando detalles innecesarios y redundancias"},
    {"role": "user", "content": "<input text>"}
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Technical Specifications

Software

• PEFT 0.15.2
• Transformers
• bitsandbytes (4-bit quantization)
• bfloat16 mixed precision
• TensorBoard, Weights & Biases (logging)

Model Card Contact

Diego Gonzalez Suarez [email protected]