sedici-llama-dora-r32-all

SEDICI Computer Science Document Type Classifier

DoRA Adapter for Meta-Llama-3.1-8B-Instruct (r=32)

Author: Malek Camilo
Institution: Facultad de Informática, Universidad Nacional de La Plata
Thesis: Estrategias de adaptación eficiente para modelos de lenguaje abiertos
Repository: github.com/malekcamilo/tesis_maestria

This repository contains a DoRA adapter for document type classification in the Computer Science subset of SEDICI records.

Base model

[!WARNING] Base Model License: This adapter is built on top of meta-llama/Meta-Llama-3.1-8B-Instruct, which is a gated model. Users must explicitly agree to the Meta Llama 3.1 Community License on Hugging Face to use it.

The adapter was trained on top of meta-llama/Meta-Llama-3.1-8B-Instruct.

Users must have access to the base model in order to load this adapter.

Task

Given a title and abstract, the model predicts one of the following document types:

• Article
• Conference Paper
• Thesis
• Book
• Learning Resource
• Other

The generated label is returned in Spanish, matching the labels used during training.

Training data

The dataset contains 19,974 records from SEDICI in the Computer Science domain. Each instance includes title, abstract and document type.

Tokenizer

This adapter uses the original tokenizer distributed with Meta-Llama-3.1-8B-Instruct. No additional tokens or vocabulary modifications were introduced during fine-tuning.

Method

The model was adapted using DoRA (rank = 32) targeting all attention and MLP projection layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj).

Hardware

Training was performed on an Intel® Data Center GPU Max 1550 using Intel Extension for PyTorch (IPEX) with bfloat16 precision.

Training Configuration

Training Metrics

• Train Time: 16.53 hours
• Peak Memory: 22.85 GB
• Trainable Parameters: 85262336

Evaluation

Per-Class Results

Prompt Format

The adapter was trained using the Meta-Llama 3.1 chat template with the following system instruction:

System:
Eres un clasificador automático de documentos académicos. Tu única tarea es asignar la categoría correcta a los registros bibliográficos provistos.
Regla estricta: Debes responder ÚNICAMENTE con el nombre exacto de la categoría. No incluyas explicaciones, puntuación adicional ni texto conversacional.
Categorías válidas: Articulo, Objeto de conferencia, Tesis, Libro, Otro, Objeto de aprendizaje.

Usage

import torch

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = "meta-llama/Meta-Llama-3.1-8B-Instruct"
adapter = "malekcamilo/sedici-llama-dora-r32-all"

tokenizer = AutoTokenizer.from_pretrained(adapter)

model = AutoModelForCausalLM.from_pretrained(
    base_model,
    device_map="auto",
)

model = PeftModel.from_pretrained(model, adapter)
model.eval()


def classify(title: str, abstract: str) -> str:

    messages = [
        {
            "role": "system",
            "content": (
                "Eres un clasificador automático de documentos académicos. "
                "Tu única tarea es asignar la categoría correcta a los registros bibliográficos provistos.\n"
                "Regla estricta: Debes responder ÚNICAMENTE con el nombre exacto de la categoría. "
                "No incluyas explicaciones, puntuación adicional ni texto conversacional.\n"
                "Categorías válidas: Articulo, Objeto de conferencia, Tesis, Libro, Otro, Objeto de aprendizaje."
            ),
        },
        {
            "role": "user",
            "content": (
                f"Clasifica el siguiente registro:\n"
                f"<titulo>{title}</titulo>\n"
                f"<resumen>{abstract}</resumen>"
            ),
        },
    ]

    inputs = tokenizer.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        return_tensors="pt",
    )

    inputs = {k: v.to(model.device) for k, v in inputs.items()}

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=10,
            do_sample=False,
        )

    generated = outputs[0][inputs["input_ids"].shape[-1]:]

    return tokenizer.decode(
        generated,
        skip_special_tokens=True,
    ).strip()


prediction = classify(
    "Clasificación automática de documentos científicos",
    "En este trabajo se propone un método para clasificar documentos utilizando grandes modelos de lenguaje."
)

print(prediction)

Reproducibility

• Base model: Meta-Llama-3.1-8B-Instruct
• Fine-tuning: DoRA
• Rank: 32
• Seed: 42
• Dataset size: 19,974 documents
• Sequence length: 512
• Precision: bfloat16
• Frameworks:
  • Transformers
  • PEFT
  • TRL
  • Intel Extension for PyTorch (IPEX)

Limitations

This adapter was trained exclusively on Computer Science records from SEDICI. Performance on other academic repositories, languages or taxonomies has not been evaluated. The model is intended for research purposes.

Citation

If you use this adapter in your research, please cite the associated Master's thesis:

@mastersthesis{camilo2026peft,
  author = {Malek Camilo},
  title  = {Estrategias de adaptación eficiente para modelos de lenguaje abiertos},
  school = {Facultad de Informática, Universidad Nacional de La Plata},
  year   = {2026},
  type   = {Master's thesis},
  note   = {Manuscript submitted in partial fulfillment of the requirements for the Master's degree},
}