latvian-qwen-0.5b-lora-v2

Latvian Qwen2.5-0.5B LoRA V2

A Latvian language adaptation of Qwen2.5-0.5B using parameter-efficient LoRA continued pretraining.

This project explores adapting a small pretrained causal language model to Latvian using a reproducible data preparation, training, and evaluation pipeline.

Model

• Base model: Qwen/Qwen2.5-0.5B
• Architecture: Qwen2.5 causal language model
• Adaptation method: LoRA / PEFT
• Trainable parameters: 2,162,688
• Total parameters: 496,195,456
• Trainable parameter ratio: ~0.44%

This repository contains the LoRA adapter rather than a separate full copy of the base-model weights.

Training

V2 was trained using Latvian text from two training sources:

• 133,254 cleaned text rows from RaivisDejus/latvian-text
• 15,059 training sentences from the Latvian Treebank (LVTB)

The text was tokenized and packed into 512-token training sequences.

Training configuration:

• Epochs: 1
• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Learning rate: 5e-5
• Scheduler: cosine
• Warmup: 52 steps
• Batch size: 1
• Gradient accumulation: 8
• Sequence length: 512 tokens
• Precision: FP16
• Hardware: NVIDIA Tesla T4

Evaluation

V2 was evaluated on a held-out LVTB test set using the same evaluation procedure as the base model and V1 adapter.

Under this evaluation setup, V2 achieved lower test loss and perplexity than both the original base model and V1.

Intended Use

This model is intended primarily for:

• experimentation with Latvian language-model adaptation
• NLP research and education
• demonstrating parameter-efficient model training
• further Latvian language-model development
• evaluation of multilingual language models on Latvian text

It should not be considered a production-ready general-purpose Latvian language model.

Limitations

This is a relatively small-scale adaptation of a 0.5B-parameter multilingual base model.

Generated text can still contain:

• grammatical errors
• semantic inconsistencies
• unnatural phrasing
• repetition
• factual errors

The reported evaluation measures performance on the selected held-out LVTB test set. They should not be interpreted as a complete measure of Latvian language ability.

Data and Licensing

This repository does not declare a single standalone license for the adapter weights.

The model depends on a base model and training datasets with separate licensing terms. Users should review the applicable terms for each component before using, redistributing, or incorporating the adapter into another system.

Base model

Qwen2.5-0.5B is provided under the Apache-2.0 license.

Base model:

https://huggingface.co/Qwen/Qwen2.5-0.5B

Latvian Treebank

The Latvian Treebank (LVTB) is distributed under CC BY-SA 4.0.

Users should comply with the dataset's attribution and share-alike requirements.

Additional Latvian text dataset

The RaivisDejus/latvian-text dataset is currently listed on Hugging Face under CC BY 4.0.

Users should comply with the dataset's attribution requirements and review the dataset card and source-specific terms.

Loading the Adapter

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen2.5-0.5B"
adapter_id = "YOUR_USERNAME/latvian-qwen-0.5b-lora-v2"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    dtype=torch.float16,
)

model = PeftModel.from_pretrained(
    model,
    adapter_id,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
model.eval()

prompt = "Latvija ir"

inputs = tokenizer(
    prompt,
    return_tensors="pt"
).to(device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.15,
    )

print(
    tokenizer.decode(
        outputs[0],
        skip_special_tokens=True
    )
)