Llama-3.1-8B-SiDiaC-CPT

Llama-3.1-8B-SiDiaC-CPT

A LoRA adapter continually pretraining Llama-3.1-8B (via unsloth/meta-llama-3.1-8b-unsloth-bnb-4bit) on the full SiDiaC-v.2.0 corpus. Built as part of a study on diachronic semantic change in Sinhala (see Paper below), where the resulting contextual embeddings feed a leave-one-out (LOO) analysis of per-lemma semantic drift across centuries.

The tokenizer is the stock Llama-3.1 tokenizer (128,256 tokens, unmodified — no vocabulary expansion for Sinhala). Instead, the token embedding and output layers are fine-tuned directly (modules_to_save: ["embed_tokens"], lm_head in target_modules) alongside standard LoRA adapters on the attention/MLP projections. This checkpoint is referred to as Llama_NO_EXPA / Llama_FT in the accompanying codebase.

Paper

Dynamics of meaning: Towards the Evaluation of Diachronic Semantic Change in Sinhala Nevidu Jayatilleke, Nisansa de Silva (2026)

📄 arXiv:2609.08609

Adapter details

Installation

unsloth has a history of version-compatibility issues with trl/peft/pyarrow/ datasets. This is the exact install sequence used to train and validate this checkpoint (Colab, CUDA GPU) — deviating from it is the most common cause of load failures:

pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps xformers trl peft accelerate bitsandbytes
pip install --upgrade transformers accelerate bitsandbytes

# If you hit trl/peft/datasets/pyarrow conflicts, reset them explicitly:
pip uninstall -y trl peft datasets pyarrow
pip install pyarrow==10.0.1
pip install trl peft

bitsandbytes 4-bit loading (used below) requires a CUDA GPU — it will not work on CPU-only or Apple Silicon (MPS) machines.

How to use

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/meta-llama-3.1-8b-unsloth-bnb-4bit",
    max_seq_length=2048,
    load_in_4bit=True,
)
model.load_adapter("Nevidu/Llama-3.1-8B-SiDiaC-CPT")
FastLanguageModel.for_inference(model)

Or with plain transformers + peft:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "unsloth/meta-llama-3.1-8b-unsloth-bnb-4bit", load_in_4bit=True
)
tokenizer = AutoTokenizer.from_pretrained("Nevidu/Llama-3.1-8B-SiDiaC-CPT")
model = PeftModel.from_pretrained(base, "Nevidu/Llama-3.1-8B-SiDiaC-CPT")

Training data

Continually pretrained on the full Nevidu/SiDiaC-v.2.0 corpus.

Training hyperparameters

Self-supervised continual pretraining (causal LM, next-token prediction) via trl.SFTTrainer with sequence packing:

Training results

Training ran for 1,125 steps before early stopping. Evaluation loss decreased from 8.62 (step 0) to a minimum of 4.80 at step 750, after which it began to rise — the step-750 checkpoint (this one) was restored as the final model per load_best_model_at_end.

Intended use

This checkpoint was built to extract per-century contextual word embeddings for diachronic semantic-drift analysis (see the paper), not for general-purpose Sinhala instruction-following or chat. It has not been evaluated for safety, factuality, or general-purpose downstream use, and should be treated as a research artifact rather than a production-ready model.

Citation

@article{jayatilleke2026dynamicsmeaningevaluationdiachronic,
      title={Dynamics of meaning: Towards the Evaluation of Diachronic Semantic Change in Sinhala}, 
      author={Nevidu Jayatilleke and Nisansa de Silva},
      year={2026},
      eprint={2609.08609},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.08609}, 
}

Framework versions

• PEFT 0.19.1