koineformer

KoineFormer

A domain-adapted T5 encoder-decoder for Koine Greek, produced by training LoRA adapters on GreTa (a Classical Greek T5) with a 1.5M-token Koine corpus and a Classical Greek replay buffer.

Overview

KoineFormer adapts GreTa---a T5-base model trained on Classical and Medieval Greek by Heidelberg NLP---to Koine Greek, the Hellenistic dialect of the New Testament, Septuagint, and Apostolic Fathers.

The adaptation uses LoRA (Low-Rank Adaptation) to train only 3.7M of the model's 220M parameters, producing a 14 MB adapter checkpoint. Training takes under one hour on a single GPU.

Performance

Linear probe evaluation on the SynoptiQ Corpus test set:

KoineFormer improves POS accuracy by 1.30 points (28% relative error reduction) over zero-shot. Lemmatisation accuracy is comparable (82.4% vs.\ 81.3%)---span-corruption DAPT improves syntactic representations but does not expand vocabulary coverage. Full fine-tune lemma results are pending.

Intended Uses

• Part-of-speech tagging for Koine Greek texts (New Testament, Septuagint, Apostolic Fathers)
• Lemmatisation of Hellenistic Greek passages
• Feature extraction (encoder hidden states) for downstream tasks such as textual criticism, authorship attribution, and stylistic analysis
• Fine-tuning on task-specific Koine Greek datasets (e.g., dependency parsing, named entity recognition)
• Fill-in-the-blank text reconstruction for manuscript studies

Limitations

• Trained on ~1.5M Koine tokens — small by modern LM standards; may not generalize to rare vocabulary or hapax legomena
• Span-corruption DAPT improves syntax (POS) but not lexical knowledge (lemmatisation is flat at 82.4%). Do not use as a lemmatiser without task-specific fine-tuning
• DAPT corpus covers New Testament and Apostolic Fathers only; the Septuagint (~500K additional Koine tokens) is not yet included
• Not evaluated on non-literary Koine (papyri, inscriptions, ostraca)
• Generative output is illustrative, not production-quality; the model was trained for representation learning, not text generation

Usage

from peft import PeftModel
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

base = AutoModelForSeq2SeqLM.from_pretrained("bowphs/GreTa")
model = PeftModel.from_pretrained(
    base, "ainouche-abderahmane/koineformer"
).merge_and_unload()  # bake LoRA into base weights
tokenizer = AutoTokenizer.from_pretrained("bowphs/GreTa")
tokenizer.add_special_tokens({"pad_token": "[PAD]"})
model.resize_token_embeddings(len(tokenizer))

# Fill-in-the-blank: comma marks the missing word
text = "Ἀρχὴ τοῦ, Ἰησοῦ Χριστοῦ υἱοῦ θεοῦ."
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(
    **inputs, max_new_tokens=40, num_beams=5,
    no_repeat_ngram_size=3, repetition_penalty=2.0,
    early_stopping=True, pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
# → πρώτην τοῦ εὐαγγελίου ἰησοῦ χριστοῦ υἱοῦ θεοῦ.

Training Data

A Classical Greek replay buffer (First1KGreek: Homer, Plato, Xenophon) was interleaved at 30% to prevent catastrophic forgetting.

Training

• Objective: T5 span corruption (15% noise, 512-token packed sequences)
• Optimizer: AdamW (lr=1e-4, cosine to zero)
• Precision: FP16 (AMP)
• GPU: NVIDIA A10G (24 GB), 58 minutes
• Reproducibility: python scripts/train_dapt.py --smoke-test

Full pipeline at SynoptiQ.

Citation

@inproceedings{ainouche2026koineformer,
  title     = {KoineFormer: Domain-Adaptive Language Modeling for Koine Greek},
  author    = {Ainouche, Abderahmane},
  booktitle = {Proceedings of LaTeCH-CLfL},
  year      = {2026},
}

Dataset

Evaluated on the SynoptiQ Corpus.

License

CC-BY-SA 4.0 (MorphGNT share-alike requirement).