koine-t5

<div align="center">

Koine-T5

One model, four tasks for Ancient Greek: POS tagging · lemmatization · text infilling · synoptic style transfer

A 104 MB LoRA adapter that turns GreTa into a multitask Ancient Greek workhorse — 96.6% POS token accuracy on New Testament Koine.

</div>

---

👉 A newer model is available

Koine-T5-Omni is this adapter's successor — same GreTa backbone, same LoRA rank, same 104 MB — and it is better on every POS metric while adding two further working tasks: full morphological parsing and diacritic restoration.

on PROIEL test, both re-scored with identical code	pooled POS token acc.	pooled exact match
Koine-T5-Omni	0.942	0.745
Koine-T5 (this model)	0.910	0.709

Prefer Omni for new work. Koine-T5 is not deprecated: it remains a valid model, this repo will not change, and the dev-split numbers published below stand as reported. (The table above is a separate head-to-head on the held-out test split under convention-neutral scoring, so its figures are not the same quantity as the 91.7% dev-split accuracy reported further down.)

---

What is this?

Koine-T5 is a LoRA adapter (r=64, 27.1M trainable parameters — 12% of the base) for bowphs/GreTa, a T5-base model pre-trained on Ancient Greek. It was trained jointly on four task pools sampled into every batch, so a single checkpoint handles all four — no task-specific heads, no per-task fine-tunes:

It was built as part of SynoptiQ, a neural source-criticism framework for the Synoptic Gospels, but the POS / lemma / infilling tasks are general-purpose: they cover both New Testament Koine and Classical Greek (Herodotus), thanks to the PROIEL treebank.

Results

Evaluated on the held-out UD Ancient Greek PROIEL dev set (greedy decoding, 250 NT + 250 Classical sentences; best checkpoint selected on pooled POS token accuracy):

For context: a dedicated linear probe on a DAPT'd GreTa encoder (KoineFormer) reaches 96.62% POS accuracy on a comparable NT corpus — Koine-T5 matches that via free seq2seq generation, while also lemmatizing, infilling, and paraphrasing with the same weights.

Quickstart

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PeftModel

tokenizer = AutoTokenizer.from_pretrained("bowphs/GreTa")
tokenizer.pad_token = "<pad>"   # GreTa ships <pad>=0 and </s>=1 but doesn't register them
tokenizer.eos_token = "</s>"
# Map the 100 T5 sentinels onto GreTa's pre-trained "ghost" slots (32003–32102).
# No embedding resize needed — the slots already exist in the checkpoint.
tokenizer.add_special_tokens(
    {"additional_special_tokens": [f"<extra_id_{i}>" for i in range(100)]}
)

base = AutoModelForSeq2SeqLM.from_pretrained("bowphs/GreTa")
model = PeftModel.from_pretrained(base, "ainouche-abderahmane/koine-t5")
model.eval()

⚠️ Do not call tokenizer.add_special_tokens({"pad_token": "[PAD]"}) + resize_token_embeddings (a common GreTa recipe). It desyncs the pad id from T5's decoder-start id and collapses generation. Bind the existing <pad> / </s> as above.

POS tagging

inputs = tokenizer("pos: καὶ φωνὴ ἐγένετο ἐκ τῶν οὐρανῶν", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=256, num_beams=1, do_sample=False)
tags = tokenizer.decode(out[0], skip_special_tokens=True).upper()
print(tags)
# C- N- V- P- RA N-
# καί=conjunction, φωνή=noun, ἐγένετο=verb, ἐκ=preposition, τῶν=article, οὐρανῶν=noun ✓

The .upper() matters: GreTa's tokenizer lowercases all text, so the model emits lowercase tag codes. MorphGNT codes are case-unique, so upper-casing restores them losslessly.

<details> <summary><b>MorphGNT tag codes</b> (click to expand)</summary> </details>

Lemmatization

inputs = tokenizer("lemma: καὶ φωνὴ ἐγένετο ἐκ τῶν οὐρανῶν", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=256, num_beams=1, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))
# καί φωνή γίγνομαι ἐκ ὁ οὐρανός

Lemma conventions follow the PROIEL treebank (the majority training source), e.g. Classical γίγνομαι rather than Koine γίνομαι. Output is lowercase (tokenizer property, see above).

Text infilling (span corruption)

No prefix — pass the masked text directly, exactly as in T5 pre-training:

text = "Ἀρχὴ τοῦ εὐαγγελίου Ἰησοῦ <extra_id_0> καθὼς γέγραπται ἐν τῷ <extra_id_1> τῷ προφήτῃ"
inputs = tokenizer(text, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64, num_beams=4, early_stopping=True)
print(tokenizer.decode(out[0], skip_special_tokens=False))
# <extra_id_0> χριστοῦ <extra_id_1> νόμῳ

The model fills Mark 1:1–2's gaps with χριστοῦ (exact) and νόμῳ ("the law" — the gold is Ἠσαΐᾳ, but grammatically and idiomatically coherent). Another example: καὶ <extra_id_0> ἐγένετο ἐκ τῶν <extra_id_1> → κραυγὴ / οὐρανῶν ("a cry came from the heavens").

Synoptic style transfer (experimental)

mark = "καὶ πρωῒ ἔννυχα λίαν ἀναστὰς ἐξῆλθεν καὶ ἀπῆλθεν εἰς ἔρημον τόπον κἀκεῖ προσηύχετο."  # Mark 1:35
inputs = tokenizer("synoptic mark_to_luke: " + mark, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=256, num_beams=4,
                     repetition_penalty=1.25, no_repeat_ngram_size=3,
                     encoder_no_repeat_ngram_size=3, early_stopping=True)
print(tokenizer.decode(out[0], skip_special_tokens=True))
# ἰδὼν δὲ ὁ ἰησοῦς τοὺς ὄχλους ἀνέβη εἰς τὸ ὄρος προσεύξασθαι καὶ ἐγένετο ἐν τῷ
# προσεύχεσθαι αὐτὸν καὶ αὐτὸς ἦν διανυκτερεύων ἐν τῷ ἱερῷ καὶ προσευχόμενος

This pool is tiny (155 curated Mark→Matthew / Mark→Luke parallel pericope pairs from the SynoptiQ corpus), so treat outputs as stylistically evocative, not faithful translations: the example above is fluent, recognizably Lukan-flavored Greek about withdrawing to pray (ἀνέβη εἰς τὸ ὄρος προσεύξασθαι, διανυκτερεύων — cf. Luke 6:12), but it is a free composition, not Luke's actual parallel.

Training

Data. Three sources, four pools:

PROIEL contributes ~214K tokens spanning NT Koine (Gospels, Acts, Epistles, Revelation) and Classical Greek (Herodotus' Histories) — which is why the model tags Classical text at 87.7% despite the Koine focus. POS codes come from PROIEL's fine-grained XPOS column mapped to the 13-code MorphGNT tagset (not UPOS/FEATS, whose PronType=Dem on the article would mislabel it).

Balanced sampling. Every micro-batch draws each of its 4 slots from a task pool with weights pos: 3/8, denoise: 3/8, lemma: 1/8, synoptic: 1/8 (with replacement per task). The 155-example synoptic pool is upsampled so it can never be starved out, and the massive PROIEL pools can never drown it — one adapter serves all four tasks without catastrophic forgetting.

Procedure. LoRA r=64, α=128, dropout 0.05 on all attention (q,k,v,o) and FFN (wi,wo) projections of encoder and decoder; base frozen in bfloat16. 30,000 micro-steps (batch 4 × grad-accum 8 = effective 32), AdamW lr 1e-4, weight decay 0.01, 5% warmup then cosine to zero, max sequence length 256, single A10G GPU on Modal. Checkpoint selection: pooled POS token accuracy on PROIEL dev every 1,000 steps; this repo ships the best checkpoint (step 28,000).

Limitations

• Lowercase output. GreTa's tokenizer case-folds everything; the model cannot produce capital letters. Upper-case POS codes yourself (lossless); for lemmas/prose, expect lowercase.
• 256-token window. Inputs beyond ~2–3 verses are truncated. Tag long passages sentence-by-sentence.
• Classical Greek is the weaker domain (87.7% vs 96.6% token accuracy) — the training mix, the tokenizer (1.38 subwords/word on Koine vs 1.95 on Classical), and the eval all favor Koine.
• Lemma conventions are PROIEL's, which can differ from NT dictionaries (γίγνομαι vs γίνομαι).
• Synoptic transfer is exploratory. 155 training pairs produce style pastiche, not reliable parallel reconstruction. Do not use it for text-critical claims.
• No copying-direction inference. This model does not (and, per the SynoptiQ project's negative result, cannot) determine the direction of literary dependence between gospel texts.

License and attribution

Released under CC BY-NC-SA 4.0 (NonCommercial, ShareAlike). The adapter was trained on the UD Ancient Greek PROIEL treebank (CC BY-NC-SA 3.0), and this license mirrors that source's terms; the other sources are SBLGNT (CC BY) and MorphGNT (CC BY-SA), via the SynoptiQ corpus (CC BY-SA 4.0). The base model bowphs/GreTa is Apache-2.0 and is not included in this repo — only the LoRA adapter is.

Related work

• Koine-T5-Omni — the successor to this model. Six working tasks instead of four (adds morphological parsing and diacritic restoration), +3.2 pp pooled POS token accuracy and +3.6 pp exact match on the held-out PROIEL test split, same size and same backbone. Start there for new work.
• KoineFormer — sibling adapter: GreTa DAPT'd on Koine prose (SBLGNT + Apostolic Fathers), for encoder/probing use (96.62% POS via linear probe, 14 MB).
• SynoptiQ corpus — 49,061 morphologically annotated tokens, 170 pericopes, 235 alignments across Matthew, Mark, and Luke.
• SynoptiQ — the parent project: neural source criticism of the Synoptic Gospels.

Citation

@misc{ainouche2026koinet5,
  author = {Ainouche, Abderahmane},
  title  = {Koine-T5: a multitask LoRA adapter for Ancient Greek
            (POS tagging, lemmatization, infilling, synoptic style transfer)},
  year   = {2026},
  url    = {https://huggingface.co/ainouche-abderahmane/koine-t5},
  note   = {LoRA adapter for bowphs/GreTa, trained on UD Ancient Greek PROIEL
            and the SynoptiQ corpus}
}

Please also cite the base model (Riemenschneider & Frank, 2023) and, if you use the POS/lemma tasks, the PROIEL treebank (Haug & Jøhndal, 2008).