maltese-ocr-trocr

Maltese OCR — TrOCR char-decoder (best_v4)

A printed-text OCR model for Maltese, built for the DocEng 2026 "OCRs for Corpus Extraction for the Maltese Language" competition. It is a TrOCR vision-encoder-decoder whose decoder has been re-vocabularised to a character-level Maltese vocabulary and fine-tuned only on synthetic data (per the competition's synthetic-data-only rule).

• Architecture: ViT/DeiT image encoder (768-d, 384×384 input, 16-px patches)
  • Transformer text decoder (1024-d, 12 layers) — a VisionEncoderDecoderModel.
• Vocabulary: 120-token character vocab (<pad>/<bos>/<eos> + 117 characters), covering the Maltese alphabet (Ċ ċ Ġ ġ Ħ ħ Ż ż), Latin letters, digits, punctuation and the accented/typographic characters seen in the corpus. This replaces TrOCR's default subword (RoBERTa) vocabulary.
• Dtype: load in fp16 on CUDA (the eval GPU is a Turing RTX 2080 Ti, which has no native bf16), fp32 on CPU.
• Dev CER: ≈ 0.3912 — official corpus-aggregate Character Error Rate on the 422-image competition dev set, computed with evaluate.load('cer'). (The training-time per-step eval reported 0.3945 at step 36 000; see best.json.)

Scope. This is a synthetic-data-only research model. The competition's Tesseract + NOMOCRAT baseline scores a lower CER but is excluded here by the synthetic-data-only rule and by NOMOCRAT licensing/provenance, so it is reported only as a baseline — not submitted.

How to use

The competition interface is a no-argument constructor plus a single transcribe(PIL.Image) -> str method:

from PIL import Image
from competition_transcriber import CompetitionTranscriber  # ships with the submission

transcriber = CompetitionTranscriber()          # downloads this repo on first use, then caches
text = transcriber.transcribe(Image.open("line.png"))
print(text)

CompetitionTranscriber downloads this repo via from_pretrained(...) (model + processor) plus one hf_hub_download(...) for the character tokenizer, then runs beam search (num_beams=2, no_repeat_ngram_size=3, repetition_penalty=1.2, length_penalty=1.0, max_new_tokens=511) at batch size 1.

To load the pieces directly:

import json, torch
from huggingface_hub import hf_hub_download
from transformers import TrOCRProcessor, VisionEncoderDecoderModel

REPO = "BarzinV/maltese-ocr-trocr"
dtype = torch.float16 if torch.cuda.is_available() else torch.float32
model = VisionEncoderDecoderModel.from_pretrained(REPO, torch_dtype=dtype).eval()
processor = TrOCRProcessor.from_pretrained(REPO)          # image preprocessing
chars = json.load(open(hf_hub_download(REPO, "char_tokenizer.json")))["chars"]
id_to_token = ["<pad>", "<bos>", "<eos>", *dict.fromkeys(chars)]  # ids 0/1/2 are special

px = processor.image_processor(images=img.convert("RGB"), return_tensors="pt").pixel_values.to(dtype)
ids = model.generate(pixel_values=px, num_beams=2, no_repeat_ngram_size=3,
                     repetition_penalty=1.2, length_penalty=1.0, max_new_tokens=511)[0].tolist()
text = "".join(id_to_token[i] for i in ids if i > 2)       # skip pad/bos/eos

Note: decoding uses the character tokenizer in char_tokenizer.json, not the processor's bundled RoBERTa tokenizer (which is vestigial — only the image processor is used). The decoder's positional table is 512, so cap generation at 511 new tokens.

Training

• Data — synthetic only. Maltese sentences from the MLRS/korpus_malti corpus were rendered to images by a custom renderer (multiple serif/sans fonts, regular/bold/italic, varied sizes, widths, justification and backgrounds), then fed as image→text pairs. No real/scanned competition images were used for training.
• Lineage. Stage 1 self-supervised SeqCLR pretraining of the ViT encoder → Stage 2 supervised fine-tune after swapping the decoder to the 120-token character vocabulary. The supervised stage went v2 → v3 → v4.
• The short-crop fix (the v4 win). Earlier checkpoints (v2) handled full paragraphs but ran away on short crops — over-generating long hallucinated tails (≈11 inserted chars/image on the shortest [0–128]-px bucket). v4 fixes this with two changes to the synthetic pipeline/objective:
  1. Crop rendering — adding short text fragments / cropped lines to the training mix so the model sees short targets, not only full paragraphs; and
  2. EOS-weighted loss — up-weighting the end-of-sequence token (×2.5) so the decoder learns to stop on short inputs. Together these crushed the short-crop CER from 0.68 → 0.069 and over-generation on short crops from ≈11.0 → ≈0.99 inserted chars/image, taking mean-per-image CER from 0.599 (v2) to 0.1512 (v4) and corpus-aggregate CER to 0.3912.
• Decode tuning. With v4's over-generation fixed, a read-only length-penalty sweep found length_penalty=1.0 optimal (a <1 penalty only over-truncates legitimate longer text); this is what the submission uses.

Files in this repo

Limitations

• Synthetic→real domain gap: dev CER (≈0.39) is far above the model's synthetic-eval accuracy; errors concentrate on noisy/low-resolution real crops.
• The decoder's 512-position limit means inputs whose ground-truth text exceeds ~511 characters cannot be fully transcribed (a small fraction of dev images).
• Trained for Maltese printed text; not intended for handwriting or other languages.

License

Released under the MIT license, matching the base model microsoft/trocr-base-handwritten.