byt5-small-geccc-mate

Model Card for byt5-small-geccc-mate

The byt5-small-geccc-mate model is a sequence-to-sequence model performing grammar error correction in Czech described in the paper Refining Czech GEC: Insights from a Multi-Experiment Approach. It is a finetuned version of byt5-small using the MATE method and the GECCC dataset.

Model Description

• Developed by: Seznam.cz and Charles University, MFF, ÚFAL
• Language(s) (NLP): Czech
• Model type: character-based encoder-decoder Transformer model
• Finetuned from model: google/byt5-small
• Finetuned on:
  • first synthetic errors generated by the MATE method (see the paper)
  • then the GECCC dataset
• License: CC BY-NC-SA 4.0

Model Sources

• Repository: https://github.com/ufal/tsd2025-gec
• Paper: Refining Czech GEC: Insights from a Multi-Experiment Approach
• Dataset: GECCC dataset

Evaluation

<div align="center"> <img src="https://github.com/ufal/tsd2025-gec/blob/main/figures/bubble_chart.svg?raw=true" width="75%" alt="Performance bubblechart" /> </div>

Uses

The model can be directly used to process space-tokenized input Czech text and produce grammar-corrected Czech text.

How to Get Started with the Model

Use the code below to get started with the model. Note that the input must be space-tokenized, i.e., every token (using the UDPipe 1 tokenizer czech-pdt-ud-2.5-191206.udpipe) must be space-separated.

tokenizer = transformers.AutoTokenizer.from_pretrained("ufal/byt5-small-geccc-mate")
model = transformers.AutoModelForSeq2SeqLM.from_pretrained("ufal/byt5-small-geccc-mate")

batch = tokenizer(["Sveřepý šakali zavile vyly na býlí mesýc ."], return_tensors="pt")
outputs = model.generate(batch.input_ids, max_length=256, num_beams=4)

print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

BibTeX Citation

@InProceedings{10.1007/978-3-032-02551-7_7,
  author="Pechman, Petr and Straka, Milan and Strakov{\'a}, Jana and N{\'a}plava, Jakub",
  editor="Ek{\v{s}}tein, Kamil and Konop{\'i}k, Miloslav and Pra{\v{z}}{\'a}k, Ond{\v{r}}ej and P{\'a}rtl, Franti{\v{s}}ek",
  title="Refining Czech GEC: Insights from a Multi-experiment Approach",
  booktitle="Text, Speech, and Dialogue",
  year="2026",
  publisher="Springer Nature Switzerland",
  address="Cham",
  pages="64--76",
  isbn="978-3-032-02551-7",
  doi="10.1007/978-3-032-02551-7_7"
}