OmniGEC-Fluency-12B

Model Card for Model ID

OmniGEC-Minimal-12B extends the open-weight Gemma-3-12B-IT with instruction tuning and supervised fine-tuning on OmniGEC, a silver-standard GEC corpus that includes MultiGEC-25, Wikipedia, Reddit edits for 11 low-/mid-resource European languages. The result is a single model capable of paragraph-level correction across all covered languages achieving State-Of-The-Art (SOTA) results for paragraph-based editing in minimal and fluency tracks.

Per-language GLEU scores on MultiGEC-25 test set (Minimal edits)

Per-language GLEU scores on MultiGEC-25 test set (Fluency)

Training Data

Silver corrections were created with a three-step prompt → generate 3 candidates → aggregate pipeline using o1-preview and GPT-4o-mini.

Evaluation

• Metric: GLEU via the official MultiGEC-25 CodaLab evaluator (minimal & fluency tracks).
• Both OmniGEC-tuned models surpass the paragraph-based baseline LLaMA-3-8B by +9–10 GLEU on the minimal track and deliver the current best open scores for Estonian and Latvian.

OmniGEC-Tuned Checkpoints

AYA-Expanse-8B · Gemma-3-12B-IT

🔧 Quick start

pip install transformers
git clone https://github.com/r-kovalch/omnigec-models.git
cd multigec-models

from transformers import AutoTokenizer, AutoModelForCausalLM
from src.instruction_templates import multigec_prompts
from src.utils.multigec import LANG_TO_CODE, LANG_CODE_TO_TOKEN

# For AYA-based models (OmniGEC-Minimal-8B, OmniGEC-Fluency-8B)
def formatting_prompts_func(example):
    language_code = LANG_TO_CODE[example["language"]]
    language_token = LANG_CODE_TO_TOKEN[language_code]

    user_input = example['feature']
    prompt_template = multigec_prompts[example["language"]].prompt_template
    instruction = prompt_template.format(original_text=user_input)

    text = f"<|START_OF_TURN_TOKEN|><|USER_TOKEN|>{language_token}{instruction}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>"

    return text

# For Gemma-based models (OmniGEC-Minimal-12B, OmniGEC-Fluency-12B)
def formatting_prompts_func(example):
    language_code = LANG_TO_CODE[example["language"]]
    # Since special tokens for Gemma models does not have |, we remove them
    language_token = LANG_CODE_TO_TOKEN[language_code].replace("|", "")

    user_input = example['feature']
    prompt_template = multigec_prompts[example["language"]].prompt_template
    instruction = prompt_template.format(original_text=user_input)

    text = f"<start_of_turn>user\n{language_token}{instruction}<end_of_turn>\n<start_of_turn>model\n"

    return text

repo = "lang-uk/OmniGEC-Minimal-8B"   # or -Fluency-8B / -12B
tok  = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto")

lang = "en"
text = "She go to school every day ."
# Choose formatting func accordingly to base model Gemma/Aya
prompt = formatting_prompts_func(text)

out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=1600)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

• Reddit and UberText corrections are machine-generated; noise remains, esp. in slang.
• Sequences > 1,600 tokens are truncated unless you raise max_new_tokens.

Details

• For details on use, please refer to our GitHub
• We strongly recommend you to follow the inference code we used in notebooks for both gemma and aya, as there's additional parameters, like temperature, top_k, max_new_tokens and others, specific for each model.

Authors

Petro Ivaniuk, Mariana Romanyshyn, Roman Kovalchuk