Flan-T5-Large_Grammar_Ro

Flan-T5_Grammar (Romanian)

<h5 style="font-family: 'Calibri'; margin-bottom: 20px;"> <a href="https://huggingface.co/google/flan-t5-large" target="_blank">Flan-T5-Large</a> is an instruction-tuned language model that treats all NLP tasks as text-to-text problems, excelling at grammar-related tasks like correction, rephrasing, and sentence completion through natural language prompts.</h5>

---

<h2>Dataset<h2> <h5 style="font-family: 'Calibri'; margin-bottom: 20px;"> I fine-tuned Flan-T5-Large on <a href="https://huggingface.co/datasets/upb-nlp/gec-ro-comments" target="_blank">gec-ro-comments</a> and <a href="https://huggingface.co/datasets/upb-nlp/gec_ro_cna" target="_blank">gec-ro-cna</a> datasets. The split was created by combining train (635 pairs), test (686 pairs), validation (666) from gec-ro-comments and train (1286) from gec-ro-cna to create the training set and test (1234) from gec-ro-cna for testing. </h5>

---

<h2>Configuration<h2> <h5 style="font-family: 'Calibri'; margin-bottom: 2px;"> <ul> <li><strong>Model</strong> = “google/flan-t5-large”</li> <li><strong>Learning rate</strong> = 5e-5</li> <li><strong>Batch size</strong> = 4 (for both dataloaders)</li> <li><strong>Optimizer</strong> = AdamW</li> <li><strong>Epochs</strong> = 10</li> <li><strong>Scheduler</strong> = Linear (with warmup = 0.1)</li> </ul> </h5> <h5 style="font-family: 'Calibri'; margin-bottom: 20px;"> The condition for saving the model is that the test loss, wer, cer must be lower than the previously recorded best values. </h5>

---

<h2>Results</h2>

<img src="https://huggingface.co/ionut-visan/Flan-T5-Large_Grammar_Ro/resolve/main/loss_plot.png" alt="Error Rates Plot" width="350" style="margin-left: 10px;"> <img src="https://huggingface.co/ionut-visan/Flan-T5-Large_Grammar_Ro/resolve/main/error_rates_plot.png" alt="Loss Plot" width="350" style="margin-left: 10px;"> <img src="https://huggingface.co/ionut-visan/Flan-T5-Large_Grammar_Ro/resolve/main/learning_rate_plot.png" alt="Loss Plot" width="350" style="margin-left: 10px;"> <img src="https://huggingface.co/ionut-visan/Flan-T5-Large_Grammar_Ro/resolve/main/bleu_plot.png" alt="Loss Plot" width="350" style="margin-left: 10px;"> <img src="https://huggingface.co/ionut-visan/Flan-T5-Large_Grammar_Ro/resolve/main/gleu_plot.png" alt="Loss Plot" width="350" style="margin-left: 10px;"> <img src="https://huggingface.co/ionut-visan/Flan-T5-Large_Grammar_Ro/resolve/main/rouge1_plot.png" alt="Loss Plot" width="350" style="margin-left: 10px;"> <img src="https://huggingface.co/ionut-visan/Flan-T5-Large_Grammar_Ro/resolve/main/rouge2_plot.png" alt="Loss Plot" width="350" style="margin-left: 10px;"> <img src="https://huggingface.co/ionut-visan/Flan-T5-Large_Grammar_Ro/resolve/main/rougeL_plot.png" alt="Loss Plot" width="350" style="margin-left: 10px;">

<h5 style="font-family: 'Calibri'; margin-bottom: 5px;"> The fine-tuned model was saved at epoch 5 with Test Loss: 0.3151, WER: 0.0893, CER: 0.0304, BLEU: 0.8424, GLEU: 0.8405, ROUGE-1: 0.9294, ROUGE-2: 0.8723, ROUGE-L: 0.9279. </h5>

---

<h2>How to use<h2>

import torch
from transformers import T5Tokenizer, T5ForConditionalGeneration

# Set device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Load model and tokenizer
model_name = "ionut-visan/Flan-T5-Large_Grammar_Ro"
tokenizer = T5Tokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name).to(device)
model.eval()

# Function to correct grammar
def correct_sentence(sentence):
    input_text = "grammar: " + sentence
    inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
    with torch.no_grad():
        outputs = model.generate(inputs, max_length=128, num_beams=4, early_stopping=True)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Interactive loop
print("Romanian Grammar Corrector (type 'exit' to quit)")
while True:
    user_input = input("\nEnter a sentence to correct: ")
    if user_input.lower() == "exit":
        print("Exiting. 👋")
        break
    corrected = correct_sentence(user_input)
    print("Corrected:", corrected)

---

<h2>Communication<h2> <h5 style="font-family: 'Calibri'; margin-bottom: 2px;"> For any questions regarding this model or to explore collaborations on ambitious AI/ML projects, please feel free to contact me at: </h5> <h5 style="font-family: 'Calibri'; margin-bottom: 2px;"> <ul> <li><em>[email protected]</em></li> <li><em><a href="https://www.linkedin.com/in/ionut-visan/" target="_blank">Ionuț Vișan's Linkedin</a></em></li> </ul> </h5>