roberta-spanish-cefr

Spanish CEFR Classification with BERTIN

Model summary

pymlex/roberta-spanish-cefr is a Spanish text classifier fine-tuned from bertin-project/bertin-roberta-base-spanish for CEFR level prediction. It is intended for Spanish learner-text classification and readability-style proficiency assessment.

Training data

The model was trained on UniversalCEFR/caes_es, a Spanish dataset of learner texts with CEFR annotations. The dataset has 31.1k rows.

Evaluation

Results for the test set:

• Accuracy: 0.9882
• Precision: 0.9896
• Recall: 0.9892
• F1: 0.9894

Comparison with other CEFR Spanish classifiers

Our model's performance (F1: 0.9894) is SOTA. Most documented Spanish CEFR classifiers fall within the 0.75 – 0.88 F1-score range. The obtained results significantly outperform these common baselines:

Inference

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "pymlex/roberta-spanish-cefr"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

def predict_cefr(text, top_k=3):
    inputs = tokenizer(
        text,
        return_tensors="pt",
        truncation=True,
        max_length=512,
    )
    with torch.no_grad():
        logits = model(**inputs).logits
        probs = torch.softmax(logits, dim=-1)[0]

    k = min(top_k, probs.numel())
    values, indices = torch.topk(probs, k=k)

    return [
        {
            "label": model.config.id2label[i.item()],
            "score": float(v.item()),
        }
        for i, v in zip(indices, values)
    ]

text = "Estimados señores, les escribo para solicitar información sobre el curso."
print(predict_cefr(text, top_k=3))