RoBERTaSense-FACIL

Model Card for RoBERTaSense-FACIL

RoBERTaSense-FACIL (RoBERTa Fine-tuned for Accessible Comprehension In Language) is a Spanish RoBERTa model fine-tuned to assess meaning preservation in Easy-to-Read (E2R) adaptations. Given a pair of texts {original, adapted}, it predicts whether the adaptation preserves the meaning of the original.

⚠️ Deprecation notice (base model): This model was fine-tuned from PlanTL-GOB-ES/roberta-base-bne. As for September 2025, this checkpoint is deprecated and no longer actively maintained. For actively maintained Spanish RoBERTa models, please see the BSC-LT organization: https://huggingface.co/BSC-LT

---

🚀 How to Use

  from transformers import AutoTokenizer, AutoModelForSequenceClassification
  import torch

  repo = "oeg/RoBERTaSense-FACIL"
  tokenizer = AutoTokenizer.from_pretrained(repo)
  model = AutoModelForSequenceClassification.from_pretrained(repo)
  model.eval()

  # Example pair
  original = "Juan llegó tarde ya que perdió el autobús."
  adapted = "Juan perdió el autobús. Por eso, Juan llegó tarde."

  # Tokenization
  inputs = tokenizer(
      original,
      adapted,
      return_tensors="pt",
      truncation=True,
      max_length=512
  )

  # Inference
  with torch.no_grad():
      logits = model(**inputs).logits
      probs = torch.softmax(logits, dim=-1)

  # Meaning preservation score (0–1)
  meaning_score = probs[0, 1].item()

  print("Meaning Preservation Score:", meaning_score)

---

Model Description

• Developed by: Ontology Engineering Group (UPM) / Authors: Isam Diab Lozano and Mari Carmen Suárez-Figueroa
• Funded by: "Ayudas para la contratación de personal investigador predoctoral en formación para el año 2022" (Reference: PIPF-2022/COM-25762) by Comunidad Autónoma de Madrid (Spain)
• Model type: Encoder-only Transformer (RoBERTa) with a classification head
• Language: Spanish (es)
• License: Apache-2.0
• Finetuned from model: PlanTL-GOB-ES/roberta-base-bne (deprecated; see notice above)

---

Uses

Direct Use

• Automatic scoring of meaning preservation for Spanish Easy-to-Read adaptations.
• As a signal in content quality checks for accessibility pipelines.

Out-of-Scope Use

• Clinical, legal, or other high-stakes decisions without human expert oversight.
• Non-Spanish or out-of-domain texts without prior adaptation or re-training.

---

Bias, Risks, and Limitations

• Domain limitation: trained for Spanish E2R; performance may degrade on other genres/domains.
• Binary labels: compress nuanced cases; borderline adaptations may require human review.
• Synthetic negatives: not all human errors are covered by synthetic negative strategies.
• Base deprecation: the upstream base model is deprecated; security/robustness updates won’t be inherited.

Recommendations

• Calibrate probabilities (e.g., temperature scaling) and expose confidence scores.
• Use threshold tuning (e.g., Youden’s J) to trade precision/recall for your setting.
• Keep a human-in-the-loop for critical use cases and periodic error audits.

---

How to Get Started with the Model

See How to Use above. For pairwise inputs, encode as sentence pairs:

inputs = tokenizer(text_original, text_adapted, return_tensors="pt", truncation=True, max_length=512)

---

Training Details

Training Data

• Source: Spanish pairs (original - adapted) curated/validated by experts.
• Columns: text1 (original), text2 (adaptation), Label (0/1), neg_type.
• Labels: 1 = PRESERVES_MEANING, 0 = DOES_NOT_PRESERVE.
• Negative types used in training data construction: shuffle, dropout, mismatch (derangement), paraphrase_distortion, nli_contradiction.
• Split: 80/20, stratified by Label (random_state=42).

Training Procedure

Preprocessing

• Pair tokenization with truncation at 512 tokens:

tokenizer(text1, text2, truncation=True, max_length=512)

Training Hyperparameters

• Training regime: fp16 mixed precision (if supported; otherwise fp32)

• Arguments:

  • num_train_epochs=5
  • per_device_train_batch_size=32
  • per_device_eval_batch_size=16
  • learning_rate=2e-5
  • weight_decay=0.01
  • warmup_ratio=0.1
  • evaluation_strategy="epoch", save_strategy="epoch"
  • load_best_model_at_end=True, metric_for_best_model="f1"

• Optimizer: AdamW

• Loss: CrossEntropy (2 logits)

Evaluation

Testing Data, Factors & Metrics

Testing Data

• Held-out 20% stratified split of the curated E2R pairs.

Factors

• Report per-negative-type breakdown (e.g., performance on mismatch, paraphrase_distortion, etc.).

Metrics

• Accuracy, F1, ROC-AUC.

Results

• Accuracy: 0.81
• F1: 0.84
• ROC-AUC: 0.83
• Threshold tuned via Youden’s J for operating point selection.

Technical Specifications

Model Architecture and Objective

• Encoder-only RoBERTa with a classification head (Linear(hidden → 2)).
• Objective: supervised cross-entropy on binary label.

---

Citation

BibTeX:

@software{roberta_facil_2025,
  title  = {RoBERTaSense-FACIL: Meaning Preservation for Easy-to-Read in Spanish},
  author = {Diab-Lozano, Isam and Suárez-Figueroa, Mari Carmen},
  year   = {2025},
  url    = {https://huggingface.co/oeg/RoBERTaSense-FACIL}
}

APA: Diab-Lozano, Isam and Suárez-Figueroa, Mari Carmen. (2025). RoBERTaSense-FACIL: Meaning Preservation for Easy-to-Read in Spanish. Hugging Face. https://huggingface.co/oeg/RoBERTaSense-FACIL