roberta-incoherence-classifier

<h1 align="center">roberta-incoherence-classifier</h1>

Encoder-based classifier for document inconsistency detection in Polish. This model evaluates the semantic consistency between two text fragments (e.g. sections of legal, procurement or organizational documents). It follows an NLI-like setup but redefines labels specifically for document coherence auditing. This model was initalized from PKOBP/polish-roberta-8k and adapted into an inconsistency classifier through supervised training on high-quality document-style pairs.

---

Intended Use

• Document consistency auditing (legal, public tender, IT documentation, organizational materials)
• Detecting contradicting statements, scope mismatches, term/role/format inconsistencies
• NLI‑like semantic relation classification with adapted label semantics

Not intended for:

• Fact-checking against external world knowledge
• Non‑Polish language inputs
• General misinformation / sentiment / toxicity detection

Finetuning on specific domain data is recommended for best production accuracy.

---

Label Definition (Adapted vs. Classical NLI)

Rule: A single critical mismatch (date / territory / authority / format / obligation vs. optional) is sufficient for contradiction, even if most of the text agrees.

---

Model Details

• Base architecture: RoBERTa‑large (encoder‑only)
• Classification head: standard HF linear head on pooled representation
• Language: Polish only
• License: CC-BY-SA 4.0
• Repository: asseco-group/roberta-incoherence-classifier

---

Training

• Precision: bfloat16
• Epochs: 5
• Global batch: 96 × 2 devices, gradient_accumulation_steps=11
• Learning rate: 2e-5, warmup ratio: 0.1, weight decay: 0.01
• Label smoothing: 0.05
• Gradient checkpointing: True
• Model selection: best macro F1 on validation

---

Dataset

• ~1.3M labeled pairs (train + val + test)
• Balanced class distribution
• Data sources include:
  • Polish subset of MoritzLaurer/multilingual-NLI-26lang-2mil7 (only high‑quality Polish NLI pairs)
  • Synthetic high‑quality document‑style pairs generated specifically for this inconsistency detection task
  • No additional classical NLI datasets were used, standard NLI label semantics do not fully align with this model’s stricter document‑consistency definitions
• Focus on Polish formal/procedural language (laws, tenders, IT specs, institutional instructions)

---

Evaluation (on asseco-group/incoherence-bench, test split)

               precision    recall  f1-score   support
   entailment       0.94      0.90      0.92       150
      neutral       0.87      0.91      0.89       150
contradiction       0.93      0.93      0.93       150

     accuracy                           0.91       450
    macro avg       0.91      0.91      0.91       450
 weighted avg       0.91      0.91      0.91       450

While the task is NLI-like, the label semantics are redefined for document-level procedural consistency, for which no direct open-source baselines currently exist.

---

Usage Example (Transformers)

import torch
from transformers import pipeline

device = "cuda" if torch.cuda.is_available() else "cpu"

classifier = pipeline(
    "text-classification",
    model="asseco-group/roberta-incoherence-classifier",
    tokenizer="asseco-group/roberta-incoherence-classifier",
    top_k=None,
    return_all_scores=True,
    device=device
)

premise = (
    "Wykonawca dostarczy pliki w formacie .shp zgodne z oprogramowaniem ArcGIS 10.2, "
    "wraz z mapami wydrukowanymi w formacie A4."
)

hypo = (
    "Wykonawca przekaże wyłącznie pliki .kml kompatybilne z QGIS "
    "i przygotuje dokumentację w formacie A3."
)

result = classifier({"text": premise, "text_pair": hypo})
print(result)

Batch / lower-level

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

name = "asseco-group/roberta-incoherence-classifier"
tokenizer  = AutoTokenizer.from_pretrained(name, use_fast=True)
model  = AutoModelForSequenceClassification.from_pretrained(name).eval()
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

pairs = [
    ("Zwrot kosztów w 60 dni ...", "Zwrot kosztów nastąpi w 30 dni ..."),
]
enc = tokenzier(
    [p for p, h in pairs],
    [h for p, h in pairs],
    padding=True, truncation=True, max_length=512,
    return_tensors="pt"
).to(device)

with torch.no_grad():
    logits = model(**enc).logits
probs = logits.softmax(-1).cpu()
print(probs)

---

Limitations & Recommendations

• Polish‑only checkpoint: out‑of‑language input not supported
• Complex tabular / OCR / mixed‑language content may degrade quality
• Domain‑specific fine‑tuning is recommended for production

---

Citation

@misc{asseco2025incoherence,
  title  = {Polish RoBERTa-based Incoherence/Consistency Classifier (encoder-only)},
  author = {Asseco Group},
  year   = {2025},
  url    = {https://huggingface.co/asseco-group/roberta-incoherence-classifier}
}