deberta-v3-base-fever

deberta-v3-base-fever

microsoft/deberta-v3-base fine-tuned on pietrolesci/nli_fever for 3-way fact verification, framed as sequence-pair classification (evidence, claim). Full fine-tuning, no adapters or quantization.

Labels

Input format

• Segment A: evidence (Wikipedia page title + evidence sentences joined with .; may be empty).
• Segment B: claim.
• tokenizer(evidence, claim, truncation="only_first", max_length=256) — the claim is never truncated; the tail of the evidence is truncated. Rows whose claim leaves < 32 evidence tokens use longest_first.

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo_id = "thealper2/deberta-v3-base-fever"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSequenceClassification.from_pretrained(repo_id).eval()

evidence = "Paris . Paris is the capital and most populous city of France ."
claim = "Paris is the capital of France."
inputs = tokenizer(evidence, claim, truncation="only_first", max_length=256, return_tensors="pt")
with torch.inference_mode():
    probabilities = model(**inputs).logits.softmax(dim=-1)[0]
print({model.config.id2label[i]: round(p.item(), 4) for i, p in enumerate(probabilities)})

Training data

Dataset field mapping: premise = claim, hypothesis = evidence, fever_gold_label = label. The raw integer label (entailment/neutral/contradiction) was re-mapped to the ids above; verifiable was not used (it is a function of the label).

• The Hub test split is unlabeled; the labeled, class-balanced Hub dev split is used as the held-out test set. It was not used for checkpoint selection or hyperparameter tuning.
• Validation is used for checkpoint selection (best validation macro F1).
• No deduplication was applied; duplicate/conflicting rows were kept as released.
• Train: 12,040 duplicate (claim, evidence) rows, 496 pairs with conflicting labels; all 5,194 empty-evidence rows are NOT ENOUGH INFO.
• Train/test overlap: 3 shared claims, 2 shared (claim, evidence) pairs, 98 shared non-empty evidence strings.

Training procedure

Libraries: torch 2.11.0+cu128, transformers 5.17.0, datasets 4.3.0, accelerate 1.12.0, scikit-learn 1.8.0, numpy 2.1.3

Evaluation (held-out test = Hub dev, n=19,998)

Confusion matrix (rows = true, columns = predicted):

Slices:

Limitations

• Evidence is the FEVER retrieval output shipped with NLI-FEVER; performance with other retrievers/evidence formats is not measured.
• Evidence longer than 256 tokens (with the claim) is truncated from the end.
• In training data, empty evidence always co-occurs with NOT ENOUGH INFO; the model can learn this shortcut (see the empty_evidence slice).
• Label noise: some identical (claim, evidence) pairs carry different labels in the training data.
• English Wikipedia claims only; the train (imbalanced) and test (balanced) class priors differ.