vera-deberta-v3-large

VERA

Variant Evaluation from Real Analytics.

VERA ranks short persuasive text by predicted click rate. One forward pass. No prompt. No generation.

Training fitted 47,168 arms across 16,129 randomized A/B tests. Every label is a measured click rate.

Usage

Pass the variants you wrote for one piece of content. Higher score wins.

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

name = "NovusEdge/vera-deberta-v3-large"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()

candidates = [
    "9 Things Nobody Tells You About Renting",
    "This Landlord Trick Costs Tenants Thousands",
    "Renting Advice",
]

enc = tok(candidates, padding=True, truncation=True, max_length=64, return_tensors="pt")
with torch.no_grad():
    scores = model(**enc).logits.squeeze(-1)

for text, score in sorted(zip(candidates, scores.tolist()), key=lambda p: -p[1]):
    print(f"{score:+.2f}  {text}")

+4.78  This Landlord Trick Costs Tenants Thousands
+3.21  9 Things Nobody Tells You About Renting
-0.80  Renting Advice

Score a set. A single score carries no meaning. The training target is an arm's deviation from its own test mean. VERA learned relative standing inside a candidate set. Give it 3 to 6 variants for one send.

Results

Holdout split. VERA never saw it during training or ablation. Every pair of arms inside a test, 18,485 of them.

Standard error on the first row is 0.003.

Most pairs in this archive carry no real difference: 29% reach significance at 5%. The first row mixes those in and is the number to quote when nothing says otherwise. Rows below it say how VERA does where the experiment resolved.

An earlier version of this card reported 0.812 here and called it "all within-test pairs". That figure came from one pair per test, the highest-CTR arm against the lowest, which is the widest gap a test offers. 70% of those pairs clear p<0.05 against 29% across the real set. The number is reproducible and it is not an all-pairs number.

Prior work on the same archive. No row is a like-for-like comparison. Each paper builds its own candidate set and two report a different metric.

Measured here, on VERA's own holdout pairs, each asked twice with the order of the two headlines swapped. A pair counts only when both orders name the same headline. Models favour the first option regardless of content, so a single-order run scores itself on a subset it chose.

Rows measured on different pair sets do not compare. The pair column says which.

Decidable means the pair's click-rate gap reaches p<0.05. Undecidable is the rest, where the recorded winner is weak evidence and every system lands in the same band.

Batching twenty pairs into one call lets a model see the other nineteen while answering, which is a different task from the isolated-pair protocol.

4,571 humans score at chance on this task. A frontier reasoning model does not, so the human result carries no claim about models. VERA's margin is seven points at roughly a thousandth of the cost per call.

Laya answers at coin-flip consistency here. Its card reports on invoice processing, security incidents, customer service and agent traces, and headline click rate falls outside all four. Read the number as domain fit.

VERA is the first pairwise accuracy reported from a fine-tuned encoder on unfiltered within-test pairs, headline text only. It is not a new state of the art, because no prior work measured this quantity.

The Pythia-12B reward model is the strongest neural result on this archive. It trains on pairs filtered to a significant CTR difference and reads the article lede and the post timestamp alongside the headline.

Metrics that transfer

Each measure below is a ratio inside one test. The base rate cancels out.

Quote the chance column with them. Tests ran a mean of 4.47 arms, so a random pick already avoids the worst one three times in four.

Picking the best variant carries the largest edge of the three, at 1.9x chance. The 90.3% sounds strongest and says least.

Median headroom exceeds pooled headroom by 34 points. VERA takes most of the available gain on the majority of tests and almost none on a minority. Quote both numbers.

One metric that does not transfer

Across 2,140 tests, VERA's top pick earns a 1.42% click rate. The test mean is 1.20%. A perfect pick earns 1.60%.

That is +18.3% relative. The figure depends on this publisher's 1.20% base rate and on the spread between their variants. Expect a different number on your corpus.

Where the accuracy comes from

Most arm pairs in this archive carry no real difference. 29% differ significantly at the 5% level. Accuracy split on that line:

Accuracy rises with the strength of the underlying effect. A model reading something it should not would score flat across these rows.

A score above 0.500 on the insignificant stratum needs no leakage explanation. Gemini 3.1 Pro posts 0.641 there having never seen these labels, so those pairs hold real gaps the experiment lacked power to prove.

The three archive splits hold disjoint test ids, so no arm appears on both sides of training. Every pair is built inside one test after the split.

Memorisation

The archive assigns tests to splits at random. Upworthy rewrote one article many times. 41% of holdout pairs therefore contain a headline with Jaccard overlap of 0.9 or higher against the training text.

Quote 0.671 when the question is what VERA knows about text it has never seen. The 4.4-point gap is what the rewrites are worth.

The overlap runs the opposite way across splits, which is the finding that killed an earlier hypothesis. Matched filters give 0.8% overlap on confirmatory's 2015 tail against 30.5% on holdout, and holdout scores lower.

Method

The loss is Bradley-Terry over arm pairs inside one test. For each pair VERA maximises logsigmoid(score_winner - score_loser). Each pair carries the log impressions of its thinner arm as weight. A comparison is only as trustworthy as the side with fewer impressions. 47,168 fitted arms yield 76,892 pairs.

The objective outweighs the data. The same recipe with MSE on shrunk logit click rate scores 0.637. Adding 28% more training data buys +0.053. Switching to Bradley-Terry buys +0.107.

Training: exploratory and confirmatory splits, 2 epochs, batch 32, lr 6e-6, one-cycle schedule, one L4, about one hour.

DeBERTa-v3-large diverges at 2e-5, the rate that suits ModernBERT. An identical run at 2e-5 held -log(0.5) for all 4,804 steps and scored 0.519.

The base model contributes nothing measurable. The same recipe on plain microsoft/deberta-v3-large scores 0.805 against VERA's 0.812 on the same pair set, a gap inside its own standard error of 0.009.

The jump from 0.519 came from the learning rate, and that holds for both bases. Nothing here supports a claim that decision-model pretraining helped.

Limits

Domain. Every number here comes from 2013–2015 viral social headlines at one publisher.

Measured, not assumed: VERA scores 0.522 on 117,118 Reddit title pairs against a 0.500 chance rate. The SNAP resubmission set has the same shape as the archive — one image, many titles, a measured outcome — and pairs are formed inside one image and one subreddit with the repost-decay curve residualised out per subreddit.

At that sample size the effect is far from chance and far too small to use. Whatever VERA learned is specific to this publisher's voice. Reddit upvotes are not a click rate and nothing randomised them, so the test cannot separate "no transfer" from "the confounds swamp it".

Transfer to email subject lines, ad copy or push notifications remains untested. No public dataset with real send outcomes exists to test it against.

Units. Bradley-Terry optimises ordering. The raw score carries no click-rate units. calibrator.json holds an isotonic fit from score to expected lift, fitted on VERA's own holdout scores over 9,558 arms, 63 knots spanning −5.32 to +5.50.

Intervals come from a bootstrap over tests, since arms in one test share an article. The fit is in-sample on holdout with no separate calibration split, so treat the curve as indicative.

Versions of this file before 2026-09-24 shipped a calibrator fitted on a ModernBERT run, whose score scale differs.

Language. English.

Ranking only. VERA ranks what you give it. Your best candidate sets the ceiling. On this archive a perfect pick reached +33%.

Citation

@misc{khimani_2026_vera,
  author       = {Khimani, Aliasgar},
  title        = {VERA: Variant Evaluation from Real Analytics},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10573},
  url          = {https://huggingface.co/NovusEdge/vera-deberta-v3-large}
}

Cite the archive as well. VERA is a model fitted to their data.

Model sources

Environmental impact

One NVIDIA L4 for about one hour on Google Cloud us-central1. Every run in the project, including failures and ablations, totals under 13 GPU-hours.

Data

The Upworthy Research Archive. Matias, J., Munger, K., Le Quere, M.A., Ebersole, C. (2021), The Upworthy Research Archive, a time series of 32,487 experiments in U.S. media, Nature Scientific Data. CC BY 4.0.

The archive asks researchers to pre-register confirmatory-split analyses. That norm targets p-hacking. VERA used confirmatory as training data and tested no hypothesis on it. Every number above comes from the untouched holdout split.

Experiments between 2013-06-25 and 2014-01-10 are excluded. The maintainers disclosed a randomization failure in that window in 2024.