qa-completeness-regressor

QA Completeness Regressor

Give it a question and a long answer. It tells you how complete the answer is.

A complete answer covers everything the question asks for. An incomplete one leaves things out. This model reads the question and the answer together and returns one number for that.

You do not need a reference answer. That is the point of it.

It is bert-base-uncased with a small regression head, trained on synthetic data. The paper is Beyond accuracy: completeness and relevance metrics for evaluating the quality of long answers, Language Resources and Evaluation 60(3), article 58 (2026). It is open access: doi.org/10.1007/s10579-026-09936-6.

---

Quick start

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

name = "egcortes/qa-completeness-regressor"
model = AutoModelForSequenceClassification.from_pretrained(name, trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(name)

question = "How does RAM memory work?"
answer = "RAM stores data your computer is using right now. It is fast but it forgets everything when you turn the power off."

text = f"Question: {question}\n\nfAnswer: {answer}\n\nHow complete is this answer?"
inputs = tokenizer(text, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    score = model(**inputs).logits.item()

print(round(score, 3))   # 0 = nothing is covered, 1 = everything is covered

You can also use a pipeline:

from transformers import pipeline

pipe = pipeline("text-classification", model="egcortes/qa-completeness-regressor",
                trust_remote_code=True, function_to_apply="none")
pipe(text)   # [{'label': 'completeness', 'score': 0.26}]

Two things to know:

• trust_remote_code=True is needed. The head is a plain linear layer on the CLS token, which is not one of the standard transformers heads, so the model class ships with the model.
• The input format matters. Build the text exactly as shown, including the odd f before Answer:. That typo was in the training code, so the model expects it. Without it the scores drift.
• The output is a score, not a class. The model is tagged as text classification because Hugging Face has no tag for regression, but it has a single output and no labels. In a pipeline the number comes back in a field called score, which normally means a probability. It is not one here, so always pass function_to_apply="none".

---

Results

Tested on the 212 human-scored answers in the companion dataset, against the average of four annotators:

That beats every reference-based metric in the paper except ROUGE, and unlike ROUGE it does not need a reference answer.

It also behaves sensibly when you take an answer apart. Cutting an expert answer down to fewer and fewer sentences lowers the score every time:

The score went down at every step, for all 8 answers tested.

---

Limitations

It does not check facts. An answer can be complete and still be wrong.

It was trained on one domain. The training data is "How to..." questions, mostly about computer science, from ELI5 and WebGPT. It may behave differently elsewhere.

The output is not calibrated. Treat it as a ranking signal, not an absolute percentage. Across the 212 answers in the dataset it ranged from 0.08 to 0.76. Multiply by 100 to compare it with the dataset scores.

---

Training

Long answers were taken from ELI5 and WebGPT, then broken on purpose: sentences were removed to make them less complete, and unrelated sentences were added to make them less relevant. Each damaged answer got a score from how much of the original survived. That gave about 100,000 training examples.

The model then learned to predict that score from the question and the answer.

• Base model: bert-base-uncased
• Head: one linear layer, 768 to 1, on the CLS token
• Loss: MSE, 5 epochs, batch size 8, Adam, learning rate 1e-5
• Max length: 512 tokens

Because the training answers were damaged in a mechanical way, the model is good at spotting missing content and less tested against the messier ways real answers go wrong.

---

Links

• Dataset: egcortes/qa-completeness-relevance, 106 questions and 212 answers scored by four annotators
• Code and data files: github.com/eduardogc8/QA-CompletenessRelevance
• Paper: doi.org/10.1007/s10579-026-09936-6

---

Citation

Please cite the paper:

@article{cortes2026beyond,
  title   = {Beyond accuracy: completeness and relevance metrics for
             evaluating the quality of long answers},
  author  = {Cortes, Eduardo G. and Vieira, Renata and Barone, Dante A. C.},
  journal = {Language Resources and Evaluation},
  volume  = {60},
  number  = {3},
  pages   = {58},
  year    = {2026},
  doi     = {10.1007/s10579-026-09936-6}
}

---

Contact

[email protected]