Multi_EvalSumViet2

Naming note. EViSE is the current name of the evaluator originally released as MultiEvalSumViet2. The same evaluator was used under the name MultiEvalSumViet2 in our previously published IEEE Access study. The repository identifier is retained to preserve DOI and citation continuity.

EViSE: Efficient Evaluation for Vietnamese Summarization

EViSE is a compact Vietnamese learned evaluator for scoring a candidate summary against its source document on three related and complementary criteria:

• Faithfulness (F): whether information stated in the summary is supported by the source; unsupported entities, relations, quantities, attributions, or causal claims reduce the score.
• Coherence (C): whether the summary is logically organized, self-contained, and linguistically well formed.
• Relevance (R): whether the summary preserves salient source information while avoiding tangential or low-value details.

A summary can therefore be factually supported but still receive a lower Relevance score if it omits important source content. The criteria are modeled separately while sharing the same document-summary representation.

Naming continuity. EViSE is the current name of the evaluator previously released as MultiEvalSumViet2. The underlying released evaluator is the same model used under the name MultiEvalSumViet2 in the IEEE Access study Reinforcement Learning With Verifier Guidance and Penalty Shaping for Vietnamese Summarization Using Small Language Models (DOI: 10.1109/ACCESS.2026.3701952). The EViSE name reflects the current emphasis on efficient multi-criteria evaluation and does not denote a different checkpoint.

Main characteristics

EViSE is designed around three forms of efficiency:

1. Multi-criteria evaluation efficiency: Faithfulness, Coherence, and Relevance are predicted jointly from one shared source-summary encoding.
2. Computational efficiency: all three criterion scores are produced in one local, non-autoregressive forward pass without requiring a reference summary at inference time.
3. Downstream utility: the frozen evaluator can be reused as a criterion-aware reward component in repeated optimization workflows such as reinforcement learning for summarization.

Output

For each (document, summary) pair, EViSE returns:

• pred_faith in [0, 1]
• pred_coherence in [0, 1]
• pred_relevance in [0, 1]

Optional mapping to the original 1--5 scale:

score_1to5 = 4 * score_0to1 + 1

When one scalar is required for secondary analysis, filtering, preference construction, or reward computation, the associated study uses:

Overall = 0.5 * F + 0.3 * R + 0.2 * C

Criterion-wise outputs remain the primary model outputs.

Architecture

EViSE uses a ViDeBERTa document-summary cross-encoder. The source document and candidate summary are jointly serialized so that source and summary tokens interact contextually before prediction.

document + summary
      ↓
ViDeBERTa cross-encoder
      ↓
masked mean pooling
      ↓
Linear(256) + GELU + Dropout(0.1)
      ↓
three independent criterion heads
      ↓
F, C, R

A single shared encoding is reused by all three prediction heads. This avoids three separate encoder evaluations for the three criteria.

Training objective

The evaluator combines criterion-wise regression with within-document pairwise ranking:

L_hybrid = L_reg + λ L_rank

• L_reg calibrates predictions to the absolute human-reviewed score scale.
• L_rank preserves criterion-specific ordering among alternative summaries of the same source document.
• Ranking is group-balanced by source document.
• Ranking margin: m = 0.05.
• Ranking-loss weight: λ = 0.35.

The two constituent losses are established objectives. The methodological contribution is their criterion-wise, document-conditioned, and group-balanced integration. Regression supplies absolute calibration, whereas ranking exploits relative quality information already available among summaries of the same source. Ranking supervision operates on predictions obtained from the shared encoder and therefore does not require an additional encoder pass for each comparison pair.

Data and supervision

EViSE is developed from a Vietnamese News evaluation resource containing:

• 13,476 VnExpress source articles from 2022--2024;
• 6 candidate summaries per source;
• 80,856 (document, summary) evaluations;
• 72,768 evaluations from 12,128 source documents in the training-and-validation pool;
• 8,088 evaluations from 1,348 source documents in the leakage-safe held-out News test split;
• candidate systems: GPT-4o, Gemini, LLaMA-3.2 1B, LLaMA-3.2 3B, LLaMA-3.1 8B, and a ViT5-large summarizer fine-tuned on a filtered VNDS subset.

The evaluation prompt was calibrated on 100 candidate summaries using human ratings as reference. Gemini then provided initial 1--5 F/C/R ratings, which were reviewed across the full corpus by 12 trained volunteer annotators; approximately 10% of initial labels were revised. A separate 450-pair blind audit assesses rubric reproducibility without exposing the original labels to the re-annotators.

The accompanying dataset is available at:

https://huggingface.co/datasets/phuongntc/EViSE-Dataset

Evaluation

In-domain News

The held-out News test split contains 1,348 source documents / 8,088 document-summary pairs, separated by doc_id. EViSE improves all three criteria over a shared-head regression baseline and shows stronger agreement with human-reviewed aggregate judgments than ROUGE-Lsum and PhoBERT-large BERTScore-F1 in the associated evaluation.

Out-of-domain IT-textbook benchmark

The OOD benchmark contains 900 document-summary pairs = 150 IT source passages × 6 heterogeneous summarization systems, covering 13 IT subject areas.

Against human-reviewed F/C/R ratings, the weighted Overall score reaches:

• Pearson r = 0.829
• Spearman ρ = 0.691
• MAE = 0.107

This benchmark provides evidence of transfer from news to technical educational text; it should not be interpreted as universal cross-domain robustness.

Efficiency profile

A matched profiling experiment on 1,344 Vietnamese document-summary pairs using a single NVIDIA Tesla T4 reports:

The full loaded EViSE evaluator contains approximately 183.952M parameters in this profiling setup. EViSE is not intended to be faster than lightweight lexical metrics such as ROUGE; its efficiency claim concerns multi-criteria semantic evaluation, where three criterion scores are produced jointly without reference summaries.

For broader context, a same-count UniEval profile on 1,344 CNN/DailyMail examples using the same Tesla T4 required approximately 2,954 seconds and 6.5 GB peak GPU memory for three dimensions. Because the UniEval profile uses a different language and dataset, it is contextual computational evidence rather than a quality-matched head-to-head comparison.

Language portability

The released checkpoint is Vietnamese-specific, but the training formulation itself is not inherently tied to Vietnamese. The same framework can be instantiated for another language by replacing the encoder with a suitable language-specific or multilingual backbone and constructing grouped criterion-labeled data under the same F/C/R protocol. Cross-language transfer should therefore be understood as methodological portability; it has not yet been established by a matched non-Vietnamese training experiment.

Recommended usage

This repository contains custom criterion heads. Do not use it as a text-generation or summarization pipeline. Use the included modeling_summary_evaluator.py loader so that the backbone, trunk, and three criterion heads are restored correctly.

import os
import importlib.util
import torch
from huggingface_hub import snapshot_download
from transformers import DataCollatorWithPadding

REPO_ID = "phuongntc/EViSE"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

repo_dir = snapshot_download(repo_id=REPO_ID, repo_type="model")

loader_path = os.path.join(repo_dir, "modeling_summary_evaluator.py")
spec = importlib.util.spec_from_file_location("evise", loader_path)
evise = importlib.util.module_from_spec(spec)
spec.loader.exec_module(evise)

model, tokenizer, _ = evise.load_for_inference(repo_dir, device=DEVICE)
model.eval()

docs = ["Văn bản nguồn ..."]
summaries = ["Bản tóm tắt ..."]

enc = evise.encode_pair(tokenizer, docs, summaries)
collator = DataCollatorWithPadding(tokenizer=tokenizer, padding=True)
features = [{k: enc[k][0] for k in enc.keys()}]
batch = collator(features)

with torch.inference_mode():
    scores = model(
        batch["input_ids"].to(DEVICE),
        batch["attention_mask"].to(DEVICE)
    )[0].detach().cpu().tolist()

print({
    "faithfulness": scores[0],
    "coherence": scores[1],
    "relevance": scores[2],
})

Intended use

Appropriate uses include:

• Vietnamese summarization evaluation;
• large-scale filtering and dataset curation;
• within-source preference construction;
• repeated local multi-criteria scoring;
• a frozen scoring/reward component in downstream optimization workflows.

The previously published verifier-guided PPO/GRPO study used the same evaluator, then named MultiEvalSumViet2, as a frozen reward component. Those policy-optimization results provide downstream application evidence rather than additional training results for this repository.

Limitations

• Training data are centered on Vietnamese news.
• The IT benchmark provides one technical-educational OOD setting but does not establish universal genre robustness.
• Language portability is a property of the framework and has not yet been validated through matched training in another language.
• Runtime and memory depend on hardware, implementation, input length, and batching.
• As with any learned evaluator used in reward optimization or filtering, EViSE should not be treated as the sole signal when reward hacking or systematic evaluator bias is a concern.

Reproducibility

For reproducible experiments, pin a specific repository revision:

repo_dir = snapshot_download(
    repo_id="phuongntc/EViSE",
    repo_type="model",
    revision="<COMMIT_HASH_OR_TAG>",
)

License

Model and code files in this repository are released under the Apache License 2.0.

Citation and naming history

Model DOI: 10.57967/hf/7956

EViSE is the current name of the model object originally released as MultiEvalSumViet2. When citing the Hugging Face model DOI, please note that the DOI metadata may retain the legacy repository title until the repository/DOI metadata is updated by Hugging Face.

@misc{evise_model,
  title        = {EViSE: Efficient Evaluation for Vietnamese Summarization},
  author       = {Thi Thu Phuong Tran},
  year         = {2026},
  howpublished = {Hugging Face Hub},
  url          = {https://huggingface.co/phuongntc/EViSE},
  doi          = {10.57967/hf/7956},
  note         = {Previously released as MultiEvalSumViet2}
}

Contact

Maintainer: Thi Thu Phuong Tran
Affiliations: Hanoi Metropolitan University; VNU University of Engineering and Technology, Vietnam National University
Email: [email protected]