lfm25-titlegen-17-languages

lfm25-titlegen-17-languages

Abstract

This model release is evaluated as the 17-language training variant in a reproducible, 17-language conversation-title benchmark. The study contains 3,400 inputs per model and distinguishes 2,550 evaluation inputs from 850 development inputs. Each generated title receives one automated Luna assessment on seven explicit ordinal criteria where applicable. This card reports observed scores and denominators; it does not establish calibrated acceptance rates, causal training effects, or production readiness.

Dataset and detailed results · Interactive benchmark · Pinned aggregate JSON

1. Model and task

The intended task is to generate a concise, informative title from a user conversation in its target language. Titles should preserve central intent, essential entities and literals, and meaningful uncertainty. Training details and the original model documentation are retained verbatim in Appendix A. Variant names identify checkpoints; this comparison does not isolate the causal effect of a training method.

• Checkpoint evaluated: b5e0088662920027ae73bb5c91fab347677bfaa7
• Repository: ManhHoDinh/lfm25-titlegen-17-languages
• Weight manifest fingerprint: 848a486d9f5f6ac7d468d62f420c76cc06b9884d8140beff78a7656747048788
• This documentation update changes no model weights.

2. Experimental design

The mixed-v2 set contains 200 inputs in each of 17 languages. The primary evaluation partition has 150 inputs per language; 50 development inputs per language are reported separately. All four local model variants use the same input set. Generation uses greedy decoding, float32, and at most 32 new tokens (Torch 2.14.0; Transformers 5.13.1). Local generation timing is not a deployment load test.

Original benchmark prompts contain the user message only, passed through each local tokenizer's chat template with an assistant generation prompt. The separately generated Luna reference adds an explicit title-task system instruction; it is not a matched-prompt baseline.

The pointwise judge is cx/gpt-5.6-luna, temperature 0, low reasoning effort, with structured JSON output. The rubric covers relevance, intent coverage, faithfulness, entity fidelity, concision, fluency and safety. Each applicable criterion is scored from 0 to 4. Criterion applicability is input-specific: unassessed criteria are excluded rather than assigned zero. Pending agent-authored expectations guide evaluation; they are not native-speaker gold labels. Requests, raw replies and parsed checkpoints are validated before aggregate publication.

3. Primary evaluation results

Cells show mean / 4 (applicable n) over the 2,550 evaluation inputs. Means summarize ordinal scores; they are not percentages or pass probabilities.

For this checkpoint, the lowest observed mean criterion is intent_coverage (1.249/4; n=2,505). This identifies cases to inspect; it does not by itself diagnose a data or training cause.

Results by language

Development and combined partitions

4. Detailed diagnostics and Luna reference

The linked dataset and interactive report expose per-input outputs, original judge reasons, and supplemental feedback. The supplemental protocol generates a Luna reference title and assesses five anonymously ordered candidates. It uses a different prompting protocol and output budget. Luna also judges its own generated reference, so self-judge bias is a material limitation. The verified release at dataset revision 779c3bedc27a2eaa6a173a69eebb4deb1215bf9d contains 3,400 Luna reference titles and 3,400 complete feedback rows (17,000 candidate feedback records). Feedback provenance is 3,227 original responses, 172 candidate-remediation-v2 responses and one issue-schema-remediation-v3 response. Paired comparisons are separated by feedback protocol; remediation cases are a selected failure subset. Supplemental scores are kept separate from the pointwise results in Section 3. Suggested titles are proposals, not gold labels.

5. Limitations and production readiness

Production readiness: NOT_ESTABLISHED. The study has 200 inputs per language versus the configured 500-item requirement, one automated judge versus three, and no verified native audit versus the required 100 items per language. Deterministic compliance rates, calibrated semantic acceptance, native agreement, real deployment load tests, fairness acceptance gaps and regression gates remain unestablished. Ordinal rubric means cannot be directly substituted for similarly named production-policy thresholds.

The published evaluation inputs are now exposed. Tuning on these inputs or their feedback invalidates future held-out claims for this set; use a newly collected, independently audited evaluation set for subsequent optimization claims.

Additional limitations include pending annotation review, unmeasured independent judge agreement, potential language-dependent judge bias, and incomplete evidence of exclusion from every training corpus. A reached-limit flag is a proxy, not proof of truncation. Low intent coverage motivates a human label and intent-diversity audit; low entity fidelity motivates entity-retention examples and a controlled output-cap experiment. These are hypotheses requiring held-out validation, not demonstrated optimization gains.

6. Reproducibility and release artifacts

The aggregate JSON preserves score distributions and applicable denominators. Generation and judge checkpoints bind inputs, output manifests, prompts and response evidence to immutable identities. The dataset repository distributes versioned diagnostic snapshots and source artifacts; the Space provides an inspection interface. Use the checkpoint revision above and the dataset snapshot revision when citing or comparing results. Inspect the license of each distributed artifact before reuse; the original model license metadata remains unchanged.

• Aggregate methodology

• Dataset files and source artifacts

• Interactive report

• Evaluate your own model: portable quickstart

• End-to-end local inference, benchmark and playground

• Hướng dẫn sử dụng và đo benchmark (Tiếng Việt)

• Release scope, status guide and optimization workflow

Inference quickstart

The following matches the benchmark's greedy, user-only title-task interface. It is an illustrative invocation; the recorded benchmark runtime remains authoritative.

pip install torch==2.14.0 transformers==5.13.1

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = 'ManhHoDinh/lfm25-titlegen-17-languages'
revision = 'b5e0088662920027ae73bb5c91fab347677bfaa7'
tokenizer = AutoTokenizer.from_pretrained(repo_id, revision=revision, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    repo_id, revision=revision, torch_dtype=torch.float32,
    trust_remote_code=False, use_safetensors=True,
).eval()
messages = [{'role': 'user', 'content': 'How can I plan a three-day trip to Hanoi with children?'}]
inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_tensors='pt', return_dict=True,
)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False, num_beams=1)
title_tokens = output[0, inputs['input_ids'].shape[-1]:]
print(tokenizer.decode(title_tokens, skip_special_tokens=True).strip())

Citation

This is a versioned software and benchmark release, not a claim of peer-reviewed publication.

@misc{titlegen_mixed_v2_luna_2026,
  title = {TitleGen Mixed-v2: Multilingual Title Generation with Luna Evaluation},
  author = {ManhHoDinh},
  year = {2026},
  howpublished = {Hugging Face model and benchmark release},
  url = {https://huggingface.co/datasets/ManhHoDinh/titlegen-benchmark-luna},
  note = {Model multilingual, evaluated revision b5e0088662920027ae73bb5c91fab347677bfaa7; cite the dataset snapshot revision used}
}

Appendix A. Historical model card

The original card is preserved below as historical documentation; the current benchmark qualifications and scope above govern interpretation of this new evaluation.

<details><summary>Original documentation</summary> <!-- titlegen-research-history:start -->

LFM2.5 TitleGen 17-language SFT

This model is the 17-language supervised fine-tuning stage of the LFM2.5 TitleGen experiment. Model repository: ManhHoDinh/lfm25-titlegen-17-languages. Benchmark publication timestamp: 2026-08-16T00:00:00+07:00.

Preliminary benchmark

The current model passed 179 of 300 evaluated English and Vietnamese examples (59.7%).

Language coverage

0 means no evaluated examples when the status is NOT_EVALUATED; it is not a measured zero score.

Methodology

This preliminary benchmark evaluates aggregate English and Vietnamese results with deterministic decoding (do_sample: false, max_new_tokens: 32). Automated rubric identifiers: 3-8_tu, khong_cham_cuoi, mot_dong, dung_ngon_ngu, khong_chep. The remaining contracted languages are shown explicitly as not evaluated.

Limitations

• Only English and Vietnamese have evaluated examples.
• Language correctness uses an automated heuristic.
• No native review or blind preference evidence is included.
• These aggregate automated results do not establish production readiness, causal improvement, or statistical significance.

Machine-readable results

See benchmark-report.json for the validated aggregate report.

<!-- titlegen-mixed-v2-luna:start -->

Mixed-v2 Luna benchmark: multilingual

3,400 / 3,400 judgments for this model. Single Luna automated assessment on a 0..4 ordinal scale. Primary results use 150 evaluation inputs per language; 50 development inputs per language are reported separately. Agent-authored annotations are not native-speaker gold. Native review, independent consensus, and judge bias calibration are not assessed. This release covers only the named model. Missing applicability is excluded from criterion counts. Provider billing and upstream retry totals are unknown.

• Report (HTML)
• Aggregate JSON
• Methodology and per-language scores

<!-- titlegen-mixed-v2-luna:end --> <!-- titlegen-research-history:end --> </details>