xdomain-ser-extractor

xdomain-ser-extractor

A LoRA adapter for Llama-3.2-3B-Instruct that extracts structured meaning representations (slot-value pairs) from generated text, for measuring semantic fidelity in meaning-to-text NLG. Part of the cross-domain slot-error-rate (SER) evaluation framework described in our GEM 2026 paper.

What it does

Given a natural-language realization and a domain hint map, the adapter extracts the slot-value pairs expressed in the text. Comparing the extracted MR against a gold MR yields per-example SER, slot F1, and substitution/deletion/insertion counts. The framework reaches 86.8% SER agreement across 23 domains when combined with over-generate-and-rank and NLI routing, without per-domain rules.

Usage

This is a PEFT adapter. The base model must be obtained separately from Meta and is governed by the Llama 3.2 Community License (see License below).

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
model = PeftModel.from_pretrained(base, "DavanHarrison/xdomain-ser-extractor")
tokenizer = AutoTokenizer.from_pretrained("DavanHarrison/xdomain-ser-extractor")

For the full pipeline (extraction, over-generate-and-rank, NLI routing, CLI), see the code repository: https://github.com/Vrindiesel/xdomain-ser

Training

• Base model: meta-llama/Llama-3.2-3B-Instruct
• Method: 4-bit QLoRA supervised fine-tuning
• LoRA config: r=4, alpha=16, dropout=0.05, rank-stabilized (rsLoRA), EVA initialization
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Training data: multi_ser_v9, 20 topics x 200 examples (~4,000 examples)
• Trainable parameters: ~0.34% of the base model

Intended use

Automatic slot-level semantic fidelity evaluation of data-to-text and task-oriented dialogue generation. Research use. The adapter is English-only and was trained on task-oriented domains; behavior on other languages or open-domain text is not characterized.

Limitations

Extraction accuracy degrades on meaning representations with many slots (7-8+). The adapter has not been evaluated by human annotators on the extraction task itself; reported numbers use a synthetic modified-MR evaluation protocol and a cross-validation set of 1,000 gold-annotated examples. English-only.

License

The adapter weights and accompanying code in this repository are released under the Apache License 2.0, covering our contribution only.

Use of this adapter requires the base model, meta-llama/Llama-3.2-3B-Instruct, which is distributed under the Llama 3.2 Community License Agreement. You must obtain the base model from Meta and accept that license to use this adapter. This release does not redistribute any Meta weights and does not grant any rights to the base model.

Citation

@inproceedings{harrison2026xdomainser,
  title     = {Cross-Domain Semantic Fidelity Evaluation for Meaning-to-Text NLG},
  author    = {Harrison, Davan and Walker, Marilyn},
  booktitle = {Proceedings of the GEM Workshop at ACL 2026},
  year      = {2026}
}