sigext-arxiv-en-allenai-1k-080t

SigExt — 1k samples, threshold 0.80 (arxiv-en, allenai)

SigExt is a lightweight salient-information extractor: a Longformer fine-tuned for binary token classification that marks the sentences of a source document that should appear (in some form) in its summary. Extracted keyphrases steer a frozen generator LLM — this is the extractor component of the SM-SIP pipeline (Semantic & Multilingual Salient Information Prompting).

• Language / corpus: English — scientific articles (ArXiv)
• Backbone: allenai/longformer-base-4096 (English Longformer, RoBERTa-initialized)
• Training samples: 1k
• Labeling threshold: 0.80 — training labels are silver labels: a source sentence is marked salient (1) if its Sentence-BERT embedding (sentence-transformers/paraphrase-multilingual-mpnet-base-v2) reaches cosine similarity > 0.80 with any summary sentence. Semantic supervision replaces the fuzzy lexical matching of prior work, which is what makes this work multilingual.
• Task: token classification, 2 classes (salient / not salient)

Why

Generic LLM summarizers hallucinate and omit salient facts. SM-SIP decouples what to say (this extractor) from how to say it (a steered generator): keyphrases extracted by SigExt are injected into the generation prompt as hard constraints, grounding the LLM in the source's most informative content. On the 10k-sample WITS configuration the full pipeline reaches BERTScore 0.6245 / ROUGE-1 0.1772 / KIR 28.93% (Keyphrase Inclusion Rate, zero-shot steering of Llama-3-8B-Instruct).

Naming legend

sigext-<corpus>-<backbone>-<samples>-<threshold> — this model is one point of a grid swept over corpus (wits-it / arxiv-en), backbone (xlmr / allenai), training-set size (1k / 2500 / 5k) and S-BERT labeling threshold (0.60 / 0.70 / 0.80). See the collection for the other configurations.

Training

• Weighted cross-entropy (salient class ×10 vs ×1) to fight the ~10% positive-class imbalance
• max_length 2048, lr 2e-5, fp16, effective batch 8 (2 × grad-accum 4), 2 epochs
• Labels projected onto the start of each sentence's token span (-100 elsewhere)

Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

name = "LookUpMark/sigext-arxiv-en-allenai-1k-080t"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForTokenClassification.from_pretrained(name)

inputs = tok(document, return_tensors="pt", truncation=True, max_length=2048)
with torch.no_grad():
    preds = torch.argmax(model(**inputs).logits, dim=2)[0]

salient = [t for t, p in zip(inputs.tokens(), preds.tolist()) if p == 1]
print(tok.decode(tok.convert_tokens_to_ids(salient)))

Citation & links

• Project write-up: SM-SIP implementation guide
• Methodological reference: Xu et al., 2024 — industry-track paper on salient-information prompting for summarization (2024.emnlp-industry.4)

Research artifact: intermediate checkpoints of a course-project grid sweep, shared for reproducibility. Each configuration was trained with identical hyperparameters; only corpus, backbone, data size and labeling threshold differ.