BART-base-KP

Model Card for BART-base-KP

This is a BART-base model for multi-domain keyphrase generation, fine-tuned jointly on computer science, biomedical, and news corpora. It serves as a multi-domain seq2seq baseline described in:

Saber Zahhar, Nédra Mellouli, Christophe Rodrigues, Nicolas Travers. Multi-Domain Keyphrase Prediction via Retrieval-Augmented Ranking: A Resource-Efficient Alternative to Seq2Seq Generation. DKE 2026.

Model Details

Model Description

We fine-tuned facebook/bart-base jointly on the training splits of three multi-domain keyphrase datasets (kp20k, kpbiomed, kptimes) in a One2Seq setting: given a source document as input (title and abstract concatenated), the model generates all keyphrases as a single <KP>-separated sequence.

• Developed by: Saber Zahhar (Devoteam / De Vinci Research Center)
• Model type: Conditional language model (seq2seq)
• Language(s): English
• License: Apache 2.0
• Parent model: facebook/bart-base

Usage

from transformers import BartForConditionalGeneration, AutoTokenizer

model = BartForConditionalGeneration.from_pretrained('sabsab129/BART-base-KP')
tokenizer = AutoTokenizer.from_pretrained('sabsab129/BART-base-KP')

# inputs are formatted as "title<s>abstract"
input_text = "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension<s>We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and other recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa on GLUE and SQuAD, and achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 3.5 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also replicate other pretraining schemes within the BART framework, to understand their effect on end-task performance."
input_ids = tokenizer(input_text, return_tensors="pt")

outputs = model.generate(**input_ids, num_beams=7)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
# </s><s>denoising<KP>sequence-to-sequence pre-training<KP>natural language generation<KP>pret</s>

Keyphrases in the output are delimited by <KP> tokens.

Training Details

Training Data

The model was fine-tuned on the concatenated training splits of three multi-domain keyphrase datasets:

Input is the concatenation of a document's title and abstract (formatted as title<s>abstract). The encoder caps input at 512 tokens (a threshold exceeded by a subset of kptimes documents).

Training Procedure

Fine-tuning used the One2Seq paradigm: all gold keyphrases for a document are concatenated into a single output sequence separated by the <KP> special token, ordered present → absent, which empirically optimises both extractive and abstractive keyphrase recovery.

• Optimiser: AdamW, learning rate 5×10⁻⁵
• Batch size: 168
• Epochs: 10
• Decoding: beam search with num_beams=7
• Hardware: 1× NVIDIA Tesla V100 32 GB
• Wall-clock training time: ~16 hours
• Inference time (60k test documents): ~4 hours 40 minutes

Citation

If you use this model, please cite the paper:

@inproceedings{zahhar2026searchkeys,
  title     = {Multi-Domain Keyphrase Prediction via Retrieval-Augmented Ranking: A Resource-Efficient Alternative to Seq2Seq Generation},
  author    = {Zahhar, Saber and Mellouli, N{\'e}dra and Rodrigues, Christophe and Travers, Nicolas},
  booktitle = {DKE 2026},
  year      = {2026}
}