RKMVNLP-scihigh-task1-flan-t5-xl-run2

RKMVNLP-scihigh-task1-flan-t5-xl-run2

LoRA adapter for research highlight generation from scientific abstracts, trained for Subtask 1 of the SciHigh shared task at FIRE 2026 by team RKMV_NLP. This is run 2 of our two submitted systems; run 1 is RKMVNLP-scihigh-task1-bart-large-run1.

Model

• Base model: google/flan-t5-xl (3B), frozen
• Adapter: LoRA r=16, alpha=32, dropout=0.05 on q, k, v, o, wi_0, wi_1, wo (35.4M trainable parameters, 1.23% of the model)
• Trained on the 10,000 abstract-highlight pairs of the MixSub-SciHigh training split
• bf16, gradient checkpointing, effective batch 12, LR 1e-4 with cosine decay and 100 warmup steps, label smoothing 0.1
• Best checkpoint selected by validation ROUGE-L, evaluated every 100 optimizer steps (step 600 of 1600)

Results

Validation, MixSub-SciHigh. Head-to-head against our run 1 on an identical 500-example subset, each model using its own swept decoding configuration:

The two systems are statistically indistinguishable: this model leads by 0.09 ROUGE-L and trails slightly on ROUGE-1 and ROUGE-2, and the gap is smaller than the bootstrap noise of the metric itself. We report it as a tie. Notably, a 7x larger model did not improve over bart-large on this task, whereas the same scale increase was worth +2.28 ROUGE-L on abstract-to-title generation (Subtask 2) — highlight generation appears limited by the task's referential ambiguity rather than by model capacity.

Decoding

Selected by an 8-configuration sweep on held-out validation data:

gen_kwargs = {"num_beams": 4,
              "length_penalty": 0.8,
              "min_length": 30,
              "no_repeat_ngram_size": 3,
              "max_new_tokens": 128}

Wider beams and higher length penalties both hurt: the sweep spanned 24.58 to 24.99 ROUGE-L, with the least aggressive configuration winning.

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel

PREFIX = "Summarize the key research highlights of the following scientific abstract: "
tok = AutoTokenizer.from_pretrained("google/flan-t5-xl")
base = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-xl", dtype="bfloat16")
model = PeftModel.from_pretrained(base, "namra07/RKMVNLP-scihigh-task1-flan-t5-xl-run2").cuda().eval()

inputs = tok(PREFIX + abstract, return_tensors="pt", truncation=True, max_length=512).to("cuda")
out = model.generate(**inputs, **gen_kwargs)
print(tok.decode(out[0], skip_special_tokens=True))

Team

RKMV_NLP, Ramakrishna Mission Vidyamandira. Other systems from this team: Subtask 1 run 1 (bart-large), Subtask 2 runs 1-2 (flan-t5-xl, flan-t5-large), Subtask 3 runs 1-2 (NLLB-200-distilled 1.3B and 600M).