lfm2.5-350M-datause-multitask-tiered

lfm2.5-350M-datause-multitask-tiered

LoRA SFT of LiquidAI/LFM2.5-350M for the tiered data-use workflow: provenance attributes (producer/year/geography/acronym + data_type/specificity/bibtex) and usage/impact classification (usage_action/impact_label/usage_summary). Trained exclusively on T1∪T2-anchored rows of rafmacalaba/data-use-sft-tiered — see the dataset card for the tier policy.

Training

• base model: LiquidAI/LFM2.5-350M
• dataset: rafmacalaba/data-use-sft-tiered (provenance + usage_impact configs)
• epochs: 5 | lr: 0.0002
• LoRA: r=16 alpha=32 dropout=0.05
• completion-only masking (loss on the assistant JSON turn)

Holdout

n=8804. Full per-row gold/pred pairs are in holdout_predictions.jsonl on this repo.

Metrics recomputed offline from holdout_predictions.jsonl with string-aware JSON extraction (training/score_holdout_predictions.py). An earlier in-trainer extraction bug suppressed predictions containing bibtex values; the numbers below are the corrected ones.

Provenance — verbatim attrs (exact match)

bibtex (gold rows with producer+year):

• exact: 195 (36.9%) | semantic (author+title+year, key ignored): 310 = 58.6% (key-only diffs: 115)

• format: 98.9% of 364 emitted are canonical @dataset entries

• omitted: 165 (all missing producer/year inputs) | hallucinated on gold-none rows: 175

• data_type macro-F1: 0.4905 (n=1198)

• specificity macro-F1: 0.6307 (n=3325)

Usage/impact

• usage_action macro-F1: 0.5441 (n=5479)
• impact_label macro-F1: 0.6519 (n=5479)

Top usage_action confusions: analyze -> inform (241), inform -> analyze (239), curate -> analyze (143), source -> analyze (141), analyze -> source (127) Top impact_label confusions: policy -> evidence-base (320), evidence-base -> none (263), none -> evidence-base (262), evidence-base -> policy (214), policy -> none (72)

usage_summary: mean_sim=0.5587 grounded_rate=0.6623