structure-extraction-t2-paper-split

Claim Information Extractor for Brain Tumor Research

Model: Flan-T5-base (fine-tuned for structured extraction)

Task: Extract structured claim information from research paper sentences.

Overview

This model extracts structured information from claim sentences in brain tumor research papers. It identifies key fields: model, task, dataset, metric, value, comparison, and domain.

The train/validation/test split is paper-based: all sentences from the same DOI/paper are kept in one split to avoid leakage.

Performance

• Validation Overall Exact Match: 0.5447
• Test Overall Exact Match: 0.4793
• Validation Macro Presence F1: 0.5402
• Test Macro Presence F1: 0.5055

Field-level Performance

Dataset

• Training: 4,040 claim sentences from 1,233 papers
• Validation: 481 claim sentences from 155 papers
• Test: 459 claim sentences from 155 papers
• Total: 4,980 claim sentences from 1,543 papers
• Split method: paper/DOI-level split, not sentence-level split

Slot Fields

The model extracts the following structured fields:

• model: Name of the machine learning or deep learning model
• task: Type of task (e.g., classification, segmentation)
• dataset: Dataset or benchmark used
• metric: Evaluation metric (e.g., accuracy, Dice score)
• value: Numeric value or performance result
• comparison: Comparative statements (e.g., "outperforms ResNet")
• domain: Application domain (e.g., medical imaging)

Quick Start

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import json

model_name = "nawazishpatana/claim-extractor-brain-tumor"  # Example
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Example input
prompt = '''Extract structured claim information as JSON with keys model, task, dataset, metric, value, comparison, domain. Use null for missing values.
Title: Dilated SE-DenseNet for Brain Tumor Segmentation
Year: 2024
Claim Sentence: Our model achieved 95% Dice score on BraTS dataset.'''

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
prediction = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(prediction)
# Output: {"model": "Dilated SE-DenseNet", "dataset": "BraTS", "metric": "Dice", "value": "0.95", ...}

Training Details

• Base Model: google/flan-t5-base
• Optimizer: AdamW
• Learning Rate: 2e-05
• Per-device Batch Size: 2
• Gradient Accumulation Steps: 1
• Effective Batch Size: 2
• Epochs: 5 (with early stopping)
• Max Input Length: 256
• Max Output Length: 128

Evaluation Metrics

• Exact Match (EM): Percentage of predictions that exactly match ground truth
• Non-Null Accuracy: Accuracy on fields where ground truth is not null
• Null Accuracy: Accuracy in predicting null when ground truth is null

Post-Processing

The model includes automatic post-processing:

1. Fuzzy matching for model names
2. Dataset name normalization
3. Metric standardization
4. Numeric value parsing

Citation

If you use this model, please cite:

@misc{claim-extractor-brain-tumor,
  title={Claim Information Extractor for Brain Tumor Research},
  author={Your Name},
  year={2025},
  howpublished={\url{https://huggingface.co/username/claim-extractor-brain-tumor}}
}