bioproc-benchmark

BioProCorpus: a corpus for robustness-oriented biomedical NER benchmarking and HybridNER evaluation under heterogeneous conditions

This repository accompanies our study on robustness-oriented biomedical named entity recognition (BioNER). It provides the resources needed to reproduce the benchmark construction, baseline experiments, and HybridNER results reported in the paper.

The repository includes:

1. dataset splits for four BioNER corpora,
2. training and evaluation code for five baseline models,
3. released checkpoints and prediction outputs,
4. HybridNER training code and trained models,
5. original data preprocessing and split-generation code for BioProCorpus.

1. Brief Paper Overview

BioNER is commonly evaluated on homogeneous benchmarks, which can mask issues with robustness and cross-domain generalization. We introduce BioProCorpus, a task-oriented corpus from iProX proteomics project descriptions, designed to emphasize lexical variability, boundary ambiguity, and long-range context. Under a unified protocol, we benchmark BigBird, BioBERT, DeBERTa, KeBioLM, and BINDER across four datasets and propose HybridNER, which achieves the best performance on BioProCorpus.

1.1. Repository Structure

├─ 1.Data_comparison/
│  ├─ 1.1_Datasets/
│  │  ├─ RevisedJNLPBA/
│  │  ├─ BioRED/
│  │  ├─ BioProCorpus/
│  │  └─ AnatEM/
│  │
│  ├─ 1.2_Train_code/
│  │  ├─ training_and_testing_RevisedJNLPBA.ipynb
│  │  ├─ training_and_testing_BioRED.ipynb
│  │  ├─ training_and_testing_BioProCorpus.ipynb
│  │  ├─ training_and_testing_AnatEM.ipynb
│  │  └─ BINDER-model/  # BINDER training/validation/testing on 4 datasets
│  │
│  └─ 1.3_Best_models_and_predictions/
│     ├─ RevisedJNLPBA/
│     ├─ BioRED/
│     ├─ BioProCorpus/
│     └─ AnatEM/
│
├─ 2.1HybridNER_training/ # HybridNER training code
│  ├─ train_stage_a.py
│  ├─ train_stage_b_kd_multi.py
│  ├─ train_stage_c_finetune.py
│  └─ eval_full_report.py
│
├─ 2.2HybridNER_trained/
│  ├─ model.pt
│  └─ model-PG_emphasis.pt
│
├─ 2.3Original_data_processing_code/
│  ├─ training_and_testing.ipynb
│  ├─ check_data_distribution_balance.ipynb
│  ├─ query_entity_terms_by_type.ipynb
│  ├─ entity_term_statistics.ipynb
│  ├─ data_split_processing_stratified.ipynb
│  ├─ data_split_processing_cross_file.ipynb
│  ├─ data_whitespace_cleaning.ipynb
│  └─ data_splitting.ipynb
│
├─ Entity Annotation Guidelines.pdf
└─ README.md

2. What’s Included:

2.1 Datasets

Four datasets are provided with train/development/test splits:

• Revised JNLPBA
• BioRED
• AnatEM
• BioProCorpus

These datasets are organized for unified BioNER benchmarking under the same training and evaluation protocol.

2.2 Baseline Training Code

The folder for baseline experiments contains notebook-based training, validation, and testing code for:

• BigBird
• BioBERT
• DeBERTa
• KeBioLM

It also includes the BINDER-model/ subfolder for BINDER experiments on all four datasets.

2.3 Best models and Predictions

This folder stores:

• Corresponding prediction model (e.g., best test predictions)

2.4 HybridNER Training and Released Models

2.1 HybridNER_training/ contains HybridNER training code.

2.2 Trained_models/ provides two released model checkpoints:

• model.pt
• model-PG_emphasis.pt

2.5 Original Data Processing Code

The folder 2.3Original_data_processing_code/ contains the preprocessing notebooks used to construct and inspect BioProCorpus before model training. These notebooks cover:

• whitespace and format cleaning,
• entity-term statistics,
• entity-type inspection,
• distribution balance checking,
• stratified data splitting,
• cross-file data split analysis, and
• corpus split generation.

This part is included to support reproducibility from the original annotated data to the final benchmark-ready train/development/test partitions.

3. Environment Setup

3.1 Recommended System Environment

The experiments were developed and tested in a Linux-based environment. A typical setup is:

• OS: Ubuntu 22.04
• Python: 3.10 or 3.11
• CUDA: 12.1
• PyTorch: 2.3.0
• GPU: NVIDIA RTX 3090 / RTX 4090 or equivalent

A GPU environment is strongly recommended for both baseline training and HybridNER training.

3.2 Create a Conda Environment

conda create -n bioproc python=3.10 -y
conda activate bioproc

3.3 Install Core Dependencies

Please install the required packages manually or from your own pinned requirement file. A typical environment includes:

pip install torch torchvision torchaudio
pip install transformers datasets tokenizers sentencepiece
pip install scikit-learn seqeval pandas numpy tqdm matplotlib jupyter notebook
pip install accelerate optuna

If your code uses additional packages such as torchcrf, evaluate, or domain-specific tokenizers, please also install them as needed:

pip install torchcrf evaluate

3.4 Launch Jupyter for Notebook-Based Experiments

jupyter notebook

Then open the notebook files in:

1.Data_comparison/1.2_Train_code/

or

2.3Original_data_processing_code/

depending on whether you are reproducing preprocessing or baseline experiments.

4. How to Reproduce the Workflow

4.1 Step 1: Data Preprocessing

To reproduce the processing pipeline for BioProCorpus, first go to:

2.3Original_data_processing_code/

Recommended execution order:

1. data_whitespace_cleaning.ipynb Clean redundant whitespace and formatting artifacts.
2. entity_term_statistics.ipynb Compute corpus-level and entity-level statistics.
3. query_entity_terms_by_type.ipynb Inspect entity terms by category.
4. check_data_distribution_balance.ipynb Check data distribution and category balance.
5. data_split_processing_stratified.ipynb Generate within-file stratified train/development/test splits.
6. data_split_processing_cross_file.ipynb Analyze or compare cross-file split settings if needed.
7. data_splitting.ipynb Finalize the benchmark-ready split files.

This stage reconstructs the corpus from cleaned annotated data to the final benchmark partitions used in the paper.

4.2 Step 2: Run Baseline Experiments

Navigate to:

1.Data_comparison/1.2_Train_code/

Open the notebook corresponding to the target dataset and run all cells in order.

Example notebooks include:

• training_and_testing_RevisedJNLPBA.ipynb
• training_and_testing_BioRED.ipynb
• training_and_testing_BioProCorpus.ipynb
• training_and_testing_AnatEM.ipynb

These notebooks implement training, validation, and testing under the unified protocol for BigBird, BioBERT, DeBERTa, and KeBioLM.

For BINDER, use the scripts or notebooks in:

1.Data_comparison/1.2_Train_code/BINDER-model/

4.3 Step 3: Run HybridNER

Navigate to:

2.1HybridNER_training/

Run the following scripts in order:

python train_stage_a.py
python train_stage_b_kd_multi.py
python train_stage_c_finetune.py
python eval_full_report.py

Suggested interpretation of the stages:

• train_stage_a.py: initial supervised training
• train_stage_b_kd_multi.py: multi-teacher knowledge distillation stage
• train_stage_c_finetune.py: final fine-tuning
• eval_full_report.py: evaluation and report generation

Please ensure that dataset paths, pretrained model paths, output directories, and GPU settings are configured correctly in each script before execution.

4.4 Step 4: Use Released Checkpoints

If you do not want to retrain HybridNER from scratch, you can directly use the released checkpoints in:

2.2HybridNER_trained/

Available checkpoints:

• model.pt
• model-PG_emphasis.pt

Please load the desired checkpoint in the evaluation script and update the checkpoint path accordingly.

5. Notes on Data and Splits

BioProCorpus is organized into fixed training, development, and test sets for all downstream experiments. The benchmark split was designed to support fair comparison across all baseline models and HybridNER. The preprocessing notebooks in 2.3Original_data_processing_code/ document the main steps used to prepare the final corpus partitions and inspect benchmark composition.

---

6. Availability and Implementation

The BioProCorpus dataset, HybridNER source code, preprocessing notebooks, split-generation scripts, released checkpoints, and related benchmark resources used in this study are freely available for non-commercial use at:

https://huggingface.co/Shengyu/bioproc-benchmark/

---

7. Citation

If you use this repository or BioProCorpus in your research, please cite the corresponding paper.

@article{your_paper_here,
  title   = {BioProCorpus: A corpus for robustness-oriented biomedical NER benchmarking and HybridNER evaluation under heterogeneous conditions},
  author  = {Liu, Shengyu and others},
  journal = {To be updated},
  year    = {2026}
}

---

8. Contact

For questions about the dataset, code, or reproducibility, please contact:

• Sizhu Wu: [email protected] / [email protected]
• Shengyu Liu: [email protected] / [email protected]