openmed-autoresearch

OpenMed × Autoresearch: Cross-Dataset Transfer Discovery

Autonomous discovery of optimal training curricula for biomedical NER, using Karpathy's autoresearch loop on OpenMed datasets.

Results (~120 experiments on RTX 4090)

Disease NER (NCBI Disease target — 93 experiments)

Chemical NER (BC5CDR-Chem target — 26 experiments)

Transfer Affinity Matrix (ΔF1 from 50/50 pretrain→finetune)

Reading the matrix: each cell shows the F1 change when pretraining on the source (row) before fine-tuning on the target (column). Bold = significant positive transfer.

Key Discovery: Asymmetric Transfer

Transfer is 3.8x stronger from chemicals to diseases than the reverse. This asymmetry likely arises because (1) BC5CDR contains both chemical AND disease annotations, and (2) chemical entities are more lexically distinctive while disease entities benefit more from contextual pretraining.

Other key findings:

1. Sequential 3-stage curriculum beats single-source and mixing approaches (both targets)
2. Cosine scheduler is more reliable than constant_with_warmup (higher mean, lower variance)
3. Batch size 64 is critical in time-budgeted training (3x GPU utilization)
4. JNLPBA (proteins) helps both targets — proteins interact with both chemicals and diseases
5. Negative transfer from species (Linnaeus) and gene-only (BC2GM) datasets
6. Default BERT hyperparameters are near-optimal — 50+ tuning experiments found no improvement

See FINDINGS.md, FINDINGS_CHEM.md, and results.tsv for full analysis.

What this does

An AI agent (Claude Code) runs ~100 experiments overnight on your GPU, systematically exploring which biomedical NER datasets help each other through transfer learning. The target: maximize F1 on NCBI disease NER by finding the best cross-dataset pretraining curriculum.

Available datasets

Setup

# 1. Clone and enter
git clone <this-repo>
cd openmed-autoresearch

# 2. Install dependencies
pip install -r requirements.txt

# 3. Prepare data (downloads & tokenizes all datasets)
python prepare.py

# 4. Verify
ls ~/.cache/openmed-autoresearch/
# Should see: bc5cdr_chem/ ncbi_disease/ bc2gm/ jnlpba/ linnaeus/ meta.json

# 5. Test baseline
python train.py
# Should print val_f1 and peak_vram_mb after ~5 minutes

# 6. Run autoresearch with Claude Code
claude --dangerously-skip-permissions
# Then tell it: "Read program.md and start the autoresearch loop"

Changing the target dataset

To evaluate on a different entity type (e.g., chemicals instead of diseases), edit TARGET_EVAL_DATASET in train.py. The curriculum exploration then discovers what helps that entity type.

Analyzing results

After a run, use analyze.py to generate the transfer affinity heatmap:

python analyze.py results.tsv

Output

• results.tsv: Full experiment log with F1 scores
• Git history on the autoresearch/<tag> branch: only the improvements
• Transfer affinity insights from the experiment data

Credits

• OpenMed by Maziyar Panahi — datasets and models
• autoresearch by Andrej Karpathy — the autonomous experiment loop pattern
• Base model: ModernBERT-base