bfm-rollout-finetuned-large

BFM Rollout Finetuned Large (BioCube, VeRA)

Rollout-finetuned version of the BioAnalyst Foundation Model (BFM) Large (702M params) on BioCube monthly climate + biodiversity data using the paper's BFMRollout pipeline (arXiv:2507.09080v2, Section 3.2).

Model Details

• Base model: BFM Large (embed_dim=512, depth=10, Swin 3D backbone, 702M params)
• PEFT: VeRA (rank=256, d_initial=0.1) — only 63.5K trainable params (0.009%)
• Finetuning data: BioCube (28 species, 0.125 deg, monthly, Europe)
• Rollout: 12-step push-forward with TBPTT-2 (gradients on last step only)
• Training: 50 epochs, AdamW (lr=5e-5, wd=5e-7), bf16-mixed
• Best val_loss: 0.627335 (epoch 36)
• Architecture: Full BFM pipeline (Encoder → Swin Backbone → Decoder), encoder/decoder frozen

Evaluation on BioCube 28 Species (48 test pairs)

Key Findings

• VeRA with only 63.5K trainable parameters (0.009% of 702M) is insufficient for significant improvement over the pretrained baseline on BioCube 28 species
• The paper's results (F1=0.9964, S=0.31) likely used a much larger trainable parameter budget or different training setup (e.g., full finetune or higher-rank PEFT)
• Species predictions are mostly negative (mean=-0.019, positive_frac=3.3%), indicating the model learned to suppress species signals rather than enhance them

Usage

import torch
from safetensors.torch import load_file
from bfm_model.bfm.model import BFMRollout

# 1. Build BFMRollout with VeRA
model = BFMRollout(
    embed_dim=512, depth=10, patch_size=8,
    swin_backbone_size="large", perceiver_latents=16100,
    rollout_steps=1, finetune_mode="peft",
    use_vera=True, peft_r=256, d_initial=0.1,
    peft_mode="single", peft_steps=1,
    # ... (see finetune_rollout_large.py for full config)
)

# 2. Load pretrained base weights
state = load_file("bfm-pretrain-large.safetensors", device="cpu")
model.load_state_dict(state, strict=False)

# 3. Load finetuned VeRA weights from Lightning checkpoint
ckpt = torch.load("epoch=036-val_loss=0.627335.ckpt", map_location="cpu")
model.load_state_dict(ckpt["state_dict"], strict=False)
model.eval()

Files

• epoch=036-val_loss=0.627335.ckpt - Best Lightning checkpoint (epoch 36)
• last.ckpt - Final checkpoint (epoch 49)
• bfm_rollout_eval_results.json - Full evaluation results
• finetune_rollout_large.py - Finetuning script
• eval_rollout_large.py - Evaluation script

Citation

@article{foerster2025bfm,
  title={A Foundation Model for Forecasting Biodiversity Dynamics},
  author={Foerster et al.},
  journal={arXiv preprint arXiv:2507.09080v2},
  year={2025}
}