scaling-mmbert-40k

scaling-mmbert-40k (Stage A, warmed base)

Research checkpoint from the mmBERT head-init data-scaling curve (private). mmBERT-base with fresh GLiNER2 heads (from_encoder) warmed on ~40,000 structure/argument-dense event records (multilingual, assembled from public event corpora; RAMS/WikiEvents held out to avoid leakage), 5 epochs.

This is the N=40k point's warmed base — the reusable head-init artifact. It is not fine-tuned on any single downstream task; fine-tune it (e.g. on RAMS) to obtain a usable event model. The paired fine-tune is whr778/scaling-mmbert-40k-rams.

Purpose

Measures how much structure/argument data is needed to warm mmBERT's fresh extraction heads before downstream fine-tuning (head-initialization thesis). See the experiment spec SCALING_CURVE_EXPERIMENT.md.

• Encoder: jhu-clsp/mmBERT-base (native long-context, 2048 training window)
• Stage-A corpus size: ~40,000 records
• Loss: bce_posweight (pos_weight 4.0); differential-style LR (encoder 2e-5 / heads 5e-4)

Caveats

Research artifact, private, not benchmarked as a general model. Label quality is bounded by the source event corpora. One point on a scaling curve, not a release.