ear-eeg-fm-benchmark-loso

Ear-EEG FM Benchmark — LOSO checkpoints

Per-fold leave-one-subject-out (LOSO) checkpoints for the scalp-EEG vs. in-ear-EEG sleep-staging benchmark in Ear-EEG-FM-Benchmark: three datasets x two modalities x up to seven models, one checkpoint per held-out subject. 386 checkpoints, 14.3 GiB.

These are fine-tuned task checkpoints, not pretrained encoders. Each file is the adapter state_dict selected by the best validation balanced accuracy for that fold — the test set is never used for model selection.

Layout

The tree mirrors the benchmark repo's own runs/, so a download can be dropped straight into runs/ and every training/eval script resolves it unchanged:

<dataset>_loso/<dataset>-<modality>/<model>_ep<E>_lr<LR>/
    loso_results.csv            one row per held-out subject + mean/std
    aggregate.json              mean+-std over subjects, val and test
    seed1/resolved_config.yaml  the fully resolved run config
    seed1/fold-sub-sub-XXX/
        best.pt                 val-selected adapter state_dict
        metrics.json            val/test metrics, per-epoch history, fold_meta
manifest.json                   every run-dir, its folds, and the seed

<modality> is scalp-eeg or in-ear-eeg; fold-sub-sub-XXX names the held-out test subject. Only seed1 is published.

Coverage

REVE places channels by 3D coordinate. EESM17's ear electrodes (ELA..ERI) are not in its position bank, so EESM17 uses the ch_embed=mastoid variant; EESM23's channel names resolve directly, so it uses plain reve. EESM19 currently has only the three models above — biot / labram_base / reve were not run on it.

Results (test balanced accuracy, mean +- std over held-out subjects, 5 classes, chance 0.20)

EESM17 (9 subjects)

EESM19 (20 subjects)

EESM23 (10 subjects)

Cohen's kappa and weighted F1 for every run are in each aggregate.json; per-subject numbers are in loso_results.csv. EEGNet's low scores on EESM17/EESM23 are what the shared recipe produced for it on those datasets — they are reported as measured, not tuned away.

Training recipe

One shared recipe for every model (training/config.py in the source repo): AdamW (beta 0.9/0.999, weight decay 0.05), batch size 64, 100 epochs, cosine decay to min_lr 1e-6 with 10% linear warmup, label smoothing 0.1. Only the learning rate differs by family — 1e-3 for the from-scratch models (eegnet, eegconformer, eegdeformer), 5e-4 for the foundation models (biot, cbramod, labram_base, reve). There is no per-model LR search. Each run's exact resolved settings are in its seed1/resolved_config.yaml.

Usage

hf download Zachary1150/ear-eeg-fm-benchmark-loso --local-dir runs/

import torch
from model.registry import build_fm          # from the Ear-EEG-FM-Benchmark repo
from dataset.loader import read_meta

meta = read_meta("data/EESM23-Processed/eesm23-in-ear-eeg.h5")
adapter = build_fm("cbramod", num_classes=meta["n_class"], ch_names=list(meta["ch_names"]))
sd = torch.load("runs/eesm23_loso/eesm23-in-ear-eeg/cbramod_ep100_lr5e-4/"
                "seed1/fold-sub-sub-001/best.pt", map_location="cpu")
adapter.load_state_dict(sd)                  # fold-sub-sub-001 -> evaluate on sub-001

A checkpoint is only valid for the fold it names: every other subject was in its training set, so evaluating it on them leaks.

Provenance

The foundation-model checkpoints are fine-tuned from their upstream pretrained weights (BIOT, CBraMod, LaBraM, REVE — see the source repo for the exact releases); the traditional models are trained from scratch. Training data is the EESM17 (OpenNeuro ds004348), EESM19 and EESM23 sleep datasets. Any redistribution or downstream use of these weights is subject to the terms of those upstream pretrained models and datasets. Not a medical device; not for clinical use.