amplify-ablations

AMPLIFY-ablations

Pretraining data ablations from "pLM representations unlock metagenomic space beyond homology" (Le Breton et al.). They are research artifacts for studying how training data composition shapes protein language model representations. For the final models the paper releases, use flair-bio/AMPLIFY-B or flair-bio/AMPLIFY-C.

Every model is a subfolder, grouped by ablation family. All models share the 120M architecture (24 layers, hidden size 640). Most were trained for 100k steps (about 200B tokens). Names ending in _1m were trained for 1M steps (about 2T tokens). The clustering-fixed-epochs models are the exception, running a fixed 2 epochs over their clustered set, so their step count varies with dataset size. The exact data mixture behind each name (RED filtering quantile, clustering threshold, metagenomic subsampling rate) is tabulated in the paper's supplementary material. Per-model evaluation scores are in the supplementary data under in_silico_results.csv.

Usage

from transformers import AutoModel, AutoTokenizer

repo = "flair-bio/AMPLIFY-ablations"
model = AutoModel.from_pretrained(repo, subfolder="uniref100-filtering/MILA_U100_0.0", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="uniref100-filtering/MILA_U100_0.0", trust_remote_code=True)

inputs = tokenizer(["MSVVGIDLGFQSCYVAVARAGGIETIANEYSDRCTPACISF"], return_tensors="pt")
out = model(**inputs, output_hidden_states=True)

Models

uniref100-filtering/

UniRef100 at a sweep of RED filtering quantiles. (SI Table S7.)

bfd-filtering/

BFD at a sweep of RED filtering quantiles. (SI Table S8.)

mgnify-filtering/

MGnify at a sweep of RED filtering quantiles. (SI Table S9.)

combined-filtering/

UniRef100, BFD and MGnify filtered together at a common RED quantile. (SI Table S10.)

subsampling/

Filtered metagenomic data mixed with UniRef100 at a sweep of subsampling rates. (SI Table S11.)

red-subsampling/

A common RED threshold plus metagenomic subsampling. (SI Table S12.)

clustering/

UniRef100 clustered at a sweep of identity thresholds, at a fixed 100k-step budget. (SI Table S13.)

clustering-fixed-epochs/

The same clustering sweep trained for a fixed 2 epochs instead of a fixed step budget. (SI Table S13.)

Control runs for the clustering sweep, trained for 2 epochs over the clustered set rather than to a fixed step budget, so the step count varies with dataset size. The clustering result reported in the paper (Figure 2E-F) uses the clustering/ models above.

unfiltered/

Baselines on unfiltered dataset mixtures. (SI Table S6.)

Citation

@article{lebreton_plm_metagenomic,
  title  = {pLM representations unlock metagenomic space beyond homology},
  author = {Le Breton, Lola and Heurtel-Depeiges, David and Millar, Douglas C. and
            Zetzsche, Lara E. and Vernon, Robert M. and Langmead, Christopher James and
            Chandar, Sarath and Fournier, Quentin}
}