BulkFormer-147M

BulkFormer-147M

A foundation model for bulk RNA-seq transcriptomes. This is the largest of five released BulkFormer variants (37M / 50M / 93M / 127M / 147M parameters), converted to a transformers-compatible format.

• Paper: Kang, Fan et al., Cell Systems 2026. DOI
• Original repo: KangBoming/BulkFormer
• HF-compatible package: zhfanrui/BulkFormer_hf

Usage

pip install bulkformer

from bulkformer import BulkFormerModel, extract_features
from bulkformer.preprocessing import normalize_data, align_to_vocab

model = BulkFormerModel.from_pretrained("zhfanrui/BulkFormer-147M")

# counts_df: [n_samples, n_genes] raw counts, columns = Ensembl gene IDs
log_tpm = normalize_data(counts_df)
aligned, mask_prob = align_to_vocab(log_tpm, model.config.gene_ids)

sample_emb = extract_features(
    model, aligned.values, output_feature_type="sample_level",
    aggregate_type="mean", mask_prob=mask_prob, device="cuda", batch_size=32,
)

See the package README for gene-level / expression-level extraction, fine-tuning (classification head, continued masked-imputation pretraining), and ESM2 embedding fusion.

This checkpoint bundles the pretrained gene-gene graph directly into model.safetensors as non-trainable buffers -- no separate graph file needed. config.json includes the exact gene vocabulary (gene_ids, 20,010 Ensembl IDs) this checkpoint expects, in order.

Citation

@article{kang2026bulkformer,
  title = {BulkFormer: A large-scale foundation model for bulk transcriptomes},
  author = {Kang, Boming and Fan, Rui and Yi, Meizheng and Cui, Chunmei and Cui, Qinghua},
  journal = {Cell Systems},
  year = {2026},
  pages = {101657},
  doi = {10.1016/j.cels.2026.101657}
}

License

MIT.