seqlens-v2-micro-16k

SeqLens v2 Micro 16K

A compact genomic language model pre-trained on prokaryotic genomes for microbial bioinformatics tasks.

Model Description

SeqLens v2 is a bidirectional genomic language model built on BiMamba (bidirectional Mamba2 SSM) with interleaved sliding-window attention. It is designed for microbial genomics — taxonomic classification, antimicrobial resistance detection, plasmid identification, and metagenomic analysis.

The Micro variant is the smallest in the SeqLens v2 family, targeting high-throughput, low-latency inference.

Architecture Details

• BiMamba blocks: Bidirectional Mamba2 SSM — processes sequences in both forward and reverse directions using shared weights. Provides O(L) scaling with sequence length.
• Sliding-window attention: Applied every 4th layer with window size 512. Captures fine-grained local patterns (codons, motifs) that SSMs can miss.
• Attention-weighted pooling: Learned pooling for sequence-level embeddings (superior to mean pooling for downstream tasks).
• SwiGLU FFN: Gated feed-forward with 4× expansion at each layer.

Training

Data

Pre-trained on prokaryotic genomes from OpenGenome2 (Apache 2.0):

• GTDB v220: 113,379 species-cluster representative genomes
• Single-nucleotide tokenization, 16,384 bp chunks
• Quality filtered: sequences with >10% N or low Shannon entropy excluded

Hyperparameters

Compute

Evaluation

Coding vs Non-coding Classification (linear probe, frozen backbone)

Genus Classification (50 held-out genera, linear probe, frozen backbone)

Note: Genus classification is composition-dominated (GC content, tetranucleotide frequencies), where k-mer baselines are expected to be competitive. The coding/non-coding task better reflects the model's learned structural and positional representations.

Extended Benchmark Comparison

All evaluations below use the frozen linear probe protocol (frozen embeddings → LogisticRegression). Published fine-tuned baselines (e.g., ProkBERT's fine-tuned MCC scores) are not directly comparable — this is an apples-to-apples comparison across models with a fixed downstream classifier.

ProkBERT Prokaryotic Benchmarks (accuracy)

NTv3-8M leads this suite (trained on 9T bp across all species, ~36× more data than our 248B prokaryotic tokens). Our v2_standard (this model) is the best of our own three recipes here — the opposite ranking from CDS/GenomicBenchmarks, where the novel recipe wins. See PROJECT_STATE.md §3.4 for a per-task recipe breakdown.

GenomicBenchmarks — Eukaryotic Tasks (accuracy)

The human_ensembl_regulatory task is excluded: our single-nucleotide tokenizer preserves exact sequence length, and length alone is a near-complete shortcut for this task's 3-way label (a Random Forest trained on sequence length alone reaches 91.3% accuracy). Our models score ~0.99–1.00 on this task as an artifact of that leakage, not because of learned regulatory biology. Despite training exclusively on prokaryotic genomes, our 10M models remain competitive with human/multi-species models up to 10× our size on the remaining 7 eukaryotic tasks.

Full results: experiments/eval_reports/genomic_benchmarks/comparison.json and experiments/eval_reports/prokbench/comparison.json.

Usage

import torch
from model import SeqLensForMLM
from config import SeqLensConfig, MICRO_CONFIG
from tokenizer import NucleotideTokenizer

# Load model
device = torch.device("cuda")
ckpt = torch.load("seqlens-v2-micro-16k.pt", map_location=device)
model = SeqLensForMLM(MICRO_CONFIG).to(device).to(torch.bfloat16)
model.load_state_dict(ckpt["model"])
model.eval()

# Tokenize a DNA sequence
tokenizer = NucleotideTokenizer(max_len=16384)
seq = "ATGCGATCGATCG..." # your DNA sequence
token_ids = torch.tensor([tokenizer.encode(seq)], dtype=torch.long).to(device)

# Get sequence-level embeddings (for classification tasks)
with torch.no_grad():
    embeddings = model.get_embeddings(token_ids, pool="attention")  # (1, 256)

# Or get per-position predictions (MLM)
with torch.no_grad():
    output = model(token_ids)
    logits = output["logits"]  # (1, L, 9)

Model Family

Limitations

• Pre-trained on prokaryotic genomes only — may underperform on eukaryotic tasks
• 16K context may truncate long contigs; longer variants planned
• Current model trained for ~1.2B tokens; extended training may improve performance
• The human_ensembl_regulatory GenomicBenchmarks task is excluded from our averages — it's a confirmed sequence-length artifact of single-nucleotide tokenization, not a genuine capability (see Extended Benchmark Comparison above)

Citation

@misc{seqlens-v2-2026,
  title={SeqLens v2: Compact Genomic Language Models for Microbial Bioinformatics},
  author={SeqSight Team},
  year={2026},
  url={https://huggingface.co/seqSight/seqlens-v2-micro-16k}
}

License

Apache 2.0