moss-dna-gpt-20m-patens

moss-dna-gpt-20m-patens

This is a custom PyTorch model, not a Hugging Face Transformers AutoModel-compatible checkpoint.
Loading it requires the moss_dna_gpt package from github.com/Unjuno/moss-dna-gpt.

Experimental 20M-parameter decoder-only Transformer trained for DNA next-base prediction on Physcomitrium patens (moss) genomic sequence.

Key result: The model achieves 1.41665 bits/base on a held-out sequence-level test split, outperforming all Markov baselines (including Interpolated Markov Model up to order 5) by 24.9% in next-base prediction cross-entropy.

Model description

• Architecture: Decoder-only Transformer (GPT)
• Parameters: 21,691,008
• Context length: 1,024 bases
• Vocabulary: A, C, G, T, N + special tokens (BOS, EOS, PAD, UNK)
• Tokenizer: Single-nucleotide (no k-mer or BPE)
• Training steps: 8,000,000
• Optimizer: AdamW (lr=3e-4, weight_decay=0.1)
• Hardware: Single GPU
• Training time: ~2 weeks

Intended use

• Research on DNA sequence patterns and next-base prediction
• Comparing neural language models against classical Markov baselines
• Educational demonstrations of small-scale genomic language modeling

Not intended for

• SNP effect or variant pathogenicity prediction
• Gene function or phenotype prediction
• Environmental adaptation prediction
• Clinical, agricultural, or ecological decision-making
• Cross-species generalization (trained on moss only)

This repository does not provide a Hugging Face Transformers pipeline.
Use the custom PyTorch loading code below.

Quickstart

# Clone and install the package
git clone https://github.com/Unjuno/moss-dna-gpt.git
cd moss-dna-gpt
pip install -e ".[hf]"

import torch
from safetensors.torch import load_file

from moss_dna_gpt.model import GPT, GPTConfig
from moss_dna_gpt.tokenizer import DnaTokenizer
from moss_dna_gpt.sampling import sample

device = "cuda" if torch.cuda.is_available() else "cpu"

# Load config
config = GPTConfig(
    vocab_size=9, block_size=1024, n_layer=12,
    n_head=8, n_embd=384, dropout=0.1, bias=True,
)

# Load weights
model = GPT(config).to(device)
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()

# Generate
tokenizer = DnaTokenizer()
prefix = "ACGTACGTACGT"
ids = tokenizer.encode(prefix, unknown="n")
idx = torch.tensor([ids], dtype=torch.long, device=device)
allowed = [tokenizer.stoi[b] for b in tokenizer.dna_tokens]
out = sample(model, idx, max_new_tokens=128, temperature=0.8, top_k=4, allowed_token_ids=allowed)
print(tokenizer.decode(out[0].tolist(), skip_special=True))

Evaluation results

Evaluation on held-out sequence-level test windows (177,723,392 tokens):

Note on evaluation settings. The final step-8M evaluation used the full test set (~177M tokens). The intermediate curve points used a smaller subset (eval_batches=200). Learning-curve points and the release metric are not directly interchangeable due to different evaluation batch sizes. The canonical release claim is 1.41665 bits/base from results/eval_markov_20m_step8000000.json.

The learning curve shows DNA-GPT surpassing all Markov baselines within the first 750K steps:

Files

Training data

• Organism: Physcomitrium patens ecotype Gransden 2004
• Assembly: GCA_000002425.3 / Phypa_V5
• Source: NCBI GenBank genomic FASTA (not redistributed; obtain from NCBI and regenerate windows locally)
• Window size: 1,024
• Stride: 512
• Split: Sequence-level split (train/val/test)
• Raw FASTA and processed shards are not distributed in this repository.

Citation

@software{moss_dna_gpt_2026,
  author = {Unjuno},
  title = {moss-dna-gpt: Minimal DNA Language Model for Moss Genome},
  year = {2026},
  url = {https://github.com/Unjuno/moss-dna-gpt}
}

Related models

• DNABERT — BERT for DNA
• Nucleotide Transformer — Large-scale genomic LM
• HyenaDNA — Long-context DNA model
• Evo 2 — Genome-scale generative model
• Carbon — Efficient eukaryotic DNA LM