PlasmidLM-kmer6-MoE

PlasmidLM-kmer6-MoE

A 78.3M total parameter (31.1M active) Mixture-of-Experts autoregressive language model for plasmid DNA sequence generation, trained on ~100K plasmid sequences from Addgene.

Model Details

Training

• Data: ~100K plasmid sequences from Addgene, tokenized with k-mer (k=6, stride=3)
• Steps: 35,000
• Eval loss: 0.190
• Token accuracy: 98.4%

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("McClain/PlasmidLM-kmer6-MoE", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("McClain/PlasmidLM-kmer6-MoE", trust_remote_code=True)

# Condition on antibiotic resistance + origin of replication
prompt = "<BOS><AMR_KANAMYCIN><ORI_COLE1><SEP>"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.8, do_sample=True, top_p=0.95)
print(tokenizer.decode(outputs[0].tolist()))

The model generates plasmid DNA sequences conditioned on functional annotations (antibiotic resistance markers, origins of replication) provided as special tokens in the prompt.

MoE Architecture

Each transformer layer replaces the standard dense MLP with a Mixture-of-Experts layer containing 6 expert MLPs. A learned router selects the top-2 experts per token, so only 31.1M of the 78.3M total parameters are active for any given token. This provides greater model capacity while maintaining efficient inference.

Special Tokens

Citation

If you use this model, please cite:

@misc{thiel2026plasmidlm,
  title={PlasmidLM: Language Models for Plasmid DNA Generation},
  author={Thiel, McClain},
  year={2026}
}