enbed-dna-mutation-512bp

ENBED DNA Mutation Prediction (512bp, Checkpoint-200)

Fine-tuned variant of ENBED (ByT5-based DNA foundation model) for cross-domain mutation prediction across 9 domains of life, with emphasis on horizontal gene transfer (HGT) patterns.

Model Details

• Base model: malusare/dna-base-512-pretrained (~581M params, 18 encoder / 6 decoder layers)
• Fine-tuning task: Seq2seq DNA mutation prediction (parent → child sequence)
• Training data: 4,143 mutation pairs across 9 domains (Bacteria, Archaea, Viruses, Retroviridae, Viridiplantae, Metazoa, Viroids, Satellites, Lichenes HGT)
• Best checkpoint: Step 200 (eval loss 1.250)
• Context length: 512 tokens (byte-level: 1 base = 1 token)
• Hardware: Intel Hades Canyon NUC + GTX 1080 8GB eGPU

Training Configuration

• Optimizer: Adafactor
• Batch size: 1 (gradient accumulation: 16, effective batch = 16)
• Learning rate: 1e-4
• Precision: fp16 (model loaded in fp16, no AMP scaler)
• Gradient checkpointing: enabled
• Epochs: 5 (best at epoch ~0.8)

Eval Loss Progression

Model began overfitting after step 200. This checkpoint represents the best generalization point.

Intended Use

• Cross-domain DNA mutation prediction
• Horizontal gene transfer pattern analysis
• Synthetic mutation data generation for underrepresented domains
• Research baseline for DNA foundation model comparisons

Limitations

• Overfitting: 581M params on 4K pairs → overfits quickly. Best used as a baseline or synthetic data generator.
• Context: 512bp limit truncates longer sequences
• Domains with few examples (Bacteria, Archaea) will have poor performance
• HGT events (large insertions) are the hardest prediction task and likely near-random

Citation

If you use this model, please cite both the ENBED paper and this work:

@article{malusare2024enbed,
  title={ENBED: Ensemble Nucleotide Byte-level Encoder-Decoder for DNA Foundation Models},
  author={Malusare, Aditya and Lanman, Nadia and Aggarwal, Vaneet},
  journal={Bioinformatics Advances},
  year={2024}
}

License

MIT (following base model license)