GenomeOcean-100M-v1.2

GenomeOcean-100M-v1.2

GenomeOcean-100M-v1.2 is a 100-million-parameter causal language model for microbial genomic sequences. It is a continued-training checkpoint of GenomeOcean-100M (v1.0) trained on an expanded dataset that adds GTDB r226 representative genomes, INPHARED phage genomes, and the Zenodo RNA virus database to the original metagenomic training corpus.

Release Note

An earlier publication of this repository mistakenly paired the v1.2 model card and configuration with the original GenomeOcean-100M base-model weights. The current repository contents correct that issue and now publish the intended v1.2 continued-training checkpoint.

What's new in v1.2

Model Details

• Architecture: Mistral (decoder-only transformer)
• Parameters: ~100M
• Tokenizer: BPE, vocabulary size 4096
• Context length: 1024 tokens (~5 kbp)
• Training precision: bfloat16
• Training framework: DeepSpeed ZeRO-1
• Continued from: DOEJGI/GenomeOcean-100M (v1.0)

Quick Start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "DOEJGI/GenomeOcean-100M-v1.2",
    padding_side="left",
)
model = AutoModelForCausalLM.from_pretrained(
    "DOEJGI/GenomeOcean-100M-v1.2",
    torch_dtype=torch.bfloat16,
).to("cuda")

# Generate a sequence continuation
sequence = "ATGCGATCGATCGATCGATCG"
inputs = tokenizer(sequence, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Data

Special Token Usage

The v1.2 tokenizer uses special tokens in a specific way during training:

• [CLS] — prepended to every chunk of viral/phage/RNA sequences to act as a domain tag. This allows the model to learn sequence-type-specific representations.
• [SEP] — inserted at genome boundaries (between the last chunk of one genome and the first chunk of the next) during the packing step. This teaches the model that sequences are discrete units.
• [UNK] — should never appear in clean data. IUPAC ambiguity codes are resolved before tokenization.

Citation

If you use GenomeOcean in your research, please cite:

@article{genomeocean2025,
  title={GenomeOcean: A Pretrained Microbial Genome Foundational Model},
  author={...},
  journal={...},
  year={2025}
}

License

Copyright (c) 2025, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy) and Northwestern University. All rights reserved.

If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Intellectual Property Office at [email protected].

NOTICE. This Software was developed under funding from the U.S. Department of Energy and the U.S. Government consequently retains certain rights. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, distribute copies to the public, prepare derivative works, and perform publicly and display publicly, and to permit others to do so.