Downloads · 30 days
0
hanseungwook/opengenome2-llama-small-nca-ckpts
opengenome2-llama-small-nca-ckpts is a text generation model from hanseungwook. Use it when you need the model to write or continue text. It is set up for transformers.
This repository contains OpenGenome2 byte-token Llama-small checkpoints from the NCA pre-pretraining transfer experiments. It also includes the original NCA1D k=7 and NCA-MAE source checkpoints from before OpenGenome2…
Downloads · 30 days
0
Access
Public
Updated Sep 6, 2026
Repo size
96 GB
Likes
0
Public
Click a slice to open those files.
.safetensors96 GB · 100%
From the Hugging Face model README
This repository contains OpenGenome2 byte-token Llama-small checkpoints from the NCA pre-pretraining transfer experiments. It also includes the original NCA1D k=7 and NCA-MAE source checkpoints from before OpenGenome2 training.
The checkpoints are stored in subfolders:
opengenome2-llama-small-nca-mae-mask30-srcE100-reinit-embed-7kopengenome2-llama-small-nca-mae-mask30-srcE100-reinit-embed-10kopengenome2-llama-small-nca-mae-mask30-srcseed11-srcE100-targetseed11-reinit-embed-7kopengenome2-llama-small-nca-mae-mask30-srcseed11-srcE100-targetseed11-reinit-embed-10kopengenome2-llama-small-nca-mae-mask30-srcseed13-srcE100-targetseed13-reinit-embed-7kopengenome2-llama-small-nca-mae-mask30-srcseed13-srcE100-targetseed13-reinit-embed-10kopengenome2-llama-small-nca-mae-mask30-srcseed0-srcE300-targetseed0-reinit-embed-10kopengenome2-llama-small-scratch-7kopengenome2-llama-small-scratch-10kopengenome2-llama-small-scratch-targetseed1-10kopengenome2-llama-small-scratch-defaultinit-targetseed0-10kopengenome2-llama-small-scratch-defaultinit-targetseed1-10kopengenome2-llama-small-dyck-ar-defaultinit-srcseed0-srcS5000-targetseed0-reinit-embed-10kopengenome2-llama-small-dyck-ar-defaultinit-srcseed1-srcS5000-targetseed1-reinit-embed-10kopengenome2-llama-small-nca1d-c20-k7-defaultinit-srcseed0-gzip0p5-1p0-srcE10-targetseed0-reinit-embed-10kopengenome2-llama-small-nca1d-c20-k7-defaultinit-srcseed1-gzip0p5-1p0-srcE10-targetseed1-reinit-embed-10kopengenome2-llama-small-dyck-mae-defaultinit-mask50-closeonly-srcseed0-srcS5000-targetseed0-reinit-embed-10kopengenome2-llama-small-dyck-mae-defaultinit-mask50-closeonly-srcseed1-srcS5000-targetseed1-reinit-embed-10kopengenome2-llama-small-nca-mae-mask30-defaultinit-srcseed0-srcE100-targetseed0-reinit-embed-10kopengenome2-llama-small-nca-mae-mask30-defaultinit-srcseed1-srcE100-targetseed1-reinit-embed-10kopengenome2-llama-small-nca1d-c20-k7-gzip0p5-1p0-srcE10-reinit-embed-7kopengenome2-llama-small-nca1d-c20-k7-gzip0p5-1p0-srcE10-reinit-embed-10kopengenome2-llama-small-nca1d-c20-k9-gzip0p5-1p0-srcE10-reinit-embed-7kopengenome2-llama-small-nca1d-c20-k9-gzip0p5-1p0-srcE10-reinit-embed-10kopengenome2-llama-small-nca1d-c20-k3-srcseed0-gzip0p5-1p0-srcE10-reinit-embed-10kopengenome2-llama-small-nca1d-c20-k6-srcseed0-gzip0p5-1p0-srcE10-reinit-embed-10kopengenome2-llama-small-nca1d-c20-k7-srcseed1-gzip0p5-1p0-srcE10-reinit-embed-10kopengenome2-llama-small-nca1d-c20-k7-srcseed2-gzip0p5-1p0-srcE10-reinit-embed-10kopengenome2-llama-small-dyck-50kopengenome2-llama-small-dyck-5kopengenome2-llama-small-dyck-mae-5k-ppt-7k-ptopengenome2-llama-small-dyck-mae-5k-ppt-10k-ptopengenome2-llama-small-dyck-mae-50k-ppt-7k-ptopengenome2-llama-small-dyck-mae-50k-ppt-10k-ptopengenome2-llama-small-mimetic-aqk0.0-avo0.2-bqk0.2-bvo0.2opengenome2-llama-small-mimetic-aqk0.0-avo0.2-bqk0.2-bvo0.5opengenome2-llama-small-mimetic-aqk0.0-avo0.2-bqk0.5-bvo0.2opengenome2-llama-small-mimetic-aqk0.0-avo0.2-bqk0.5-bvo0.5opengenome2-llama-small-mimetic-aqk0.0-avo0.2-bqk0.5-bvo0.5-fixedopengenome2-llama-small-mimetic-aqk0.0-avo0.2-bqk0.5-bvo0.5-fixed-defaultinit-targetseed0-10kopengenome2-llama-small-mimetic-aqk0.0-avo0.2-bqk0.5-bvo0.5-fixed-defaultinit-targetseed1-10kEach subfolder contains model.safetensors, config.json, generation_config.json, a small byte-tokenizer helper, and a conversion_report.json.
The conversion reports compare logits against the original repo checkpoint before saving and after reloading the Hugging Face export.
The 7k NCA checkpoints are included because they are already comparable to, or better than, the scratch model's 10k validation loss in the rerun.
The opengenome2-llama-small-scratch-targetseed1-10k checkpoint is an independently trained scratch baseline using OpenGenome2 target/data seed 1. The original opengenome2-llama-small-scratch-10k checkpoint uses target/data seed 0.
The opengenome2-llama-small-scratch-defaultinit-targetseed{0,1}-10k checkpoints rerun target/data seeds 0 and 1 with both the input embedding and output projection explicitly initialized from $\mathcal{N}(0, 0.02^2)$. Their final validation losses are 1.29941 and 1.29977, respectively. The earlier scratch checkpoints used the legacy PyTorch defaults for those newly constructed modules.
The default-init matrix checkpoints consistently use defaultinit in their folder names and record embedding_output_init: default in both config.json and conversion_report.json. For these runs, the newly constructed input embedding and output projection use $\mathcal{N}(0, 0.02^2)$ initialization.
| Default-init checkpoint | Source seed | Target seed | OpenGenome2 step | Validation loss |
|---|---|---|---|---|
opengenome2-llama-small-dyck-ar-defaultinit-srcseed0-srcS5000-targetseed0-reinit-embed-10k | 0 | 0 | 10,000 | 1.29050 |
opengenome2-llama-small-dyck-ar-defaultinit-srcseed1-srcS5000-targetseed1-reinit-embed-10k | 1 | 1 | 10,000 | 1.28935 |
opengenome2-llama-small-nca1d-c20-k7-defaultinit-srcseed0-gzip0p5-1p0-srcE10-targetseed0-reinit-embed-10k | 0 | 0 | 10,000 | 1.29397 |
opengenome2-llama-small-nca1d-c20-k7-defaultinit-srcseed1-gzip0p5-1p0-srcE10-targetseed1-reinit-embed-10k | 1 | 1 | 10,000 | 1.29593 |
opengenome2-llama-small-dyck-mae-defaultinit-mask50-closeonly-srcseed0-srcS5000-targetseed0-reinit-embed-10k | 0 | 0 | 10,000 | 1.29636 |
opengenome2-llama-small-dyck-mae-defaultinit-mask50-closeonly-srcseed1-srcS5000-targetseed1-reinit-embed-10k | 1 | 1 | 10,000 | 1.29723 |
opengenome2-llama-small-nca-mae-mask30-defaultinit-srcseed0-srcE100-targetseed0-reinit-embed-10k | 0 | 0 | 10,000 | 1.29512 |
opengenome2-llama-small-nca-mae-mask30-defaultinit-srcseed1-srcE100-targetseed1-reinit-embed-10k | 1 | 1 | 10,000 | 1.29855 |
opengenome2-llama-small-mimetic-aqk0.0-avo0.2-bqk0.5-bvo0.5-fixed-defaultinit-targetseed0-10k | n/a | 0 | 10,000 | 1.30113 |
opengenome2-llama-small-mimetic-aqk0.0-avo0.2-bqk0.5-bvo0.5-fixed-defaultinit-targetseed1-10k | n/a | 1 | 10,000 | 1.29769 |
The dyck checkpoints represent the dyck baseline, 5k and 50k refer to the pre-pretraining budget, where 5k outperforms.
The dyck-mae checkpoints represent OpenGenome2 training initialized from Dyck-MAE pre-trained checkpoints. In their names, 5k and 50k refer to the Dyck-MAE pre-pretraining budget, while ppt-7k-pt and ppt-10k-pt refer to the OpenGenome2 training step.
The mimetic checkpoints represent initialization from mimetic initialization. The aqk0.0-avo0.2-bqk0.5-bvo0.2 and aqk0.0-avo0.2-bqk0.5-bvo0.5 variants are best on OpenGenome language modeling.
The -fixed checkpoint family uses the corrected SVD factor orientation and full-width value/output factorization. Checkpoints without the -fixed suffix are legacy mimetic exports and should not be treated as equivalent. The fixed family preserves the existing coefficient naming convention and stores independently trained target seeds in seed-0 and seed-1 subfolders.
| Fixed mimetic checkpoint | Target seed | OpenGenome2 step | Validation loss |
|---|---|---|---|
opengenome2-llama-small-mimetic-aqk0.0-avo0.2-bqk0.5-bvo0.5-fixed/seed-0 | 0 | 10,000 | 1.30295 |
opengenome2-llama-small-mimetic-aqk0.0-avo0.2-bqk0.5-bvo0.5-fixed/seed-1 | 1 | 10,000 | 1.30078 |
These six checkpoints contain the Llama-small models immediately after NCA pre-pre-training and before their transfer to OpenGenome2. They use NCA patch vocabularies rather than the 512-token OpenGenome2 byte vocabulary.
| Family | Source seed | Source checkpoint | Vocabulary | Attention |
|---|---|---|---|---|
| NCA1D k=7 | 0 | seed-0-e10 | 160,000 | causal |
| NCA1D k=7 | 1 | seed-1-e10 | 160,000 | causal |
| NCA1D k=7 | 2 | seed-2-e10 | 160,000 | causal |
| NCA-MAE, 30% mask | 0 | seed-0-e300 | 65,540 | bidirectional |
| NCA-MAE, 30% mask | 11 | seed-11-e100 | 65,540 | bidirectional |
| NCA-MAE, 30% mask | 13 | seed-13-e100 | 65,540 | bidirectional |
Each source folder contains standard Hugging Face LlamaForCausalLM weights, its own model card, and a conversion report. All exports passed logit-parity checks against the original checkpoint before saving and after reloading at sequence lengths 8, 128, and 1024. For NCA-MAE, the model card shows how to pass an unrestricted attention mask to reproduce the source model's bidirectional inference behavior.
See PREPRETRAINED_SOURCES.md for the compact source-checkpoint catalog.
The dyck-mae checkpoints were initialized from Dyck-MAE source checkpoints at 5k or 50k pre-pretraining steps, then trained on OpenGenome2 to 7k or 10k steps.
| Checkpoint | Dyck-MAE budget | OpenGenome2 step | Best validation loss |
|---|---|---|---|
opengenome2-llama-small-dyck-mae-5k-ppt-7k-pt | 5,000 | 7,000 | 1.29361 |
opengenome2-llama-small-dyck-mae-5k-ppt-10k-pt | 5,000 | 10,000 | 1.29174 |
opengenome2-llama-small-dyck-mae-50k-ppt-7k-pt | 50,000 | 7,000 | 1.30007 |
opengenome2-llama-small-dyck-mae-50k-ppt-10k-pt | 50,000 | 10,000 | 1.29857 |
The nca-mae-mask30 checkpoints were initialized from bidirectional Llama-small NCA-MAE sources trained with 30% masking. The source uses 2D cellular automata with grid size 32, patch size 4, two colors, and gzip-complexity filtering from 0.0 to 1.0. Only the 512-token OpenGenome2 input and output embeddings were reinitialized for causal language-model training; the transformer backbone was transferred.
OpenGenome2 training used sequence length 1024, effective global batch 2048, learning rate 1e-5, and weight decay 0.1. The three lineage-labeled runs use matching source and target seeds. Every converted lineage checkpoint passed exact pre-save and post-reload logit parity at sequence lengths 8, 128, and 1024.
| Checkpoint | Source seed | Source epoch | Target seed | Step | Train loss | Validation loss |
|---|---|---|---|---|---|---|
opengenome2-llama-small-nca-mae-mask30-srcE100-reinit-embed-7k | 0 | 100 | 0 | 7,000 | 1.30228 | 1.29800 |
opengenome2-llama-small-nca-mae-mask30-srcE100-reinit-embed-10k | 0 | 100 | 0 | 10,000 | 1.30230 | 1.29628 |
opengenome2-llama-small-nca-mae-mask30-srcseed11-srcE100-targetseed11-reinit-embed-7k | 11 | 100 | 11 | 7,000 | 1.30394 | 1.29736 |
opengenome2-llama-small-nca-mae-mask30-srcseed11-srcE100-targetseed11-reinit-embed-10k | 11 | 100 | 11 | 10,000 | 1.30308 | 1.29594 |
opengenome2-llama-small-nca-mae-mask30-srcseed13-srcE100-targetseed13-reinit-embed-7k | 13 | 100 | 13 | 7,000 | 1.30613 | 1.29847 |
opengenome2-llama-small-nca-mae-mask30-srcseed13-srcE100-targetseed13-reinit-embed-10k | 13 | 100 | 13 | 10,000 | 1.30004 | 1.29691 |
opengenome2-llama-small-nca-mae-mask30-srcseed0-srcE300-targetseed0-reinit-embed-10k | 0 | 300 | 0 | 10,000 | 1.30501 | 1.29880 |
For the same source and target seed 0, extending NCA-MAE source training from epoch 100 to epoch 300 did not improve OpenGenome2 transfer: validation loss changed from 1.29628 to 1.29880.
import importlib.util
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM
repo_id = "hanseungwook/opengenome2-llama-small-nca-ckpts"
subfolder = "opengenome2-llama-small-nca-mae-mask30-srcseed11-srcE100-targetseed11-reinit-embed-10k"
model = AutoModelForCausalLM.from_pretrained(
repo_id,
subfolder=subfolder,
torch_dtype=torch.bfloat16,
device_map="auto",
)
tokenizer_path = hf_hub_download(repo_id, filename="opengenome2_byte_tokenizer.py")
spec = importlib.util.spec_from_file_location("opengenome2_byte_tokenizer", tokenizer_path)
tokenizer_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(tokenizer_module)
tokenizer = tokenizer_module.OpenGenome2ByteTokenizer()
input_ids = torch.tensor([tokenizer.encode("ACGT", add_eod=False)], device=model.device)
logits = model(input_ids).logits
These models use Evo2-compatible UTF-8 byte IDs with a 512-token vocabulary:
A C G T N map to 65 67 71 84 78eod_id = 0pad_id = 1The root opengenome2_byte_tokenizer.py file can be reused for every checkpoint subfolder. Each subfolder also includes the same helper file for self-contained downloads.