Downloads · 30 days
0
Nhoodie/omni-dna-20m-hgt-stacked-lora
omni-dna-20m-hgt-stacked-lora is a machine learning model from Nhoodie. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as mit.
This model is an unintended but valuable research artifact. It was produced across multiple interrupted training runs with known bugs. Its behavior is unpredictable and it should only be used for research into hierarc…
Downloads · 30 days
0
Access
Public
Updated Apr 16, 2026
Repo size
16.3 MB
Likes
0
Public
Click a slice to open those files.
.pt10.9 MB · 66%
From the Hugging Face model README
This model is an unintended but valuable research artifact. It was produced across multiple interrupted training runs with known bugs. Its behavior is unpredictable and it should only be used for research into hierarchical LoRA architectures.
3 sequential LoRA levels (rank=16 each) stacked on Omni-DNA-20M:
h = W_base(x) # frozen base model
h = h + B₀A₀(x) # Level 0: weight-space (fundamental)
h = h + B₁A₁(h) # Level 1: activation-space (1st harmonic)
h = h + B₂A₂(h) # Level 2: activation-space (2nd harmonic)
This is a heterogeneous architecture — each level operates in a different representational space. Global gradient clipping destroys learning by compressing all layers through a single bottleneck. Per-layer gradient norms must be tracked independently.
| Run | Steps | Data Batches | Why It Died |
|---|---|---|---|
| 1 | 0→4356 | 17,424 | OOM (eval memory leak + long sequence attention) |
| 2 | 4356→5742 | 5,544 | Killed to deploy log timestamps |
| 3 | 5742→7950 | 8,832 | OOM (same eval leak pattern) |
| 4 | 7920→9999 | 8,316 | Killed at completion |
Total: ~39,516 batches ≈ 5.0 effective epochs (but across 4 interrupted runs with reshuffled data due to broken epoch tracking — each restart reshuffles from epoch 0)
start_epoch = resume_step // steps_per_epochexport VAR=val && nohup (env var is lost)| Token | AUC | F1 | Samples (pos) |
|---|---|---|---|
| HGT_prok_host | 0.7855 | 0.0000 | 11 |
| HGT_arch_euk | 0.6417 | 0.0000 | 13 |
| HGT_detection | 0.6290 | 0.0000 | 97 |
| HGT_fungi_euk | 0.5118 | 0.0000 | 81 |
| HGT_bac_euk | 0.4557 | 0.0149 | 133 |
| HGT_prot_euk | 0.4393 | 0.0625 | 62 |
| Overall | 0.5140 | — | — |
Best overall AUC: 0.5745 at step 1287 (early training before snap-collapse oscillations)
| Token ID | Token | Type |
|---|---|---|
| 4096 | 0 | Label |
| 4097 | 1 | Label |
| 4098 | 2 | Label |
| 4117 | HGT_detection | Binary HGT detection |
| 4119 | HGT_bac_euk | Bacteria→Eukaryote transfer |
| 4120 | HGT_arch_euk | Archaea→Eukaryote transfer |
| 4121 | HGT_fungi_euk | Fungi→Eukaryote transfer |
| 4122 | HGT_prot_euk | Protist→Eukaryote transfer |
| 4123 | HGT_prok_host | Prokaryotic host transfer |
BASE_MODEL = "zehui127/Omni-DNA-20M" # expanded to omni-dna-20m-hgt
NUM_LEVELS = 3
LORA_R = 16
LORA_ALPHA = 32
TARGET_MODULES = ["att_proj", "attn_out", "ff_proj", "ff_out"]
LEARNING_RATE = 2e-4
GRAD_ACCUM = 4
MAX_LENGTH = 4096
MAX_TOKENS_PER_BATCH = 4096 # 1 sequence per batch
TRAINABLE_PARAMS = 2,693,312 # 20.41% of model
Total: 95,344 train / 23,837 eval
Training script and full eval history available at:
https://github.com/NamelessHoodie/dnabert-mut-exp/tree/dev/sad-omni-hyena/harmonics/
If you use this architecture or findings, please reference: