Llama-3-Genolator-v1-PST

Genolator V1 — DNA + PST

Genolator answers questions about human gene function from embeddings alone. It never sees the gene's name or its raw sequence. Precomputed embeddings of the coding DNA sequence and of the predicted 3D protein structure are projected into the token embedding space of a biomedical Llama-3 and prepended to the prompt as virtual tokens, so the only route to a correct answer is reasoning over the fused representation rather than recall about a named gene.

This repository holds the DNA + PST variant. It was trained and evaluated on CHGGM-Aachen/genolator-v1-qa, whose splits are gene-disjoint.

It ships as adapters only: the LoRA weights and the two projectors, without the frozen base model. That is the 752 MiB that was actually trained, rather than 15.7 GiB that is 95% a byte-for-byte copy of ContactDoctor/Bio-Medical-Llama-3-8B. The base weights are fetched from that repository at load time — see Usage.

• Base model: ContactDoctor/Bio-Medical-Llama-3-8B
• Adaptation: LoRA (r=8, alpha=32, dropout=0.05) plus two trained-from-scratch virtual token projectors
• Total parameters in the checkpoint: 0.02B
• Code: IHGGM-Aachen/Genolator
• Paper: Danner et al., bioRxiv 2025

Architecture

Two linear projectors, one per modality, each map a pooled embedding to 8 virtual tokens of width 4096:

The 2 x 8 virtual tokens are prepended to the tokenised prompt and masked out of the loss. Each projector also carries a learned null_emb, substituted when a modality is missing for a gene, so a gene without a structure prediction is still usable. Virtual embeddings are scaled to match text embedding norms (--scale_virtual_embeddings true).

LoRA adapters are attached to down_proj, gate_proj, k_proj, o_proj, q_proj, up_proj, v_proj.

Files

Every tensor here was trained. The base model's weights were frozen throughout and are not duplicated in this repository.

AutoModelForCausalLM.from_pretrained("CHGGM-Aachen/Llama-3-Genolator-v1-PST") will not work. These are plain torch.save state dicts, not a transformers directory. Nor is this a PEFT adapter directory: there is no adapter_config.json, so PeftModel.from_pretrained will not read it either. It is a filtered state_dict. See Usage for the two lines that turn it into something run_inference.py accepts.

The three .pt files are one model and have to be used together. All three were trained in the same run, dna_projector.pt included: the DNA projector is fused against predicted 3D protein structure during training and is specific to that pairing. The other Genolator V1 variant ships a dna_projector.pt of the same shape under the same name holding different weights, so substituting one for the other raises no error — shapes match, the load succeeds, and the answers simply get worse. Keep each model's three files together.

Usage

git clone https://github.com/IHGGM-Aachen/Genolator.git
cd Genolator
uv sync

hf download CHGGM-Aachen/Llama-3-Genolator-v1-PST --local-dir ./genolator_pst

Then merge the adapters into the base model once, which produces exactly the file the inference code expects. run_inference.py needs no modification:

import torch
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model, set_peft_model_state_dict

HF_TOKEN = "hf_..."                 # gated Llama-3 weights
CKPT = "./genolator_pst"

model = AutoModelForCausalLM.from_pretrained("ContactDoctor/Bio-Medical-Llama-3-8B", token=HF_TOKEN)
model = get_peft_model(model, LoraConfig(
    r=8, lora_alpha=32, lora_dropout=0.05,
    target_modules=["down_proj", "gate_proj", "k_proj", "o_proj", "q_proj", "up_proj", "v_proj"], bias="none",
))
set_peft_model_state_dict(model, torch.load(f"{CKPT}/genolator_dna_and_pst_lora.pt", weights_only=True))
torch.save(model.state_dict(), f"{CKPT}/genolator_dna_and_pst.pt")

That writes a 15 GiB genolator_dna_and_pst.pt locally — the same bytes the training run produced. Do it once; everything below then works unchanged.

export HF_TOKEN=hf_...        # gated Llama-3 weights
export CHECKPOINT_DIR=./genolator_pst
cd examples && ./inference_pst.sh

Or directly:

python3 run_inference.py \
    --embedding_type pst \
    --dataset CHGGM-Aachen/genolator-v1-qa \
    --split test \
    --output_dir ./inference \
    --trained_model_path ./genolator_pst/genolator_dna_and_pst.pt \
    --trained_dna_projector_path ./genolator_pst/dna_projector.pt \
    --trained_protein_projector_path ./genolator_pst/pst_projector.pt \
    --hf_token "$HF_TOKEN" \
    --num_virtual_tokens 8 \
    --lora_r 8 \
    --lora_alpha 32 \
    --lora_dropout 0.05 \
    --lora_all_target_modules true \
    --scale_virtual_embeddings true

The LoRA and scaling flags must match the values above. They are not stored in the checkpoint and cannot be inferred from it. Get --lora_r or --lora_all_target_modules wrong and load_state_dict fails on a shape mismatch, which is the good case; get --scale_virtual_embeddings or --num_virtual_tokens wrong and it loads fine but the fused embeddings are the wrong magnitude and the output quietly degrades.

Inference needs the precomputed cdna_seq_embedding and pst_embedding columns. To run on genes outside the published dataset, generate them with the same encoders and pooling; the model has no way to encode raw sequence itself.

This repository is 752 MiB, but a run still needs the ~16 GiB base model resident. Inference was validated on a single A100 80GB.

Training

Trained on the train split of CHGGM-Aachen/genolator-v1-qa, with early stopping on the validation split. Reproduce with:

cd examples && HF_TOKEN=hf_... ./train_pst.sh

Reproduction is not bitwise: no RNG seed is set and the data loaders shuffle, so a rerun lands on a nearby but different set of weights.

A note on that loss figure: sequences are padded to a fixed length and the padding positions keep live labels, so the value averages answer tokens together with padding and sits well below the loss on answer tokens alone. It is meaningful for comparing epochs within this run, which is what early stopping used it for, but not comparable to losses reported elsewhere. Judge quality by the task metrics reported in the paper, not by this number.

Evaluation

The model was evaluated on the held-out test split using the task metrics described in the paper. Because the splits are gene-disjoint, no gene in the test split was seen during training.

run_inference.py produces the generations that evaluation runs on: it writes a pickled DataFrame of prompts, references and predictions. It does not compute metrics itself — scoring is a separate step.

Limitations

• Human genes only, and only genes with the required embeddings available.
• Answers Gene Ontology-style questions about function. It is not a variant-effect predictor, not a structure predictor and not a clinical tool.
• Not for clinical or diagnostic use. Outputs are unverified generated text.
• Inherits the base model's biases as well as any bias in the Gene Ontology annotations, which are unevenly distributed across genes — well-studied genes are described far more richly than the rest.
• A fluent answer is not a correct one. The biomedical base model can produce plausible gene-function prose with no support from the supplied embeddings.
• Requires embeddings from specific encoders at specific pooling. Substituting another protein language model needs a retrained projector.

License

This repository contains no Llama-3 weights — only the LoRA deltas and the projectors trained here. Using the model still requires ContactDoctor/Bio-Medical-Llama-3-8B, which you obtain from its own repository under its own terms, so the Llama 3 Community License governs your use of the combination. Whether LoRA deltas are themselves a derivative work of the base model is unsettled; distributing adapters separately is nonetheless the established practice on the Hub.

Built with Meta Llama 3.

Citation

If you use this model, please cite Genolator:

@article{danner2025genolator,
  title        = {Genolator: A Multimodal Large Language Model Fusing Natural Language,
                  Genomic, and Structural Tokens for Protein Function Interpretation},
  author       = {Danner, Martin and Islam, Tanhim and Begemann, Matthias and
                  Kraft, Florian and Elbracht, Miriam and Kurth, Ingo and Krause, Jeremias},
  journal      = {bioRxiv},
  year         = {2025},
  doi          = {10.1101/2025.11.14.688396},
  url          = {https://doi.org/10.1101/2025.11.14.688396},
  publisher    = {Cold Spring Harbor Laboratory},
  elocationid  = {2025.11.14.688396},
}

This is a preprint. The reference will be updated when the peer-reviewed version appears.

and the base model it builds on:

@misc{ContactDoctor_Bio-Medical-Llama-3-8B,
  author       = {ContactDoctor},
  title        = {ContactDoctor-Bio-Medical: A High-Performance Biomedical Language Model},
  year         = {2024},
  howpublished = {https://huggingface.co/ContactDoctor/Bio-Medical-Llama-3-8B},
}

Contact

Martin Danner — model creator and person responsible for this model.

• Center for Human Genetics and Genomic Medicine — [email protected]
• scieneers GmbH — [email protected]

Questions about the model, corrections and reports of problematic content are welcome at either address.