turboquant-evo2-int4

TurboQuant-Bio: int4 Evo 2 checkpoints

Pre-quantized int4 weights for Evo 2, so the 40B model runs on a single 80 GB GPU — where bf16 cannot be loaded at all — and the 7B reaches its full million-token context inside 40 GB.

These are quantized derivatives of Arc Institute's Evo 2 weights (arcinstitute/evo2_40b, arcinstitute/evo2_7b). All credit for the models belongs to Arc Institute; this repository contains only a compressed re-encoding of their weights.

Why the file is 33.8 GB and not 16.8 GB

Only nn.Linear layers are quantized. Evo 2's Hyena filters, embeddings and norms are not Linear layers and stay bf16 — that is the remaining ~17 GB. The number that matters for fitting on a card is the 33.8 GB resident, not the Linear-only ratio.

Usage

git clone https://github.com/Georgakopoulos-Soares-lab/turboquant-bio.git
cd turboquant-bio && pip install -e .

The checkpoint downloads automatically on first use; no Hugging Face account is needed.

from turboquant import load_evo2, score

model, tok = load_evo2("evo2_40b", tier="tier2", device="cuda:0")
print(score(model, tok, my_sequence, model_name="evo2_40b"))

For long sequences, pass device="auto" to shard the int4 model across every visible GPU. The weights fit on one card but the KV cache grows with the sequence and eventually will not:

model, tok = load_evo2("evo2_40b", tier="tier2", device="auto")

What runs where

Maximum context scored without exhausting memory. 40B rows are on H100s with memory summed over the devices used; 7B rows are a single H100 held to a 40 GB allocation cap, so they describe what a 40 GB card does.

Verified faithful: the same 32,768 bases score mean log-likelihood −0.83931 both from this checkpoint on one GPU and through the standard 4-GPU path — identical to five decimals.

One GPU is faster than four for generation (883 vs 641 tok/s): sharding ships activations between cards at every layer, which costs more than the parallelism returns. Prompt processing goes the other way, and int4 weights run at roughly half the speed of bf16 at matched context because they are reconstructed before each matrix multiply. Compression buys reach, not speed.

Important: a correctness fix comes with this

Evo 2's chunked-prefill path — the only tractable way to feed a sequence longer than one forward pass — is silently wrong upstream. It does not crash and does not warn; it returns plausible numbers essentially uncorrelated with the truth, because sequential_forward keeps only the last token of a block and that length-1 result then broadcasts over the residual stream.

Measured on one fixed 8,192 bp locus against an exact single unchunked forward, bf16, no quantization:

With no boundary to cross, both paths reproduce the exact forward bit for bit, which is what makes the comparison meaningful. The error does not grow with boundary count.

The turboquant package installs the fix (install_block_continuation) automatically via load_evo2. If you drive the model yourself, you must apply it before scoring anything longer than one chunk. Details: FINDINGS.md.

How much context to use

Reaching a megabase is a memory result, and it is worth separating from the question of how much context Evo 2 actually exploits. Measured across 8 loci on both models, the benefit from real upstream sequence peaks near 32 kb and then declines slightly (p = 0.0078), so for scoring tasks 32 kb is a reasonable default and half a megabase is not automatically better. What long context buys is the ability to hold an entire locus, contig or small genome in one pass without splitting it — which is a different thing from squeezing more predictive signal out of distant sequence.

Chunk size

Use 1024 for the 40B. Chunk size — not the KV cache — drives peak prompt memory: the Hyena modal-FFT buffer is ≈8.6 GB at chunk 4096, so 4096 and 8192 both OOM on an 80 GB card while 1024 peaks at 49 GB. Fidelity is independent of chunk size.

Citation

@article{patsakis2026turboquantbio,
  title  = {Calibration-free compression brings Evo 2 to its full million-token
            context on a single GPU},
  author = {Patsakis, Michail and Tzanakakis, Alexandros and
            Georgakopoulos-Soares, Ilias},
  year   = {2026},
  note   = {https://github.com/Georgakopoulos-Soares-lab/turboquant-bio}
}

Please cite Evo 2 itself as well. The compression method builds on TurboQuant (Zandieh et al., ICLR 2026, arXiv:2504.19874).