Carbon-3B-GGUF

Carbon-3B-GGUF

GGUF quantizations of HuggingFaceBio/Carbon-3B — a generative DNA foundation model — for efficient inference with llama.cpp.

Carbon-3B is a 3B-parameter autoregressive genomic model that uses a hybrid tokenizer (6-mer chunks for DNA inside <dna> tags, BPE for English text). It matches the performance of Evo2-7B while running over 250× faster at inference.

📦 Available Quantizations

⚙️ Requirements

You must use a recent build of llama.cpp that includes PR #23410 — this added support for Carbon's custom HybridDNATokenizer. Any build from master after that PR works.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build -j

🚀 Usage

Download a quant

huggingface-cli download pankajpandey-dev/Carbon-3B-GGUF carbon-3b-Q4_K_M.gguf --local-dir .

DNA sequence continuation

Wrap DNA sequences in <dna> tags so the tokenizer switches into 6-mer mode:

./build/bin/llama-completion -m carbon-3b-Q4_K_M.gguf \
    -p '<dna>ATGCGCTAGCTACGATCGATCGTAGCTAGCTAGCTAGCTACG' \
    -n 64 --temp 0 -no-cnv

Conditional generation with metadata tags

./build/bin/llama-completion -m carbon-3b-Q4_K_M.gguf \
    -p '<vertebrate_mammalian><protein_coding_region><dna>ATGCGCTAG' \
    -n 64 --temp 0 -no-cnv

Long-context generation (up to 131k 6-mer tokens ≈ 786 kbp)

./build/bin/llama-completion -m carbon-3b-Q4_K_M.gguf \
    -c 65536 --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768 \
    -p '<dna>...' -n 64 --temp 0 -no-cnv

⚠️ Important Notes

• DNA tag is mandatory. Raw DNA without the <dna> tag will be BPE-tokenized as English text, producing meaningless results.
• 6-mer constraint. DNA sequences must contain only [ATCG] characters. Any other letters trigger an <oov> token.
• fns revision is not supported. This GGUF is converted from the main revision only. The fns branch uses a factorized nucleotide supervision head that requires the Python transformers path.
• Carbon-500M (GGUF here) can be used as a draft model for speculative decoding to speed up Carbon-3B inference at no quality loss.

📊 Quantization Method

These quants were produced from the original bf16 HuggingFaceBio/Carbon-3B safetensors using convert_hf_to_gguf.py followed by llama-quantize from llama.cpp (build b9330). Standard K-quants and Q8_0 were used — no imatrix calibration was applied, as standard text calibration corpora are not appropriate for DNA-pretrained models.

📜 License

Apache 2.0 — inherited from the source model.

🙏 Credits

• Original model: HuggingFaceBio/Carbon-3B — full credit to the Hugging Face biology team for designing, training, and open-sourcing Carbon.
• GGUF conversion & quantization: @pankajpandey-dev
• Reference: Original GGUF release HuggingFaceBio/Carbon-3B-GGUF (bf16 only). This repository complements it with smaller K-quants for CPU/edge inference.

🔗 Related

• 🧬 Carbon model family collection
• 📄 Carbon technical report
• 🎬 Interactive demo
• 💻 Carbon GitHub repo

---

Part of my GGUF Quantizations collection — making open models accessible across hardware tiers.