Zamba2-7B-Instruct-v2-GGUF

Zamba2 7B Instruct v2 - GGUF

GGUF conversions of Zyphra/Zamba2-7B-instruct-v2 for use with llama.cpp.

Architecture

Zamba2 is a hybrid Mamba-2 + shared Transformer architecture by Zyphra.

• 7B parameters, 81 layers (13 hybrid attention blocks, 2 shared transformers cycling even/odd)
• Hidden size: 3584, attention hidden: 7168
• SSM: d_state=64, d_conv=4, ngroups=2
• Requires llama.cpp with Zamba2 support (PR #21412)

Available Quantizations

Sample Output (Q4_0)

Q: What is the capital of France and why is it significant?

A: The capital of France is Paris, a city that holds significant historical, cultural, and economic importance. Paris, often referred to as the "City of Light" or "City of Love," is not only the heart of France but also a global hub for art, fashion, gastronomy, and culture. Paris is renowned for its iconic landmarks such as the Eiffel Tower, the Louvre Museum, and Notre-Dame Cathedral, which attract millions of visitors each year.

Usage

# Build llama.cpp with Zamba2 support
git clone https://github.com/echo313unfolding/llama.cpp -b zamba2-support
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build -j

# Run
./build/bin/llama-cli -m zamba2-7b-instruct-v2-q4_0.gguf \
  -p "<|im_start|>user\nWhat is quantum computing?<|im_end|>\n<|im_start|>assistant\n" \
  -n 256 -ngl 999 -e --no-conversation

Conversion Details

Converted from HF safetensors using a custom Zamba2-to-GGUF converter:

• Mamba-2 SSM layers: A_log to A conversion, conv1d squeeze, dt projection
• 2 shared transformer blocks with per-layer LoRA unfolding (W_eff = W_shared + B @ A)
• Per-layer n_head_kv array (0 for Mamba layers, 32 for attention layers)
• Group norm reshape for ngroups=2 (GGML column-major convention)
• BPE tokenizer (v2 format)
• F32 master, quantized with llama-quantize

Credits

• Architecture: Zyphra (Zamba2)
• GGUF Conversion: Echo Labs
• llama.cpp Zamba2 support: PR #21412