Mallow-A1-gguf

🌸 Mallow-A1 GGUF

📊 Complete Benchmark Report: View PNG Graph | Download PDF Report

---

GGUF quantized versions of Maazwaheed/Mallow-A1.

📐 Model Architecture

Key Features:

• Grouped Query Attention (GQA): 15 query heads → 5 KV heads (3:1 ratio), reducing KV cache memory by 67%
• Chat Template: Custom <|im_start|> role markers for structured conversations
• BOS: 1 | EOS: 2 | PAD: 2

---

🏆 Quantization Benchmark Results

All 15 variants were tested with the prompt: "What is the capital of France? The capital of France is"

Test Config: CPU-only, 2 threads, 256 ctx, temp=0.1, 16 new tokens

Quality Score: +10 for "Paris" +5 for "capital" +3 for coherent output = 15/18 max (all models passed!)

---

📏 Size Comparison Chart

F16    ████████████████████████████████████████  691.9 MB  (baseline)
Q8_0   ██████████████████████                    368.5 MB  (-47%)
Q6_K   █████████████████████                     350.3 MB  (-49%)
Q5_K_M ████████████████                          276.5 MB  (-60%)
Q5_1   ███████████████                           274.8 MB  (-60%)
Q5_K_S ███████████████                           270.1 MB  (-61%)
Q5_0   ██████████████                            256.0 MB  (-63%)
Q4_K_M ██████████████                            258.1 MB  (-63%)  ← RECOMMENDED
Q4_K_S █████████████                             247.9 MB  (-64%)
Q4_1   ████████████                              237.3 MB  (-66%)
Q3_K_L ████████████                              234.9 MB  (-66%)
Q3_K_M ███████████                               223.8 MB  (-68%)
Q4_0   ███████████                               218.5 MB  (-68%)
Q3_K_S ███████████                               208.5 MB  (-70%)
Q2_K   ███████████                               208.5 MB  (-70%)

---

🎯 Recommendations by Use Case

---

📥 Quick Start

llama.cpp

# Best balance (recommended)
./llama-cli -m mallow_a1_q4_k_m.gguf -p "Hello, my name is" --temp 0.7

# Maximum quality
./llama-cli -m mallow_a1_q8_0.gguf -p "Hello, my name is" --temp 0.7

# Fastest inference
./llama-cli -m mallow_a1_q4_0.gguf -p "Hello, my name is" --temp 0.7

Ollama

Create a Modelfile:

FROM ./mallow_a1_q4_k_m.gguf

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}}
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}}
{{ end }}<|im_start|>assistant
"""

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER stop <|im_end|>

Then run:

ollama create mallow-a1 -f Modelfile
ollama run mallow-a1

LM Studio

1. Download any .gguf file from this repo
2. Open LM Studio → My Models → Add Model
3. Select the downloaded file and load

text-generation-webui

1. Download a .gguf file
2. Place in models/ folder
3. Select from the Model dropdown

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="mallow_a1_q4_k_m.gguf",
    n_ctx=2048,
    verbose=False
)

output = llm(
    "What is the capital of France?",
    max_tokens=32,
    temperature=0.7,
    top_p=0.9
)
print(output["choices"][0]["text"])

---

🔧 Generation Config

---

📊 Quality Tier Analysis

Based on llama.cpp community benchmarks for models of similar scale (~350M parameters):

---

🛠️ Technical Details

Quantization Pipeline

• Tool: llama.cpp (commit 3fc4e10)
• Build: cmake -B build -DLLAMA_NATIVE=OFF -DLLAMA_AVX2=ON -DLLAMA_FMA=ON -DLLAMA_F16C=ON -DCMAKE_BUILD_TYPE=Release
• Converter: convert_hf_to_gguf.py (HF → F16 GGUF)
• Quantizer: llama-quantize (F16 → all types, parallel 6 workers)
• Total Time: 127.4s for all 14 quantizations
• IQ Types: Not supported (requires IQK kernel build)

Tensor Breakdown (F16)

• Total Tensors: 290 (32 layers × 9 + embeddings + norms)
• Embedding: 960 × 49,152 = ~94 MB
• Per-Layer: attn_q/output (960×960), attn_k/v (960×320), ffn_gate/up (960×2560), ffn_down (2560×960)
• Norms: F32, 960 elements each

---

📄 License

Apache 2.0 (same as base model)

🙏 Credits

• Base model: Maazwaheed/Mallow-A1
• Quantized with llama.cpp
• Benchmark report generated 2026-06-26