Avalon-2B-GGUF

<div align="center">

AVALON-2B GGUF

Quantized for On-Device Deployment

Full Model | Paper | GitHub

</div>

Overview

This is the GGUF quantized version of AVALON-2B, the first sub-3B self-reflective language model. Optimized for on-device inference with llama.cpp, Ollama, LM Studio, and other GGUF-compatible runtimes.

Available Files

Performance Benchmarks

Tested on Apple Silicon with llama.cpp:

Quality Retention

Usage

Ollama (Easiest)

# Create Modelfile
cat > Modelfile << 'EOF'
FROM ./avalon-2b-q4km.gguf

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>
"""

PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.7

SYSTEM """You are AVALON, a self-reflective AI assistant. Generate [Retrieval] for time-sensitive questions and [No Retrieval] for factual knowledge. End responses with [Utility:X]."""
EOF

# Create and run
ollama create avalon-2b -f Modelfile
ollama run avalon-2b "What is quantum computing?"

llama.cpp

# Download
wget https://huggingface.co/nuroai/Avalon-2B-GGUF/resolve/main/avalon-2b-q4km.gguf

# Run inference
./llama-cli -m avalon-2b-q4km.gguf \
    -p "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n" \
    -n 128 --temp 0.7

LM Studio

1. Download avalon-2b-q4km.gguf
2. Open LM Studio → Local Models → Import
3. Select the GGUF file
4. Configure chat template (Qwen/ChatML format)
5. Start chatting!

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="avalon-2b-q4km.gguf",
    n_ctx=4096,
    n_threads=8,
)

output = llm(
    "<|im_start|>user\nWhat is AI?<|im_end|>\n<|im_start|>assistant\n",
    max_tokens=256,
    temperature=0.7,
    stop=["<|im_end|>"]
)
print(output["choices"][0]["text"])

Self-RAG Tokens

AVALON generates special reflection tokens:

Example Output:

User: What is the capital of Japan?
Assistant: [No Retrieval]Tokyo is the capital of Japan.[Utility:5]

User: What's the current Bitcoin price?
Assistant: [Retrieval]I need current market data to answer this...[Utility:4]

Hardware Requirements

Authors

• Akhil Ponnada - [email protected]
• Naga Sri Arvapalli - [email protected]

License

Apache 2.0 - Same as the full model.

Citation

@article{ponnada2026avalon,
  title={AVALON-2B: The First Sub-3B Self-Reflective Language Model},
  author={Ponnada, Akhil and Arvapalli, Naga Sri},
  journal={arXiv preprint},
  year={2026}
}

Links

• Full Model (nuroai/Avalon-2B)
• GitHub Repository
• Nuro AI Labs