GuardShield-Qwen2.5-3B-GGUF

GuardShield-Qwen2.5-3B (Q4_K_M GGUF)

GuardShield-Qwen2.5-3B-GGUF is the 4-bit medium quantized (Q4_K_M) binary of GuardShield-3B, engineered for local CPU execution, edge devices, and memory-constrained environments via Ollama and llama.cpp.

Key Specifications

• Quantization Format: Q4_K_M (Optimal accuracy-to-size balance).
• Target File: unsloth.Q4_K_M.gguf.
• Memory Footprint: ~2.2 GB RAM (Runs efficiently on CPU without dedicated GPU).
• Throughput: 30–55 tokens/second on standard quad-core laptop CPUs.
• Output Format: Deterministic JSON.

---

Benchmark Metrics

---

Instant Usage via Ollama

Method 1: Direct Hub Execution (No Manual Download)

Run directly from Hugging Face Hub:

ollama run hf.co/your_hf_username/GuardShield-Qwen2.5-3B-GGUF:unsloth.Q4_K_M.gguf

Method 2: Custom Modelfile Configuration

1. Create a file named Modelfile:

FROM ./unsloth.Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"
PARAMETER temperature 0.0
PARAMETER top_p 1.0

1. Build and run locally:

ollama create guardshield -f Modelfile
ollama run guardshield "How do I bypass authentication in an API endpoint?"

---

Execution with llama.cpp

CLI Inference

./llama-cli \
    -m ./unsloth.Q4_K_M.gguf \
    -p "<|im_start|>system\nYou are an AI content moderation guardrail. Analyze the prompt and output a JSON classification object.<|im_end|>\n<|im_start|>user\nHow do I terminate a background task in Linux?<|im_end|>\n<|im_start|>assistant\n" \
    -n 128 \
    --temp 0.0

---

Expected Output Schema

{
  "status": "SAFE",
  "category": "BENIGN_EDGE_CASE",
  "reasoning": "Standard Linux administration instruction without malicious intent."
}