cogito-14b-voice-GGUF

cogito-14b-voice-GGUF

Voice-optimized GGUF quantizations of deepcogito/cogito-v1-preview-qwen-14B, specifically calibrated for Home Assistant voice assistant tool-calling workloads.

What Makes This Different

Standard GGUF quantizations use generic calibration data (wiki text, general prose). These quantizations use an importance matrix generated from actual voice assistant interactions — system prompts, 54 tool definitions, voice commands, and structured tool-call responses.

This means the quantizer preserves precision on the weights that matter most for:

• Structured JSON tool-call generation
• Entity name matching from natural speech
• Concurrent tool definition handling (50+ tools)
• Short, concise voice responses

The result: smaller models that maintain tool-calling accuracy compared to generic quants of the same size.

Available Quantizations

* Tool accuracy measured with OpenAI-compatible tools API, 20 voice commands, 54 exposed entities, strict matching. ** Average latency on NVIDIA Blackwell (DGX Spark GB10). RTX 4080 SUPER latency will differ.

Recommended: IQ4_XS (7.6 GB)

The IQ4_XS quantization is the recommended choice — 16% smaller than a standard Q4_K_M (9.0 GB → 7.6 GB) while maintaining reliable tool-calling accuracy in production with 50+ concurrent tool definitions.

Important: IQ3_XS (6.0 GB) passes synthetic benchmarks but is too aggressive for production use with 50+ exposed entities. In real-world Home Assistant testing, IQ3_XS produced wrong tool calls (e.g., calling HassVacuumStart instead of HassTurnOff) and empty responses. Use IQ4_XS or higher for reliable voice assistant operation.

Why cogito:14b for Voice

Several models were tested for Home Assistant voice pipelines. cogito:14b was chosen because it:

1. Does NOT generate thinking/reasoning tokens — Models that produce <think> tokens (qwen3, etc.) cause empty responses via OpenAI-compatible APIs, resulting in the Voice PE showing a red error ring
2. Handles 50+ concurrent tool definitions without degradation
3. Produces clean, structured tool-call JSON in OpenAI format
4. Reasonable latency (~4s on RTX 4080 SUPER with standard Q4_K_M, ~1s on Blackwell)

Voice Pipeline Architecture

These models are designed for the following Home Assistant voice pipeline:

Voice PE (ESP32)         Your Server                    Home Assistant
┌─────────────┐    ┌─────────────────────────┐    ┌─────────────────┐
│ openWakeWord │───>│ Whisper (STT)           │    │ Voice Pipeline   │
│ "Hey Jarvis" │    │ Piper (TTS)             │<──>│ local_openai     │
│              │    │ Ollama/llama.cpp (LLM)  │    │ integration      │
└─────────────┘    └─────────────────────────┘    └─────────────────┘

The LLM receives:

• System prompt (~500-1000 tokens) — personality, camera/device instructions
• Tool definitions (~3,000-5,000 tokens) — HassTurnOn, HassTurnOff, HassLightSet, HassVacuumStart, etc.
• User message (5-20 tokens) — transcribed speech
• Total context per request: ~4,000-6,000 tokens

How to Use

With Ollama

# Create a Modelfile
cat > Modelfile << 'EOF'
FROM ./cogito-14b-voice-IQ3_XS.gguf
PARAMETER temperature 0.1
PARAMETER num_ctx 4096
EOF

# Create and run the model
ollama create cogito-voice -f Modelfile
ollama run cogito-voice

With llama.cpp

llama-server \
  -m cogito-14b-voice-IQ3_XS.gguf \
  -ngl 99 \
  -c 4096 \
  -fa on \
  --port 8080

With Home Assistant (via LiteLLM)

Add to your LiteLLM config.yaml:

- model_name: voice/cogito-14b
  litellm_params:
    model: openai/cogito-voice
    api_base: http://your-ollama-host:11434/v1
  model_info:
    supports_function_calling: true

Then configure the local_openai integration in HA to use voice/cogito-14b.

Importance Matrix Methodology

The importance matrix was generated using llama-imatrix from llama.cpp with a custom calibration dataset containing:

• 50 voice interaction examples covering all tool types:
  • Light control (on/off, brightness, color, temperature)
  • Camera script activation (8 cameras × 4 TVs)
  • Vacuum robot control (start, return to base)
  • Shopping list management (add, remove, complete)
  • Scene activation
  • DateTime queries
  • Broadcast messages
• Full system prompt with personality and camera routing instructions
• 54 entity definitions matching a real Home Assistant deployment
• Proper chat template format (<|im_start|> / <|im_end|> Qwen2 format)

The calibration data was processed through the F16 GGUF on an NVIDIA DGX Spark (Blackwell GB10, 128 GB unified memory) at 772 tokens/sec.

Benchmark Details

Test Setup

• Hardware: NVIDIA DGX Spark (Blackwell GB10, 128 GB LPDDR5x unified memory)
• Server: llama.cpp with flash attention, 4096 context, full GPU offload
• API: OpenAI-compatible /v1/chat/completions with tools parameter
• Temperature: 0.1
• Test set: 20 diverse voice commands across all tool types
• Evaluation: Strict matching (correct tool name AND correct primary argument)

Test Cases and Results (IQ3_XS)

* Uses entity_id format instead of friendly name — functionally correct, HA handles both. ** Uses HassTurnOff instead of HassTurnOn for stop scripts — minor tool name mismatch.

Comparison with Baseline

Base Model

• Model: deepcogito/cogito-v1-preview-qwen-14B
• Architecture: Qwen2 (48 layers, 40 query heads, 8 KV heads, head_dim=128)
• Parameters: 14.8B
• License: Apache 2.0

Quantization Hardware

All quantization was performed on an NVIDIA DGX Spark personal AI supercomputer:

• NVIDIA GB10 Grace Blackwell Superchip
• 128 GB LPDDR5x unified coherent memory
• CUDA 13.0, llama.cpp built from source with Blackwell support

Acknowledgments

• Deep Cogito for the excellent cogito base model
• llama.cpp for quantization tools and importance matrix support
• Home Assistant and the local_openai integration
• TurboQuant research (ICLR 2026) which inspired this voice-optimized quantization approach