PhoneLLM-alpha-1-Q4_K_M

Pipecat PhoneLLM Alpha 1 — Q4_K_M GGUF

Quantization details

This is a Q4_K_M quantization of PhoneLLM Alpha 1 using llama.cpp's imatrix-guided quantization:

• Method: llama-quantize --imatrix (PPL-calibrated mixed precision)
• Calibration data: C4 English subset, 289 samples (~101k word-tokens)
• Perplexity on calibration set: 13.77 ± 0.15
• Tensor mapping (Q4_K_M scheme):
  • Routed expert FFN weights → q4_k
  • Attention Q/K projections → q4_k
  • Embeddings, output head, attention V → q6_k
  • Layer norms, router/gate → f16
• Reference: matches the ggml-org reference quantization size (22.4 GB for base model; this finetune is 22.83 GB due to slightly different tensor distributions post-training)

Running with llama.cpp

Requirements

• Minimum VRAM: ~25 GB combined GPU memory (e.g., RTX 4090 + RTX 3080, or single A100/H100)
• RAM: 64 GB+ recommended (for model loading buffer; --no-mmap allocates full weights to RAM at load time)
• llama.cpp version: b10673 or later (required for NemotronH hybrid Mamba architecture support)

Example: dual-GPU server (RTX 4090 + RTX 3080)

llama-server \
  -m phonellm-alpha-1-q4_k_m.gguf \
  --n-gpu-layers all \
  --split-mode layer \
  --tensor-split 7,3 \
  --main-gpu 0 \
  --ctx-size 262144 \
  -n 8192 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --cache-ram 4096 \
  --no-context-shift \
  --jinja \
  --temp 0 \
  --threads 10 \
  --no-mmap

Parameter notes:

Single-GPU (24GB+ VRAM)

llama-server \
  -m phonellm-alpha-1-q4_k_m.gguf \
  --n-gpu-layers all \
  --ctx-size 131072 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --jinja \
  --temp 0 \
  --no-mmap

CPU + GPU hybrid (limited VRAM)

For setups where the model doesn't fully fit in VRAM, offload routed experts to system RAM:

llama-server \
  -m phonellm-alpha-1-q4_k_m.gguf \
  --n-gpu-layers all \
  --cpu-moe \
  --ctx-size 32768 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --jinja \
  --temp 0

This keeps attention + shared experts on GPU and streams active routed experts over PCIe from RAM. Requires fast DDR4/DDR5 (64GB+). See the llama.cpp MoE offload guide for tuning.

About PhoneLLM Alpha 1

The Pipecat team is pleased to announce the release of PhoneLLM Alpha 1, an open-weights model for voice agent use cases.

This release is the result of our ongoing work training small, open-weights LLMs for low-latency and multi-turn agentic workloads.

When paired with transcription and text-to-speech models through a framework like Pipecat, PhoneLLM can handle incoming calls for financial services, healthcare, retail, and hospitality customer service, and perform common outbound calling agent tasks.

PhoneLLM runs at a fraction of the cost and latency of larger, general-purpose models, while delivering comparable performance for specific use cases. For example, PhoneLLM performs on par with GPT 5.6 Terra, but 94% cheaper and with 1,300ms faster P95 time-to-first-token.

PhoneLLM is an open model, so you can run it on your own infrastructure. The model is released under the BSD license, with no commercial restrictions.

Model specs

PhoneLLM Alpha 1 is a full-parameter fine-tune of NVIDIA's Nemotron 3 Nano 30B-A3B model, trained using the NVIDIA NeMo framework.

Like Nemotron Nano, PhoneLLM is a mixture-of-experts (MoE) model, with 3.5B active parameters, allowing for high-speed inference at low cost.

Important: set temperature to 0 and disable thinking. These two settings align with how the model was trained.

License

PhoneLLM Alpha 1 is released under the BSD 2-Clause License.

PhoneLLM is a derivative work of NVIDIA Nemotron 3 Nano 30B-A3B, which is licensed under the NVIDIA Nemotron Open Model License. Under Section 3 (Redistribution) of that license, if you redistribute this model or your own derivatives of it, you must (a) include a copy of the NVIDIA Nemotron Open Model License, and (b) retain the NVIDIA copyright and attribution notices. Our BSD 2-Clause terms apply to our modifications and to the model as a whole, as Section 3 permits; the NVIDIA license continues to apply to the underlying Nemotron work. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.