phonellm-alpha-1-GGUF

phonellm-alpha-1 — GGUF

Unofficial GGUF quantizations of pipecat-ai/phonellm-alpha-1.

⚠️ Disclaimer: This is a community conversion. This repository is not affiliated with, endorsed by, or maintained by Pipecat AI or NVIDIA. All credit for the model goes to the original authors — please see the official model card for full details, intended use, and limitations.

About the model

PhoneLLM Alpha 1 is a voice-agent / phone-conversation model with tool-use and function-calling support, fine-tuned by Pipecat AI from NVIDIA-Nemotron-3-Nano-30B-A3B (a Nemotron-H hybrid Mamba–attention mixture-of-experts model, 30B total / ~3B active parameters).

Conversion details

• Converted with llama.cpp release b10672 (convert_hf_to_gguf.py → F16, then llama-quantize)
• Source weights: BF16 safetensors from the official repository (revision 8e76aaa)

Files

⚠️ Important: these quants are larger than their names suggest

This architecture's weight tensors have column counts of 1856, 2688 and 3712 — none of which are divisible by 256, the superblock size that llama.cpp's K-quants require. As a result llama-quantize reports 134 of 401 tensor(s) required fallback quantization and substitutes the legacy block-32 types defined in its fallback table (src/llama-quant.cpp): Q3_K → Q4_0, Q4_K → Q5_0, Q6_K → Q8_0.

Only the 29 tensors with a 4096 column count keep their requested K-quant type. The actual composition by size is:

Practical implications:

• The files are larger than you would expect from these quant names on a conventional dense model. Q3_K_M is ~19.8 GB, not the ~13 GB a 30B Q3_K_M would normally be.
• Quality should be better than the names imply, since the fallbacks are higher-precision than the requested types.
• No Q6_K is published. Its fallback type is Q8_0, so a Q6_K build came out 99.1% Q8_0 tensors at 33.5 GB — within 0.1 GB of the Q8_0 file and marginally worse. Use Q8_0 instead.

Usage

Requires a recent llama.cpp build (b10672 or newer) with Nemotron-H support:

llama-cli -m phonellm-alpha-1-Q4_K_M.gguf -cnv

Or serve an OpenAI-compatible API:

llama-server -m phonellm-alpha-1-Q4_K_M.gguf

License

PhoneLLM Alpha 1 is released under the BSD 2-Clause License (see LICENSE).

PhoneLLM is a derivative work of NVIDIA Nemotron 3 Nano 30B-A3B, which is licensed under the NVIDIA Nemotron Open Model License (see LICENSE_NVIDIA.txt). Per Section 3 (Redistribution) of that license, this repository includes a copy of the NVIDIA Nemotron Open Model License and retains the NVIDIA copyright and attribution notices. The BSD 2-Clause terms apply to Pipecat AI's modifications and to the model as a whole, as Section 3 permits; the NVIDIA license continues to apply to the underlying Nemotron work. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.