phonellm-alpha-1-nvfp4

Pipecat PhoneLLM Alpha 1 NVFP4

This is the official Pipecat NVFP4 quantization of PhoneLLM Alpha 1. It preserves the benchmarked PhoneLLM weights and targets native NVFP4 execution on NVIDIA Blackwell hardware.

NVFP4 quick facts

Quality results

All figures below are means over ten complete PhoneBench runs at temperature zero with thinking disabled.

The recommended production recipe therefore uses NVFP4 weights with a BF16 KV cache. At the tested allocation, BF16 KV reduced available KV-token capacity by approximately 40.2% compared with FP8. The checkpoint retains NVIDIA-compatible FP8 KV metadata, so pass --kv-cache-dtype bfloat16 explicitly when maximizing benchmark quality.

Required generation settings

Always use numeric temperature=0 and disable thinking with chat_template_kwargs={"enable_thinking": false}. The unchanged generation_config.json contains do_sample: true; the explicit request value and the server override below govern the tested recipe.

Tested B200 vLLM recipe

The tested runtime is vLLM 0.27.1 on NVIDIA B200. Download the snapshot so the artifact-local reasoning-parser plugin has a stable path:

MODEL_DIR=$(hf download pipecat-ai/phonellm-alpha-1-nvfp4)

vllm serve "$MODEL_DIR" \
  --served-model-name pipecat-ai/phonellm-alpha-1-nvfp4 \
  --trust-remote-code \
  --load-format safetensors \
  --tensor-parallel-size 1 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching \
  --mamba-cache-mode align \
  --async-scheduling \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --kv-cache-dtype bfloat16 \
  --mamba-ssm-cache-dtype float32 \
  --attention-backend FLASHINFER \
  --moe-backend flashinfer_cutlass \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nano_v3 \
  --reasoning-parser-plugin "$MODEL_DIR/nano_v3_reasoning_parser.py" \
  --default-chat-template-kwargs '{"enable_thinking":false}' \
  --override-generation-config '{"temperature":0}'

This recipe exercises ModelOpt FP4 loading, FlashInferCuteDslNvFp4LinearKernel, FLASHINFER_CUTLASS NVFP4 MoE, FlashInfer attention, CUDA graphs, async scheduling, prefix caching, and Mamba align. Generic vLLM installations or non-Blackwell GPUs have not been validated for this checkpoint.

The architecture config supports longer contexts. PhoneBench was exercised at 32,768 tokens and 32 sequences; the Hub-origin release smoke uses the production-aligned 65,536-token recipe above.

Quantization and calibration

The checkpoint was quantized with ModelOpt 0.46.0rc1 using 1,000 deterministic, PhoneBench-disjoint PhoneData rows spanning 273 conversations and all 692 observed strata. The 50 longest eligible rows were forced into the sample. The lm_head, selected sensitive attention/Mamba projections, and Mamba convolution modules remain unquantized; the complete exclusion list is in hf_quant_config.json.

The global safetensor index contains 24,147 tensors and 5,968 tensors of each NVFP4 scale type (weight_scale, weight_scale_2, and input_scale). See RELEASE-MANIFEST.json for immutable source identities and per-file hashes.

License and attribution

PhoneLLM Alpha 1 NVFP4 is released under the BSD 2-Clause License. It is a quantized derivative of PhoneLLM Alpha 1 and the underlying NVIDIA Nemotron work; the NVIDIA Nemotron Open Model License and attribution notices are included unchanged.

About the BF16 PhoneLLM Alpha 1 release

The Pipecat team is pleased to announce the release of PhoneLLM Alpha 1, an open-weights model for voice agent use cases.

This release is the result of our ongoing work training small, open-weights LLMs for low-latency and multi-turn agentic workloads.

When paired with transcription and text-to-speech models through a framework like Pipecat, PhoneLLM can handle incoming calls for financial services, healthcare, retail, and hospitality customer service, and perform common outbound calling agent tasks.

PhoneLLM runs at a fraction of the cost and latency of larger, general-purpose models, while delivering comparable performance for specific use cases. For example, PhoneLLM performs on par with GPT 5.6 Terra, but 94% cheaper and with 1,300ms faster P95 time-to-first-token.

PhoneLLM is an open model, so you can run it on your own infrastructure. The model is released under the BSD license, with no commercial restrictions.

We are also announcing PhoneBench v1, a benchmark which evaluates LLMs based on their suitability for phone agent use cases. In addition to accuracy and speaking style, we measure model latency and estimate per-minute runtime cost.

The unique combination of benchmark accuracy, low latency, and low cost makes PhoneLLM one of the most compelling options for building a voice agent.

Model specs

PhoneLLM Alpha 1 is a full-parameter fine-tune of NVIDIA’s Nemotron 3 Nano 30B-A3B model, trained using the NVIDIA NeMo framework.

Like Nemotron Nano, PhoneLLM is a mixture-of-experts (MoE) model, with 3.5B active parameters, allowing for high-speed inference at low cost.

In our PhoneBench benchmark, PhoneLLM achieves accuracy comparable to or better than most of the models commonly used for production voice agents, at lower latency and lower cost.

Why a model specifically for voice agents?

In our work creating voice agents using Pipecat, we’ve found that there is a gap in the LLM space for models which (1) are low latency, and (2) invoke tools accurately.

The latest frontier models are optimized for use when thinking tokens are enabled. When a model reasons in this way, there is a long delay between the user’s query and the agent’s response.

We’ve also found that many models, large and small, struggle to accurately invoke tools in long, multi-turn conversations. This is especially true when thinking is disabled. LLMs will often say “Yes, I’ve booked that table for you” without actually doing it, which can lead to disastrous results and customer frustration.

PhoneLLM is specifically trained to call the right tools at the right time, without the need for thinking to be enabled.

More broadly, we’re seeing a shift in the industry towards small open-weights models, fine-tuned for specific purposes, which outperform general-purpose frontier models in accuracy, inference speed, cost, and data privacy.

The Pipecat team works directly with enterprise customers to train models for specific use cases, using production agent traces and proprietary data. PhoneLLM is an example of what’s possible today, building on a strong, efficient base model like Nemotron 3 Nano.

PhoneBench

When we train a new model, the first step is always to figure out how to evaluate it.

Voice agents, and LLM outputs in general, are difficult to evaluate, because the outputs are free-form and often not objectively verifiable. Speaking style, factual accuracy, and relevance to the user’s query are examples of things that can’t be checked programmatically.

Even the choice of tool calls has a subjective component: often benchmarks will expect a certain tool to be called with specific parameters at a certain time, but actually models have a high degree of flexibility about when and how they perform actions, and what information they look up.

To handle this subjectivity, PhoneBench uses a panel of LLM judges to grade model output, comparing the actions to high-quality reference samples. The judges are calibrated against human labels to ensure accuracy.

In this way, PhoneBench is able to measure detailed points of quality that would normally require human labeling: telephone speaking style, tool call accuracy, say/do consistency, factual grounding, conversation coherence, authentication and escalation discipline, and caller outcome.

The scenarios, tool lists, and system prompts in the benchmark are kept separate from the data we use to train models like PhoneLLM, so that PhoneLLM isn’t limited to the tasks it was trained on. And the benchmark verifies that the model generalizes across previously unseen scenarios, business use cases, tools lists, and prompts.

Latency

Accuracy is only one part of the equation. Building good agents is a full stack systems engineering challenge! For voice agents, response latency is a critical attribute.

In general, people expect responses to happen quite quickly in voice conversations and have very low tolerance for slow responses. “Voice-to-voice” latency needs to be around 1,500ms in most situations. (Ideally, even faster. But we have lots of empirical data about voice agent success, and people are happy talking to an agent with a P95 voice-to-voice latency of 1,500ms.)

This voice-to-voice latency includes network overhead, audio processing, application logic, and all the inference from STT, LLM, and TTS models. 1,500ms is a fairly tight latency budget. The P95 time-to-first-token for GPT 5.6 Terra running in fast mode is about 1,900ms. So just the LLM part of a voice agent built with Terra already exceeds our latency target.

Here’s a detailed breakdown of how the latency accumulates in a well-optimized voice agent that’s running on macOS and communicating via WebRTC with a voice agent server running in the cloud.

You can see that our LLM time-to-first-token target is 650ms.

There’s a trade-off between aiming for very fast time-to-first-token, and running a model cost-effectively. Serving more concurrent inference requests lowers cost, but increases latency.

A big advantage of open weights models is that we can optimize our inference stack for voice agent use cases. We can pick optimal points on the cost-latency curve. And, because the inference servers we use are open source, we can move the curve by writing new code. (We write latency-optimized inference code both for LLMs and for NVIDIA’s transcription models.)

We chose Nemotron 3 Nano as the base for PhoneLLM because the Nemotron 3 architecture scales very efficiently on modern NVIDIA hardware. We can pack quite a lot of concurrent generations on a single GPU while still maintaining excellent P95 latency.

We’ve worked with the team at Modal to develop optimized configurations for running PhoneLLM. More on that in the Running the model section, below.

Cost

Estimating LLM cost for conversational agents is tricky.

Agents are long-running interactions that take place across multiple turns. Most LLM API services charge by the token, with different rates for input tokens and output tokens, cache writes, cache reads, service tiers (for example OpenAI’s “fast” mode), regional data fencing, and zero data retention.

“Cost per minute” is the natural way to model costs for production voice agents. So it’s useful to convert from estimated token costs to average cost per minute. The formula for this conversion depends on the length of conversations, size of system prompts, typical model language and thinking token patterns, cache use statistics, and a number of other elements that vary both across use cases and between individual sessions.

It’s a bit easier to calculate cost per minute if you are running the model on your own infrastructure. You’ll need to know what your target concurrency is and make assumptions about utilization.

We’ve built a spreadsheet that shows the cost per minute for a number of LLMs, both APIs and self-hosted models, based on the inference patterns for each model in the PhoneBench benchmark.

Feel free to copy this sheet and modify or extend it. It’s quite complicated, but both Codex and Claude are now very good at digging into large spreadsheets, explaining how they work, and adding to them.

You can see some broad patterns in these numbers. Bigger models are generally more expensive to run. Gemini 3.6 Flash is an outlier, because even with thinking set to minimal it produces a lot of thinking tokens.

Self-hosting can be cheaper than using an API, but the large inference providers are good at what they do and operate at very large economies of scale. The path to saving money by self-hosting is model size arbitrage: use a smaller model that’s a perfect fit for your use case instead of a bigger, more general-purpose model.

Hosting a model cost-effectively requires utilizing each GPU cluster as much as possible, up to the point where performance degrades beyond our target metrics. For a voice agent, we want to pin each session to a single cluster so that caching is easier. And our most important metric is latency.

Here’s a summary of concurrency benchmarking sweeps for each of the models in the spreadsheet. We pick a “not to exceed” P95 time-to-first-answer-token target. Then we run simulations to find the maximum concurrency that stays below that target, for a specific agent workload.

Nemotron 3 Nano (and PhoneLLM, which is the same architecture) scales very well. Our max concurrency here is 44 processes on each NVIDIA B200. The way this sweep is constructed, that translates to 88 agent processes pinned to each B200 node.

On Modal, the base cost of a B200 is $6.2496 per hour. Region pinning adds a 1.5× multiplier, bringing the cost to $9.3744/hour. We target 70% utilization (divide by 0.70), so we have an effective cost of $13.392/hour, or $0.2232/minute.

Dividing $0.2232/minute by 88 concurrent agents equals a per-minute agent cost of $0.00025.

We’re able to achieve this high level of efficiency partly because Nemotron 3 Nano is a very efficient Mamba-Transformer Mixture of Experts architecture. But also because the team at Modal developed workload-specific optimizations for PhoneLLM running on their B200 infrastructure. Which brings us to …

Weights and deployment

The BF16 source weights are pipecat-ai/phonellm-alpha-1. This repository is the official Pipecat NVFP4 quantization for NVIDIA Blackwell GPUs. Use the tested command above, including temperature zero, thinking disabled, BF16 KV cache, and the native FlashInfer NVFP4 MoE backend.

Deploying on Modal

Modal AutoEndpoints support is currently documented for the BF16 source model. This NVFP4 repository was validated with an explicit vLLM deployment on a B200; do not assume the AutoEndpoints product path selects this release recipe until that path is separately verified.

A few more things

We created PhoneLLM using Daily’s end-to-end model training stack. Training a good model requires:

1. Collecting or synthetically generating good training data
2. Building evaluation environments
3. Running optimized training processes

We started building this tooling in 2024 to support our work evaluating models for voice use cases, and training small native audio models like the Pipecat Smart Turn turn detection model.

This year, as open weights models have improved to the point where we can use these new small, open models for production voice use cases, we’ve accelerated our work on tooling for model training and customization.

PhoneLLM is a relatively general model, designed to be good at a variety of typical customer support tasks. We can also train models for specific use cases. Each individual training run is affordable enough that updating model weights every month (or even more often) is now a viable strategy.

We think that in the near future, most production agents will continually improve, using feedback loops built around targeted evals and production instrumentation.

• Prompts and context engineering are easy to change and A/B test.
• As the Modal team is demonstrating, inference optimization can significantly lower the cost of running agents in production.
• We can now train small models to perform very well on tasks with well-defined conversation goals and tool definitions.

If you’re interested in building agents that benefit from continually improved performance metrics and cost curves, come talk to us.

If you want to explore building voice agents, the Pipecat docs are a good starting point, and the Voice AI Illustrated Primer is a good deep dive.

Thanks to the fantastic teams at NVIDIA and Modal for the base models and infrastructure that made PhoneLLM possible.

License

PhoneLLM Alpha 1 is released under the BSD 2-Clause License.

PhoneLLM is a derivative work of NVIDIA Nemotron 3 Nano 30B-A3B, which is licensed under the NVIDIA Nemotron Open Model License. Under Section 3 (Redistribution) of that license, if you redistribute this model or your own derivatives of it, you must (a) include a copy of the NVIDIA Nemotron Open Model License, and (b) retain the NVIDIA copyright and attribution notices. Our BSD 2-Clause terms apply to our modifications and to the model as a whole, as Section 3 permits; the NVIDIA license continues to apply to the underlying Nemotron work. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.