phonellm-alpha-1-int8

PhoneLLM Alpha 1 — INT8 (compressed-tensors)

A community INT8 quantization of pipecat-ai/phonellm-alpha-1, built to fit Ampere GPUs (e.g. 3x RTX 3090) that lack NVFP4 support and can't hold the full BF16 checkpoint alongside a usable KV cache.

Why this exists

Neither official PhoneLLM Alpha 1 release fits 3x RTX 3090: they're Ampere (no NVFP4 support), and the BF16 checkpoint alone is ~59GB — too tight for KV cache headroom across only 3x24GB cards. This checkpoint quantizes the MoE experts and general Linear layers to INT8 (channel-wise symmetric weights, dynamic per-token symmetric int8 activations) while keeping attention/Mamba projections, norms, gates, and lm_head in BF16, matching NVIDIA's own precedent for this architecture.

Validation

Coherent generations, and correct <tool_call> emission with accurate parameter extraction, confirmed via a say/do test. This checkpoint has not been run against PhoneBench — no formal accuracy comparison against the BF16 or NVFP4 releases is published here.

Serving with vLLM

vllm serve xanguera/phonellm-alpha-1-int8 \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --max-model-len 8192 \
  --reasoning-parser-plugin nano_v3_reasoning_parser.py \
  --reasoning-parser nano_v3

Important: the reasoning-parser plugin is required. Without it, the model's <think>...</think> reasoning text leaks directly into the visible response content (a literal </think> shows up mid-sentence) instead of landing in a separate reasoning field — setting enable_thinking: false via chat_template_kwargs alone is not sufficient. This only shows up when testing the actual served HTTP endpoint; vLLM's offline batch API doesn't exercise this code path.

Recommended inference settings match the base model: temperature=0, thinking disabled.

License

This model is a quantization of pipecat-ai/phonellm-alpha-1, released under the BSD 2-Clause License, itself a derivative of NVIDIA Nemotron 3 Nano 30B-A3B, licensed under the NVIDIA Nemotron Open Model License.

Per Section 3 (Redistribution) of the NVIDIA Nemotron Open Model License, this repository includes a copy of that license and retains the NVIDIA copyright and attribution notices carried in the bundled model code. This quantization is released under the BSD 2-Clause License, consistent with the terms under which the base model was released; the NVIDIA license continues to apply to the underlying Nemotron work. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.

Acknowledgments

Built on PhoneLLM Alpha 1 by the Pipecat team at Daily, itself built on NVIDIA Nemotron 3 Nano 30B-A3B.