Agents-A1-TextOnly-FP4

Agents-A1-TextOnly-FP4-VocabPruned

This is an optimized, hardware-accelerated 4-bit FP4 quantization of InternScience/Agents-A1 (a 35B parameter Mixture-of-Experts agentic model with hybrid Mamba-Attention layers and 2M context length) with pruned vocabulary and tuned prefill settings for extreme context capacity.

This optimized checkpoint was created by pruning the multimodal vision encoder and excluding the grafted Multi-Token Prediction (MTP) weights to drastically reduce the VRAM footprint, and then slicing the vocabulary size from 248,320 down to 152,064 tokens (mapping away unused visual/coordinate tokens).

It is designed to be served natively under vLLM V1 with FP8 KV cache support.

Key Work Done & Optimizations

1. Multimodal Pruning: Removed the visual tower (model.visual.) and skipped the MTP heads (model_mtp.safetensors), saving 2.0 GB of VRAM during execution.
2. Vocabulary Pruning: Sliced vocabulary down to 152,064 (standard Qwen-2.5-Text vocab scale) by mapping away 96,256 unused vision/coordinate tokens. This saved ~370 MiB of weight VRAM per GPU.
3. NVIDIA ModelOpt Quantization: Calibrated and quantized the weights to W4A16_NVFP4 block-wise schema (block size 16), ensuring full compatibility with vLLM's hardcoded Marlin FP4 group size constraint.
4. Prefill Tuning: Tuned chunked prefill settings to --max-num-batched-tokens 2048, freeing significant activation memory during execution.

KV Cache Capacity & Concurrency (RTX 5060 Ti 16GB TP=2)

These optimizations expand the physical KV cache capacity to 715,582 tokens (a +17.2% capacity gain over standard configurations), enabling stable long-context execution:

Hardware & Serving Recommendations

GPU Requirements

• Ideal Dual-GPU Setup: Two RTX 5060 Ti GPUs (16GB VRAM each) for budget deployment, or any Blackwell GPU pair (e.g., RTX 5080 / 5090) for larger KV cache headroom.
• VRAM Budget: The model weights occupy ~20.1 GB total (~9.56 GB per GPU under TP=2). Operating with FP8 KV cache enables massive contexts to scale comfortably.

vLLM Serving Command

vllm serve Cadododoom/Agents-A1-TextOnly-FP4 \
  --tensor-parallel-size 2 \
  --quantization compressed-tensors \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --mamba-cache-mode align \
  --max-model-len 256000 \
  --max-num-batched-tokens 2048 \
  --swap-space 16 \
  --trust-remote-code