Agents-A1-Q8-Plus

Agents-A1-Q8-Plus

Overview

This repository provides a highly optimized, custom-quantized GGUF model of Agents-A1, specifically engineered for local deployment on Dual RTX 3090 setups. The primary research objective of this quantization is to achieve an extreme context length (full 262K tokens in F16 KV Cache) while maximizing inference speed through adapted BPW and retaining most of the original model's capacities. To achieve this, the base network was quantized to Q8_0 using a custom iMatrix, while the critical layers, including outputs and embeddings were strictly preserved in f16. The base model used for this requantization is InternScience/Agents-A1.

Research & Methodology

Selective precision Quantization for high-speed inference

Most standard quantization pipelines compress the entire model, which severely degrades the quality and the inference speeds. On RTX 3090s and llama.cpp f16 enables the highest speeds and several community researches found out that f16 is largely sufficient compared with bf16.

To maintain capacities while optimizing for a Dual RTX 3090 (NVLink) setup, I have implemented a Selective Precision Mapping strategy. By carefully partitioning the model into f16 (Highest Precision) and Q8_0 tensors, it preserves the critical activation flows, without sacrificing the throughput necessary for processing massive contexts (full 262K tokens).

Strategic Layer Mapping

The following quantization scheme was applied :

--tensor-type 'blk\.[0-9]+\.ffn_gate_inp\.weight=f16'
--tensor-type 'blk\.[0-9]+\.ffn_gate_inp_shexp\.weight=f16'
--tensor-type 'blk\.[0-9]+\.ssm_alpha\.weight=f16'
--tensor-type 'blk\.[0-9]+\.ssm_beta\.weight=f16'
--tensor-type 'blk\.[0-9]+\.attn_gate\.weight=f16'
--tensor-type 'blk\.[0-9]+\.attn_qkv\.weight=f16'
--tensor-type 'blk\.[0-9]+\.ffn_gate_shexp\.weight=f16'
--tensor-type 'blk\.[0-9]+\.ffn_up_shexp\.weight=f16'
--tensor-type 'blk\.[0-9]+\.ffn_down_shexp\.weight=f16'
--tensor-type 'token_embd\.weight=f16'
--tensor-type 'output\.weight=f16'
--tensor-type 'blk\.[0-9]+\.ssm_out\.weight=f16'
--tensor-type 'blk\.[0-9]+\.attn_q\.weight=f16'
--tensor-type 'blk\.[0-9]+\.attn_k\.weight=f16'
--tensor-type 'blk\.[0-9]+\.attn_v\.weight=f16'
Q8_0

This scheme keeps the most important layers at high precision (f16) and lowers the GGUF size on disk, enabling full offload on dual RTX 3090 setups.

iMatrix Calibration

The model was calibrated using a custom, shuffled iMatrix to ensure high fidelity across coding, instruction-following, and bilingual tasks (English/French).

Chat Template

Agents-A1 appears to be fine-tuned from Qwen3.5-35B-A3B and therefore supports the same chat templates. Considering all the experience I have with both Qwen3.5 and 3.6 models, I chose to add, from the very beginning, the froggeric Chat_template made for 3.6. As far as I tested, it is fully compatible and therefore is benefiting from all the improvements made by the original creator.

Accuracy Analysis

I compared the perplexity and KL-Divergence on a custom calibration set, based on code extracted from llama.cpp, here after refered as 'Code', and Wiki-Text-Raw to evaluate the precision loss after the mixed-quantization scheme:

Key Findings:

• Near-Lossless: The perplexity degradation is minimal at +0.001071 on wiki-text-raw and +0.001227, indicating that this mixed precision layout preserves the original model's reasoning capabilities.

Recommended Usage

To replicate the optimal performance (262K context, F16 Cache, Multi-GPU) using llama.cpp, use the following llama-server command. Note the specific use of --split-mode tensor and --tensor-split 1,1 for optimal PCIe bandwidth management across dual RTX 3090s. This command appeared to be the best one I could come across using an NVLink.

/path/to/llama.cpp/build/bin/llama-server \
    -m /path/to/Agents-A1-Q8-Plus \
    --split-mode tensor \
    --tensor-split 1,1 \
    --host 0.0.0.0 \
    --port 8080 \
    --ctx-size 262144 \
    --parallel 1 \
    --gpu-layers 999 \
    --cache-type-k f16 \
    --cache-type-v f16 \
    --flash-attn on \
    -b 2048 -ub 2048 \
    --alias Agents-A1 \
    --jinja

If I may add, I also developed a proxy to enable users to select thinking or non-thinking behaviors and applied the recommended sampling parameters AND the "Preserve Thinking" option. You may find it on my GitHub.

Hardware Requirements

• Target VRAM: 48 GB (Tested on 2x NVIDIA RTX 3090 24GB).
• RAM: Minimum 32GB system RAM (Prompt caching and system overhead).
• Context limit: The command above loads ~9GB of KV cache across the two GPUs. If you experience OOM (Out of Memory) errors, consider reducing --ctx-size or using 8-bit cache (--cache-type-k q8_0 --cache-type-v q8_0, not recommended unless strictly necessary).

Acknowledgments

This project was made possible thanks to the outstanding tools and contributions from the open-source AI community. Special thanks to:

• Lei Bai et al. (InternScience): For their groundbreaking research "Scaling the Horizon, Not the Parameters" and for releasing the Agents-A1 base model.
• llama.cpp: Using the new Tensor split mode, it finally achieves extremelly high performances on dual-GPU setups.
• froggeric: For the immense work done on fixing Qwen3.5 and 3.6 chat_template.
• The Qwen Team: For researching and releasing the exceptional Qwen3.5 architecture.