Agnes-3.0-Flash-NVFP4

Agnes-3.0-Flash Preview — NVFP4

Plain, data-free NVFP4 quantization of the 33B Agnes-3.0-Flash Preview checkpoint, produced by ProCreations. The source supports 262,144 context tokens. These weights are distinct from Agnes's newer production/API checkpoint; its results and 1M-token context do not describe this model.

Weight files occupy 32.54 GB, 50.8% smaller than the 66.18 GB BF16 source. Sizes are decimal and exclude runtime buffers and context memory.

Quantization

NVFP4 uses E2M1 values with E4M3 block scales, blocks of 16, maximum-based weight scaling, and activation global scales fixed at 1.0 with dynamic per-block scales. There was no training, calibration dataset, or MSE/Hessian optimization. ModelOpt's export inspected the graph with two dummy tokens while quantizers and calibration statistics were disabled; this was not data calibration.

Agnes has a main FFN of width 17,408 and a parallel FFN of width 2,048 in every decoder layer. Before quantization, they were combined into width 19,456 using exactly the tensor concatenation used by the upstream SGLang BF16 loader: gate/up along rows and down along columns. This preserves their algebraic function and matches the served BF16 reference; BF16 arithmetic can differ slightly from separately evaluating and summing both branches.

All 1,017 unquantized tensors, totaling 19,372,570,720 bytes, were checked for identical dtype and exact values against the pinned source. All 288 packed matrices and their scales passed structural checks. The original tokenizer, chat template, image/video processors, and 15 MTP tensors are retained. The bundled loader was adapted to recognize the already-combined branches and preserve the intended precision exclusions.

Native evaluation

The main table covers nine completed diagnostic subsets with 408 paired cases on the pinned BF16 source and this exact packed checkpoint. Both used an RTX PRO 6000 with the same pinned SGLang runtime, prompts and scorers. These are subsets and custom diagnostics, not official full-benchmark scores.

The extended suite was stopped early. Coding, missing-parameter/function tool use, and synthetic long tool-workflow comparisons remain unfinished. In total, BF16 returned 439 cases and NVFP4 returned 473, with 435 matching case IDs. Partial subsets are uniformly excluded from the main table; every returned case, including partial and unpaired results, is retained in the full comparison.

BF16 recovery runs used --mem-fraction-static 0.94 --max-mamba-cache-size 16 to accommodate large prompts; weight and cache/state precision were unchanged. NVFP4 used the original --mem-fraction-static 0.88 throughout. Resumed requests used a 1,800-second HTTP deadline with unchanged generation caps. Original completed results take precedence over duplicates regardless of score. Both configurations and per-case provenance are recorded in the comparison.

On the completed subsets, NVFP4 had zero to two fewer correct cases than BF16 per subset. Both passed all multilingual JSON, simple tool-call, and long retrieval cases. These diagnostic results do not establish retention across every workload, especially the unfinished coding and synthetic long-agent comparisons.

Both models passed all 12 retrieval tests, reaching 256,036 actual prompt tokens. Simple retrieval does not establish retention for long agentic workflows: the synthetic long tool-workflow comparison has no completed matched cases. Broad reasoning used temperature 0, source xhigh thinking and an 8,192-token output cap; BFCL used 4,096 tokens per step. BFCL scores require both the upstream state/response check and its strict irrelevance check to pass; component scores are included in the report. Short exact-format tasks disabled thinking. Truncated responses remain included and are counted in the full comparison.

Small diagnostic differences do not guarantee retention across every workload. Perplexity and teacher-logit KL were not measured. Video tasks, the exact 262,144-token boundary, speculative MTP decoding, and real repository/browser agents were not evaluated. Full per-case results, dataset versions and runtime commands are in native_comparison.json.

<!-- bounded-fp8-comparison -->

Bounded FP8 versus NVFP4 comparison

A later comparison ran 40 matched cases per model, once each, using the same prompts, GPU type, pinned runtime, and generation settings. No clear FP8 quality advantage appeared in this small sample.

The models had identical pass/fail outcomes on 39/40 cases. The sole difference was strict formatting: FP8 included explanatory prose around a correct inventory JSON result; NVFP4 returned just the JSON code block. Both supplied the correct tool arguments and data. This is not evidence of a general quality ranking. Both also repeated the same inventory lookup on the earlier problematic smoke prompt, so that failure is not unique to FP8.

Both passed retrieval through 60,052 actual prompt tokens. FP8 weight files are 42.78 GB versus 32.54 GB for NVFP4. These are diagnostic subsets, not official benchmark scores or a full-context/long-agent guarantee. See the full comparison, settings, limitations, and raw responses.

<!-- /bounded-fp8-comparison -->

Inference

Use the bundled serve.sh with the tested image below. The script installs the Agnes loader into the disposable container. The validated hardware was one RTX PRO 6000 96 GB; memory use depends on context and concurrency. Native NVFP4 requires a compatible Blackwell runtime.

hf download ProCreations/Agnes-3.0-Flash-NVFP4 --local-dir ./Agnes-3.0-Flash-NVFP4
docker run --rm --gpus all --ipc=host -p 8080:8080 \
  -v "$PWD/Agnes-3.0-Flash-NVFP4:/model:ro" \
  lmsysorg/sglang@sha256:9a352a35c973a2357372e85f3bcb5388b6b3c46c1329165987260f3b089647dc \
  bash /model/serve.sh \
  --served-model-name Agnes-3.0-Flash-NVFP4 \
  --tp-size 1 --context-length 262144 \
  --max-running-requests 4 --chunked-prefill-size 4096 \
  --mem-fraction-static 0.88 --cuda-graph-max-bs 4 \
  --fp4-gemm-backend flashinfer_cutlass \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --chat-template /model/chat_template.jinja \
  --mamba-scheduler-strategy extra_buffer

For OpenAI-compatible requests, the source chat template accepts xhigh, medium, or low thinking via chat_template_kwargs, for example:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="Agnes-3.0-Flash-NVFP4",
    messages=[{"role": "user", "content": "Explain how a hash table works."}],
    max_tokens=2048,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "xhigh", "enable_thinking": True}},
)

General compatibility with unpatched SGLang, vLLM, or other loaders has not been established. Use the provided image and loader for the tested configuration.

Reproducibility and license

Source: Agnes-AI/Agnes-3.0-Flash at 24f712ce59379b54c4a141d2708c35daf5ff613b. NVIDIA Model Optimizer revision: 5cae3940402f1ced98069a666b0bec72ec8b33b5. Build dependencies: PyTorch 2.13.0, Transformers 5.12.1. ModelOpt's source archive reports version 0.0.0; its commit identifies the build.

Build provenance and inference hashes, quality report, build script, build environment, and post-export normalization are included. Every served quantized inference file was copied into local job storage and verified against its Hub content hash before loading. Publication preserves those tested hashes.

Evaluation sources: GSM8K, MMLU-Pro, DocumentVQA, HumanEval+, LongBench v2, and BFCL.

Independent ProCreations quantization, released under Apache 2.0. Source-model limitations still apply. See NOTICE for attribution and modifications.