Agnes-3.0-Flash-FP8

Agnes-3.0-Flash Preview — FP8

Plain, data-free blockwise FP8 E4M3 of the 33B Agnes-3.0-Flash Preview checkpoint, produced by ProCreations directly from the original BF16 weights. Weight files total 42.78 GB, 35.4% smaller than the 66.18 GB BF16 source. Sizes are decimal and exclude runtime buffers and context memory.

This is a precision-focused alternative to our NVFP4 version. FP8 uses more precision for both quantized weights and activations, while the components listed below remain in BF16. A later 40-case matched comparison against NVFP4 is summarized below; it is a small diagnostic sample, not a broad quality guarantee.

The source checkpoint supports 262,144 context tokens. These are Preview weights, distinct from Agnes's newer production/API model; its results and 1M-token context do not describe this checkpoint.

Precision and conversion

The 288 quantized matrices use 128 × 128 weight blocks, FP32 scales, and E4M3FN values. Each scale is the block's absolute maximum divided by 448 (an all-zero block uses scale 1). Values are converted with round-to-nearest-even. Native inference computes activation scales dynamically for each token and group of 128 elements. No training, calibration data, calibration forward passes, clipping search, or Hessian optimization was used. The checkpoint was built from BF16, not converted from NVFP4.

Each layer's 17,408-wide main FFN and 2,048-wide parallel FFN were combined into width 19,456, using the same concatenation as the upstream SGLang BF16 loader: gate/up along rows and down along columns. This preserves their algebraic function; finite-precision arithmetic may differ from separately evaluating and summing the two branches. The bundled loader handles the already-combined weights and translated attention names.

All 1,017 unquantized tensors (19,372,570,720 bytes) were verified for exact dtype and value equality with the pinned source. Original tokenizer, chat template, image/video processors, and MTP tensors are retained. Source weight hashes, all FP8 tensors, positive finite scales, and exported file hashes were checked. The quality report includes per-matrix reconstruction error; weight error is not a task-quality score.

Lightweight validation

The exact exported checkpoint loaded as native fp8 on one RTX PRO 6000 96 GB, using the pinned SGLang image below. Six smoke cases were run; five passed. Arithmetic, multilingual JSON, Python syntax, image color recognition, and retrieval passed. The simulated two-step tool exchange failed the strict JSON-format check on its final reply. That unsuccessful result is retained in the report, and tool-workflow quality retention is not established by these checks. The largest smoke input contained 13,409 prompt tokens. Generated Python was parsed but not executed.

These checks confirm basic loading and the successful functions above; they do not establish universal correctness. The initial release used only these smoke checks. A later bounded FP8/NVFP4 comparison is summarized below; BF16 was not rerun. Long agentic tasks, full 262K context, video, and speculative decoding were not tested. Smoke outputs, settings, and commands are included. The failed tool-format case records its parser error; its raw reply was not retained by the smoke harness.

<!-- bounded-fp8-comparison -->

Bounded FP8 versus NVFP4 comparison

A later comparison ran 40 matched cases per model, once each, using the same prompts, GPU type, pinned runtime, and generation settings. No clear FP8 quality advantage appeared in this small sample.

The models had identical pass/fail outcomes on 39/40 cases. The sole difference was strict formatting: FP8 included explanatory prose around a correct inventory JSON result; NVFP4 returned just the JSON code block. Both supplied the correct tool arguments and data. This is not evidence of a general quality ranking. Both also repeated the same inventory lookup on the earlier problematic smoke prompt, so that failure is not unique to FP8.

Both passed retrieval through 60,052 actual prompt tokens. FP8 weight files are 42.78 GB versus 32.54 GB for NVFP4. These are diagnostic subsets, not official benchmark scores or a full-context/long-agent guarantee. See the full comparison, settings, limitations, and raw responses.

<!-- /bounded-fp8-comparison -->

Inference

Use the bundled loader and the tested image. The example matches the smoke-test configuration, with 32K context. The source configuration retains 262K support, but this FP8 release was not smoke-tested at that limit. Runtime memory depends on context and concurrency.

hf download ProCreations/Agnes-3.0-Flash-FP8 --local-dir ./Agnes-3.0-Flash-FP8
docker run --rm --gpus all --ipc=host -p 8080:8080 \
  -v "$PWD/Agnes-3.0-Flash-FP8:/model:ro" \
  lmsysorg/sglang@sha256:9a352a35c973a2357372e85f3bcb5388b6b3c46c1329165987260f3b089647dc \
  bash /model/serve.sh \
  --served-model-name Agnes-3.0-Flash-FP8 \
  --tp-size 1 --context-length 32768 \
  --max-running-requests 2 --chunked-prefill-size 4096 \
  --mem-fraction-static 0.85 --cuda-graph-max-bs 2 \
  --fp8-gemm-backend cutlass \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --chat-template /model/chat_template.jinja \
  --mamba-scheduler-strategy extra_buffer

The server exposes an OpenAI-compatible API on port 8080. Use the source chat template's chat_template_kwargs to configure thinking, for example {"enable_thinking": true, "reasoning_effort": "low"}. General compatibility with unpatched SGLang, vLLM, or other hardware/loaders was not established. The example's CUTLASS backend was tested on the RTX PRO 6000.

Reproducibility and license

Source: Agnes-AI/Agnes-3.0-Flash at 24f712ce59379b54c4a141d2708c35daf5ff613b. Build code, smoke code, runtime image, inference hashes, and release provenance are included. Publication preserves the native-tested inference file hashes.

Independent ProCreations quantization under Apache 2.0. Source-model limitations still apply. See NOTICE for attribution and modifications.