Agents-A1-nvfp4-mlx

Agents-A1 — Unsloth NVFP4 + E4M3 FP8 weights (MLX, DGX)

Brooooooklyn/Agents-A1-nvfp4-mlx is an MLX mixed-weight-format quantization of InternScience/Agents-A1, prepared for experimental NVIDIA CUDA inference on Linux aarch64. Agents-A1 is a 35B-A3B Qwen3.5-family mixture-of-experts agent model with 40 language-model layers, 256 experts with top-8 routing, hybrid linear/full attention, and an MTP-capable config. The published source checkpoint does not include an mtp.* tensor subtree.

This model is part of the Qwen Unsloth tensor-class recipe for MLX on macOS and DGX collection.

The source was the all-BF16 checkpoint at revision 55100f11160f545dc45545c677699ace74f6bd10.

Update — re-converted 2026-09-01

Expert FFN tensors (switch_mlp, shared_expert) are NOT lifted. The norm they read also drives the softmax router and the sigmoid shared-expert gate, and neither is scale-invariant, so folding a scalar into it would move routing. Those blocks therefore carry the same annihilation as the previous revision -- measured 0.9491% of expert blocks, worst layers.0.mlp.switch_mlp.gate_proj at 41.44%. This is a known open limitation, not a regression.

Weight bytes changed; the tensor-class map, shard layout and tensor counts are unchanged. Earlier revisions remain in this repository's history.

Quantization recipe

This is a data-free, weight-only MLX storage port of the Unsloth Qwen3.6 NVFP4 recipe:

• tensors assigned NVFP4 by the recipe are stored as NVFP4, 4-bit with group size 16;
• tensors assigned FP8 by the recipe are stored as raw E4M3 FP8 weight bytes with one BF16 dequantization scale per output channel;
• every excluded tensor stays BF16.

The NVFP4 class uses MLX weight-only quantized matmul with BF16/A16 activations. For the FP8 class, mlx-node reconstructs each weight to BF16 once at load and then uses ordinary A16 matmul or gather-matmul. The serialized fp8_e4m3 form is Uint8 [..., N, K] weight plus BF16 [..., N, 1] scale; it is not MLX mxfp8 and it is not native W8A8 execution.

No imatrix, calibration dataset, AWQ-style pre-scaling, activation calibration, NVFP4 global scale, or FP8 KV-cache calibration was used. This artifact preserves the recipe's tensor-class boundaries and weight storage formats under mlx-node's A16 runtime; it does not claim numerical or performance parity with Unsloth's calibrated W4A4/W8A8 execution.

The allocation contains 192 NVFP4 modules and 179 E4M3 FP8 modules. The top-level config is nvfp4, 4-bit, group size 16, so the 192 low-class modules inherit that default. The config carries 179 explicit fp8_e4m3 overrides with bits: 8 and group_size: null. The final eight FFN layers intentionally use the higher class; this is the regular, accuracy-oriented 35B recipe rather than the all-FFN-FP4 "Fast" variant.

Target and usage

The canonical native target is aarch64-unknown-linux-gnu: Linux aarch64 with glibc and NVIDIA CUDA 13.0. mlx-node currently validates this experimental, inference-only path on NVIDIA GB10 / DGX Spark (sm_121). It is not a generic CUDA or x86_64 artifact.

At mlx-node 0.0.8, CUDA has no published prebuilt native npm binary. Build mlx-node from source on the DGX host:

git clone --branch v0.0.8 https://github.com/mlx-node/mlx-node.git
cd mlx-node
git submodule update --init --recursive
yarn install
yarn build

Paged attention is Metal-only in this release. Set both eager-mode variables for DGX inference:

MLX_QWEN35_FORCE_EAGER=1 \
MLX_QWEN35_PAGED_OVERRIDE=0 \
  yarn oxnode your-script.ts

For example, your-script.ts can load a locally downloaded copy:

import { loadSession } from '@mlx-node/lm';

const session = await loadSession('./Agents-A1-nvfp4-mlx');
const result = await session.send('Reply briefly: what can you help with?');
console.log(result.text);

This checkpoint requires the @mlx-node/lm and @mlx-node/core 0.0.8 source tree or a newer release that explicitly supports the same Linux target and serialized modes.

Reproduction

Converter: mlx-node at or after PR #131, which made the tuned MX weight encoders and the NVFP4 power-of-two lift unconditional. v0.0.8 reproduces the earlier revision of this repository, not the current weights. The reproducible invocation from the mlx-node repository root was:

mlx convert \
  --input .cache/models/agents-a1 \
  --output .cache/models/agents-a1-unsloth-nvfp4-fp8-dgx-mlx-fresh \
  --model-type qwen3_5_moe \
  --dtype bfloat16 \
  --quantize \
  --q-recipe unsloth \
  --q-mode nvfp4

The resulting five-shard SafeTensors index contains 1,437 tensor entries and reports metadata.total_size = 24,784,342,752 bytes. Its tensor dtypes are 874 BF16, 371 U8, and 192 U32 entries, with 371 scale sidecars and no quantization bias sidecars.

Validation

Static validation confirmed identical quantization and quantization_config blocks, exact index-to-shard closure, 192 inherited NVFP4 groups, 179 complete fp8_e4m3 groups, the expected storage dtypes and shapes, and BF16 preservation for protected tensors. All 333 vision tensor entries remain BF16, and there is no MTP tensor subtree.

The fixed one-token mlx-node load-and-generate smoke test exited successfully with finishReason = "length", numTokens = 1, text = "OK", and rawText = "OK". This one-token text smoke does not validate model quality, long-context behavior, tool use, or the vision path.

Benchmark

macOS A16 fallback only — these are not DGX/CUDA throughput results.

The values below are medians from three fresh child processes, each loading the checkpoint and generating a deterministic 512-token completion on an Apple M5 Max with 128 GiB of unified memory, Darwin 25.5.0/arm64, Node 24.13.1, and @mlx-node/lm, @mlx-node/core, and @mlx-node/core-darwin-arm64 0.0.8. The run used zero warmups, a 60-second cooldown, temperature 0, reasoning effort none, and the same 106-token prompt. Every sample generated all 512 tokens and ended with finishReason = "length".

The prompt and all per-run samples are recorded in benchmark.json. Load time varied strongly because the weights were read from external storage and OS page-cache state differed between fresh processes; treat that median as specific to this run. This fallback benchmark did not exercise DGX, CUDA, native W4A4/W8A8 execution, or the vision path, and it must not be used to infer model quality, memory requirements, or parity with upstream execution.

License and attribution

The source model card declares the Apache-2.0 license. Model capability and training credit belong to InternScience. The tensor-class recipe is credited to Unsloth. This repository converts the pinned BF16 source weights into the mixed NVFP4/plain-E4M3 MLX representation described above.