Agents-A1-4B-nvfp4-mlx

Agents-A1-4B — Unsloth NVFP4 + E4M3 FP8 weights (MLX, DGX)

Brooooooklyn/Agents-A1-4B-nvfp4-mlx is an MLX mixed-weight-format quantization of InternScience/Agents-A1-4B, prepared for experimental NVIDIA CUDA inference on Linux aarch64. Agents-A1-4B is the 4B dense member of the Agents-A1 agentic family: a Qwen3.5-architecture vision-language model with 32 language-model layers in a hybrid attention stack — 8 full-attention layers at indices 3, 7, 11, 15, 19, 23, 27, 31 and 24 gated-delta linear-attention layers — plus a 24-block vision tower. It has 4,539,265,536 parameters, ties its input and output embeddings, and declares mtp_num_hidden_layers: 1 although the published checkpoint ships no mtp.* tensor subtree.

This model is part of the Qwen Unsloth tensor-class recipe for MLX on macOS and DGX collection.

The source was the all-BF16 checkpoint at revision 945c40a4aa6f534d434a353207b8d42ecf7a5293.

Update — re-converted 2026-09-01

Every dense SwiGLU FFN block scale is lifted by an exact power of two, folded into the feeding norm. NVFP4 stores a block scale as amax / 6 in E4M3, and on real weights those land in E4M3's subnormal band, carrying three mantissa bits instead of four -- or rounding to the zero code, which decodes all 16 weights in the block to zero. The fold is exact, so the layer's function is unchanged.

Annihilated dense MLP blocks: 0.2706% -> 0.0000%; worst dense tensor 1.4871% -> 0.0000%.

Weight bytes changed; the tensor-class map, shard layout and tensor counts are unchanged. Earlier revisions remain in this repository's history.

Quantization recipe

This is a data-free, weight-only MLX storage port of the Unsloth Qwen3.6 NVFP4 recipe:

• tensors assigned NVFP4 by the recipe are stored as NVFP4, 4-bit with group size 16;
• tensors assigned FP8 by the recipe are stored as raw E4M3 FP8 weight bytes with one BF16 dequantization scale per output channel;
• every excluded tensor stays BF16.

The NVFP4 class uses MLX weight-only quantized matmul with BF16/A16 activations. For the FP8 class, mlx-node reconstructs each weight to BF16 once at load and then uses ordinary A16 matmul. The serialized fp8_e4m3 form is Uint8 [N, K] weight plus BF16 [N, 1] scale; it is not MLX mxfp8 and it is not native W8A8 execution.

No imatrix, calibration dataset, AWQ-style pre-scaling, activation calibration, NVFP4 global scale, or FP8 KV-cache calibration was used. This artifact preserves the recipe's tensor-class boundaries and weight storage formats under mlx-node's A16 runtime; it does not claim numerical or performance parity with Unsloth's calibrated W4A4/W8A8 execution.

That is 72 NVFP4 modules and 128 E4M3 FP8 modules, 200 in total. The top-level config is nvfp4, 4-bit, group size 16, so the 72 low-class modules inherit that default; the config carries 128 explicit fp8_e4m3 overrides with bits: 8 and group_size: null. The final eight FFN layers intentionally use the higher class; this is the regular, accuracy-oriented recipe rather than an all-FFN-FP4 "Fast" variant.

lm_head differs from the 35B sibling

Agents-A1-4B sets tie_word_embeddings: true. The source checkpoint therefore contains no lm_head.weight, and the converter drops the tied head rather than materializing it. The output head stays BF16, sharing the BF16 embed_tokens matrix. This is the one place where this artifact departs from Agents-A1-nvfp4-mlx, where lm_head is an untied E4M3 FP8 tensor.

Target and usage

The canonical native target is aarch64-unknown-linux-gnu: Linux aarch64 with glibc and NVIDIA CUDA 13.0. mlx-node currently validates this experimental, inference-only path on NVIDIA GB10 / DGX Spark (sm_121). It is not a generic CUDA or x86_64 artifact.

At mlx-node 0.0.8, CUDA has no published prebuilt native npm binary. Build mlx-node from source on the DGX host:

git clone --branch v0.0.8 https://github.com/mlx-node/mlx-node.git
cd mlx-node
git submodule update --init --recursive
yarn install
yarn build

Paged attention is Metal-only in this release. Set both eager-mode variables for DGX inference:

MLX_QWEN35_FORCE_EAGER=1 \
MLX_QWEN35_PAGED_OVERRIDE=0 \
  yarn oxnode your-script.ts

import { loadSession } from '@mlx-node/lm';

const session = await loadSession('./Agents-A1-4B-nvfp4-mlx');
const result = await session.send('Reply briefly: what can you help with?');
console.log(result.text);

This checkpoint also loads and runs on Apple Silicon with @mlx-node/lm and @mlx-node/core 0.0.8 or newer — the benchmark below was measured there — but on macOS the MXFP4 sibling is the intended artifact.

Reproduction

Converter: mlx-node at or after PR #131, which made the tuned MX weight encoders and the NVFP4 power-of-two lift unconditional. v0.0.8 reproduces the earlier revision of this repository, not the current weights. The reproducible invocation from the mlx-node repository root was:

mlx convert \
  --input .cache/models/agents-a1-4b \
  --output .cache/models/agents-a1-4b-unsloth-nvfp4-mlx \
  --model-type qwen3_5 \
  --quantize \
  --q-mode nvfp4 \
  --q-recipe unsloth

No --imatrix-path was supplied. For the fixed DGX map an imatrix is optional and only controls AWQ pre-scaling; the tensor-class assignment is identical either way. Quality may be lower than a calibrated conversion.

Validation

A local mlx-node load-and-generate smoke test loaded the checkpoint in 1309 ms and produced coherent, on-topic English at temperature: 0, terminating on EOS. This functional smoke test does not validate model quality.

Benchmark

Measured on an Apple M5 Max with 128 GiB of unified memory, Darwin 25.5.0/arm64, Node.js 24.13.1, and @mlx-node/lm, @mlx-node/core, and @mlx-node/core-darwin-arm64 0.0.8.

The table reports the median of 3 measured runs. Each run used a fresh child process and fresh model load, with 0 warmups and a 60-second cooldown. Generation was deterministic: temperature: 0, reasoning effort none, 106 prompt tokens, and a 512-token generation limit. All 3 runs generated 512 tokens and ended with finishReason: "length".

Raw per-run samples and the machine-readable summary are available in benchmark.json.

Across the three runs, decode ranged from 64.3 to 67.1 tok/s and model load from 1.051 to 1.067 s. Load timing is sensitive to storage and OS file-cache state, and sustained Apple Silicon throughput is sensitive to thermal state. Treat these figures as host- and cache-specific. They describe only this prompt and configuration and must not be used to infer model quality.

License and attribution

The source model card declares the Apache-2.0 license. Model capability and training credit belong to InternScience. The tensor-class recipe is credited to Unsloth, and this repository only converts the BF16 source weights to an MLX-native mixed NVFP4/E4M3-FP8 representation.