Agnes-3.0-Flash-bnb-4bit

Agnes-3.0-Flash — NF4 (4-bit)

A community bitsandbytes NF4 quantization of Agnes-AI/Agnes-3.0-Flash, prepared by hama-jp. The original model and implementation are by Agnes AI.

The checkpoint was reloaded in a fresh process on an RTX 3090 24GB and checked with Japanese text, arithmetic and an image. It contains 21.35GB of weights in six safetensors files, compared with the source checkpoint's 66.18GB.

This is a Transformers checkpoint, not a GGUF file. It does not use Unsloth Dynamic quantization.

Quantization details

• Source revision: 8f0c484c363cdda8384195be4a5f7730f3915bde.
• 666 linear layers quantized to NF4, with double quantization and BF16 computation.
• Embeddings, output head, vision tower, normalization parameters, and the small recurrent in_proj_a / in_proj_b projections retain BF16 weights.
• Original model Python code and parallel FFN branches retained. No fine-tuning or calibration dataset.
• The source includes 15 mtp.* auxiliary tensors. The upstream Transformers generation implementation has no MTP module and does not load them; they are omitted from this inference checkpoint. The normal generation path is retained.

Quantization settings and tensor coverage include the quantized layer list, retained dtypes and omitted auxiliary tensor names.

Install

Tested with Python 3.12, PyTorch 2.11.0+cu128, Transformers 5.12.1, bitsandbytes 0.50.2 and Accelerate 1.15.0 on WSL2 / RTX 3090 (driver 610.74).

pip install torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cu128
pip install transformers==5.12.1 accelerate==1.15.0 bitsandbytes==0.50.2 pillow==12.3.0

Text generation

The model and input processors use custom Python code, so pass trust_remote_code=True.

The tested 24GB-GPU configuration keeps the output head on CPU. Clearing unused CUDA cache after loading frees temporary loading allocations before generation.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "hama-jp/Agnes-3.0-Flash-bnb-4bit"
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map={"model": 0, "lm_head": "cpu"},
    attn_implementation="sdpa",
).eval()
torch.cuda.empty_cache()
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain LLM weight quantization in two sentences."}],
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_dict=True,
    return_tensors="pt",
).to("cuda:0")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=256, logits_to_keep=1, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The saved checkpoint already contains its quantization settings; loading it does not require quantizing the original weights again.

Image input

After loading the model above:

from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
messages = [{"role": "user", "content": [
    {"type": "image", "image": "photo.png"},
    {"type": "text", "text": "Describe this image in one sentence."},
]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt", enable_thinking=False,
).to("cuda:0")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=256, logits_to_keep=1, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Validation results

Fresh-process reload took 111.25 seconds. The checks used one request at a time, greedy decoding, thinking disabled, and a 256-token generation limit. Every response ended with EOS and all checked logits were finite.

The image answer was: “On the left there is a red circle and on the right there is a blue square.”

Full prompts, outputs and runtime records are included.

Reproduce

The reproduce/ directory contains the pinned source revision, download script, quantization script, fresh-load verification script and recorded package versions. In a separate environment, install the CUDA PyTorch build above and reproduce/requirements.txt, then:

cd reproduce
python download.py
python quantize.py
python verify_export.py

The conversion uses the original 66.18GB checkpoint and writes a new export/ directory. The quantization script stops if export/ already exists. The tested conversion and save took 822.24 seconds.

License and credits

Original weights and Python code: Agnes AI, Apache-2.0. The original LICENSE and source copyright notices are included.

Community changes by hama-jp: NF4 weight conversion, omission of the unused auxiliary MTP tensors, and these loading, reproduction and validation materials.