Gander-heretic-ARA-NVFP4

Gander-heretic-ARA-NVFP4

The Thinker of justsumguy/Gander-heretic-ARA, quantised with NVIDIA ModelOpt 0.46.1, plus the unmodified BF16 Talker. The Thinker is 7.2 GB and the Talker 0.7 GB. The abliteration details and numbers are on the BF16 page. In short: heretic ARA branch, ARA-LoRA rank 128, trial 4, refusals 98 → 7 of 100, KL 0.128 on recheck, no healing applied.

Quantisation (thinker/)

talker/ is BF16 and identical to the BF16 repository's.

• Recipes: ModelOpt NVFP4_DEFAULT_CFG for llm; FP8_DEFAULT_CFG for vpm, with patch_embedding excluded.

• llm calibration: 256 samples × 512 tokens of tatsu-lab/alpaca (shuffled, seed 0), run through the extracted text tower. Mean NLL on 8 calibration chats: BF16 4.335, NVFP4 fake-quant 4.408 (+0.073).

• vpm calibration: 80 images from johnowhitaker/imagenette2-320. Three quarters were unsliced; one quarter used max_slice_nums=4.

• vpm held-out check (16 images):

  Output	Mean cosine	Min cosine	Mean rel. L2
  vision tower, last hidden state	0.972	0.957	0.234
  resampler	0.989	0.977	0.145

• Hardware:

  • One RTX 5060 Ti 16 GB (SM120) and one RTX 3050 Ti Laptop 4 GB (SM86), with lm_head on CPU during calibration. The host had 15 GB of RAM.
  • The BF16 tower was streamed from the shards across both GPUs.

Loading: not verified

This is a ModelOpt export: llm weights are packed uint8 with weight_scale, weight_scale_2 and input_scale tensors, and hf_quant_config.json says MIXED_PRECISION. Neither from_pretrained with the remote code's pinned transformers nor the Gander runtime loads it as-is. A ModelOpt-aware loader is needed.

The Gander runtime's NVFP4 path has not been confirmed. Realtime duplex behaviour and performance have not been measured on this build.

License

Apache-2.0, as upstream. See thinker/LICENSE and thinker/NOTICE. The modifications are the abliteration of the Thinker's language tower and the quantisation described above.