granite-3.3-2b-instruct-int4-gram

Granite-3.3-2b-instruct, int4 (Hessian-aware rounding, one-sided Gram)

int4 weight-quantized ibm-granite/granite-3.3-2b-instruct in lalamo format, runs in lalamo and on-device in uzu.

Every linear is rounded against its own input activation Gram (X^T X), i.e. GPTQ / LDLQ-grade curvature, through lalamo's yaqa_round_blockwise.

This is not true YAQA. The preconditioner is an activation Gram, not a gradient Fisher, and only the input block is used (the output block is identity). The name matters because the recipe is the point: everything below is reproducible from the base model.

Results

MMLU-Pro, canonical TIGER-Lab 5-shot CoT, greedy, paired, full set n=12032, both rows generated through the same engine (lalamo reply_many):

wikitext held-out PPL (1024 tokens): bf16 5.162 · this 5.457 · int4 min/max PTQ 5.396 · int4 QAD 4.860.

Note the PPL/MMLU inversion: this checkpoint's LM perplexity is slightly worse than naive min/max int4, yet its downstream accuracy is much better (on a paired n=1000 subset: this 33.3 vs min/max PTQ 30.0, bf16 34.0). Perplexity is a poor selector between int4 rounding methods here; the reasoning benchmark separates them.

Recipe

Reproduce

python make_int_checkpoint.py \
  models/granite-3.3-2b-instruct models/granite-3.3-2b-instruct-int4-gram \
  --yaqa --calib-file calib_wikitext.txt --calib-tokens 12288 --chunk-len 512 \
  --bits 4 --group-size 64

make_int_checkpoint.py is a thin driver over lalamo.compressed: it accumulates the per-linear input Gram over the calibration chunks and passes it as the preconditioner to IntSpec.compress. down/out inputs are recovered by re-running each layer's own mlp/mixer with the trailing projection swapped for an identity, so no core lalamo changes are needed.

Caveats

• Calibration rank is load-bearing. Below full rank (calib tokens < in-dim) the Gram is rank-deficient, rounding in its null space is unconstrained, and the checkpoint collapses (we measured PPL 57487 at 242 calib tokens). Keep calibration tokens above 8192 for this model.
• Evaluated with the public TIGER-Lab MMLU-Pro protocol, not Mirai's internal reasoning-enabled harness, so the delta vs bf16 is the transferable number, not the absolute score.