Gravity-16B-A3B-Preview-GPTQ-Int4

Gravity-16B-A3B-Preview-GPTQ-Int4

A 4-bit GPTQ quantization of trillionlabs/Gravity-16B-A3B-Preview, reducing the checkpoint from ~32.5GB (bf16) to ~8.8GB, produced to make the model fit and serve on a single 24GB consumer GPU (RTX 4090).

At the time this was produced, no quantized release of Gravity-16B-A3B (Preview or Base) existed anywhere, from Trillion Labs or the community.

Quantization details

• Method: GPTQ via GPTQModel 7.1.0
• Bits: 4, group size: 128, symmetric: true, desc_act: false
• Calibration set: ~30 short samples mixing general text/code with insurance/actuarial domain text (this checkpoint was produced as part of the picardian RL-training infrastructure work; the calibration set was not tuned for broad benchmark quality and is not a rigorous production calibration)
• Base architecture: DeepseekV3ForCausalLM (16.24B total / 3.16B active params, Multi-head Latent Attention, 64 routed experts + 1 shared expert)

Known serving limitation (found and documented, not yet fixed upstream)

Serving this checkpoint on sglang-gravity-fork (or upstream sgl-project/sglang at the same commit) with --quantization gptq_marlin crashes with a CUDA illegal memory access inside DeepseekV2Model's MLA weight-absorption step (deepseek_weight_loader.py, post_load_weights, constructing w_kc/w_vc). Root cause: that code path has explicit dequantization support for AWQ (awq_dequantize_f), FP8, and INT8 block-quantization, but no GPTQ path — for GPTQ it falls through to a generic .weight access that does not correctly reconstruct the dequantized tensor, corrupting w_kc's shape/strides.

A separate, unrelated bug was also found and patched locally in the same file: is_packed_weight (used to decide whether to skip a .weight.dtype check for the shared-expert-fusion optimization) allowlists awq/awq_marlin/moe_wna16 but omits gptq/gptq_marlin, causing an AttributeError before the crash above is even reached. That specific one-line fix (adding "gptq" and "gptq_marlin" to the allowlist) is easy and worth upstreaming; the deeper MLA-dequantization gap is not.

Practical implication: this checkpoint is not yet known to serve correctly on SGLang/sglang-gravity-fork. It has not been tested against vLLM or other GPTQ-Marlin-capable servers. AWQ is the quantization format this MLA code path actually has a working dequantization path for — an AWQ requantization of Gravity-16B-A3B is the more likely to work on this same serving stack, if anyone wants to produce one.

License

Apache 2.0, inherited from the base model.