XtraGPT-14B-W4A16

XtraGPT-14B-W4A16

This is an AWQ W4A16 quantization of Xtra-Computing/XtraGPT-14B. It is intended for context-aware academic-paper revision and preserves the original tokenizer and chat template.

Quantization

• Algorithm: Activation-aware Weight Quantization (AWQ)
• Format: compressed-tensors
• Weights: asymmetric INT4
• Activations: 16-bit
• Group size: 128
• Quantized modules: Linear, excluding lm_head
• Calibration: 128 samples from Xtra-Computing/ReviseQA, maximum sequence length 2048
• Source revision: 01dd781fea28a43bb36a0d297ee8181e849fb10c
• Calibration dataset revision: 4826925f33d825be0d54ca7c5618c032d55ea337

The calibration inputs keep both the beginning of each paper prompt and the final revision question/answer when a sample exceeds the sequence limit.

Usage with vLLM

vllm serve REPO_ID \
  --served-model-name xtragpt \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.95

Use the <PAPER_CONTENT>, <SELECTED_CONTENT>, and <QUESTION> prompt tags shown in the base model card.

Build environment

• torch==2.13.0
• transformers==5.14.1
• llmcompressor==0.13.0
• compressed-tensors==0.18.0
• datasets==5.0.1
• huggingface-hub==1.28.0

Evaluation

A short generation smoke test was run during quantization unless explicitly disabled. No benchmark result is claimed here; compare the quantized checkpoint with the BF16 source on a held-out ReviseQA subset before production use.

License and attribution

The source model is released under the ModelGo Zero License 2.0 (MG0-2.0). See the base model card and license text.