qwen2.5-7b-gptq-4bit

Qwen 2.5 7B Instruct — GPTQ 4-bit

Self-quantized GPTQ 4-bit checkpoint of Qwen/Qwen2.5-7B-Instruct with fully documented calibration provenance.

Created as part of the Banterhearts research program investigating quality-safety correlation under quantization for consumer LLM deployment.

Quantization Details

Why Self-Quantized?

Pre-quantized checkpoints on HuggingFace typically have unknown calibration provenance — the dataset, sample count, seed, and group size are rarely documented. This checkpoint was self-quantized with controlled, documented settings to enable rigorous cross-method comparison (GGUF k-quant vs AWQ vs GPTQ) in a NeurIPS 2026 submission on quality-safety correlation under quantization.

Evaluation Results

Evaluation pending — quality and safety benchmarks will be run on this checkpoint and results updated here.

Other Quantization Formats

Prompt Template

<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Crusadersk/qwen2.5-7b-gptq-4bit",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Crusadersk/qwen2.5-7b-gptq-4bit")

messages = [{"role": "user", "content": "What is the capital of France?"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Inference requirements: pip install gptqmodel (Linux only) or optimum+auto-gptq

Windows users: GPTQ inference requires gptqmodel which only builds on Linux. Use Docker or WSL2.

Compatibility

Reproduction

The full quantization pipeline — Dockerfiles, quantization scripts, and a 766-line engineering log documenting every platform failure and solution — is available at:

research/tr142/expansion/

in the Banterhearts repository.

Citation

@misc{banterhearts2026qwen7bgptq,
  title = {Self-Quantized Qwen 2.5 7B Instruct (GPTQ 4-bit) for Quality-Safety Correlation Research},
  author = {Kadadekar, Sahil},
  year = {2026},
  url = {https://huggingface.co/Crusadersk/qwen2.5-7b-gptq-4bit},
  note = {Part of the Banterhearts research program. NeurIPS 2026 submission.}
}

Acknowledgments

This work is part of the Chimera/Banterhearts technical-report program on deployment-time LLM behavior, quantization, refusal robustness, batching effects, and inference-stack reliability. Canonical public archive: Chimeraforge Reports; source context: github.com/Sahil170595/Banterhearts.