Behemoth-ReduX-123B-v1.1_Compressed-Tensors

Behemoth-ReduX-123B-v1.1 — Quantized (compressed-tensors for vLLM)

This repository provides quantized runtime packages of
TheDrummer/Behemoth-ReduX-123B-v1.1-GGUF, repackaged for vLLM using the compressed-tensors format.

TL;DR

• This repo is quantized with branches W4A16 and W4A16-SEQ1024.
• Load with vLLM using --quantization compressed-tensors.
• Typical W4A16 recipe: group_size=128, keep lm_head in higher precision; inherits the parent finetune’s chat template.

---

Revisions & Branches

The main branch is a landing page (model card + links). All runnable artifacts live under per-revision branches.

• main — placeholder / landing page
• W4A16 — 4-bit weights / 16-bit activations builds and runtime assets
• W4A16-SEQ1024 — same as W4A16, but calibrated with sequence length 1024 for long-form stability

Quick links:

• 🔗 main
• 🔗 W4A16
• 🔗 W4A16-SEQ1024

---

Repository Contents (per revision)

• Sharded quantized weights in .safetensors with an index (model.safetensors.index.json)
• config.json including compressed-tensors metadata (weight_format, quantization, quantization_config)
• Tokenizer artifacts (tokenizer.json, tokenizer.model, etc.)
• Optional: chat_template.jinja (inherits the parent finetune’s chat format)

Exact files can differ by branch; see the Files and versions tab for each revision.

---

Quantization recipes (reference)

Shared choices

• Scheme: W4A16 (INT4 weights / A16 activations) via modern AWQ pipeline
• Group size: 128 (unless noted in branch config.json)
• Ignored: lm_head kept in higher precision
• Export: save_compressed=True to write compressed-tensors runtime metadata for vLLM

Branch-specific

• W4A16 — calibrated on WikiText-style prose, typical seq_len 512
• W4A16-SEQ1024 — identical settings, but calibration seq_len = 1024 to better preserve long-range statistics

Settings may vary slightly per branch; consult each branch’s config.json for the exact parameters used.

---

Quickstart — vLLM (compressed-tensors)

Install vLLM (recent version recommended):

pip install vllm