hivemind-32b-preview-AutoRound-AWQ-4bit

Hivemind 32B Preview — AutoRound AWQ 4-bit

<!-- polished-overview:start -->

Overview

hivemind-32b-preview-AutoRound-AWQ-4bit is a weight-quantized checkpoint intended for efficient GPU inference, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• *.safetensors (4 files)
• config.json
• generation_config.json
• tokenizer.json
• tokenizer_config.json
• chat_template.jinja
• quantization_config.json
• Additional configuration, tokenizer, processor, or shard files (11 visible artifacts total)

Quick start

vLLM (AWQ-compatible runtimes)

vllm serve groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit \
  --quantization awq_marlin \
  --dtype float16 \
  --trust-remote-code

The exact kernel and flags depend on the quantizer and architecture. Check the files and source model card before selecting a production serving configuration.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

<!-- polished-overview:end -->

AutoRound AWQ 4-bit quantization of Hivemind 32B Preview, packaged as a Transformers/vLLM-compatible sharded safetensors checkpoint.

Quantization

• Format: AWQ (quant_method: awq)
• Quantizer: AutoRound 0.9.2
• Weight bits: 4
• Group size: 128
• Symmetric: yes
• Zero point: no
• Weight dtype: integer
• Runtime dtype: float16

Generation parameters

The checkpoint's generation configuration uses:

max_new_tokens = 2048
temperature = 0.6
top_p = 0.95
top_k = 20

Context and tested serving profile

• Native maximum context: 40,960 tokens
• Tested with vLLM 0.19.1
• Tensor parallel size: 2
• GPU memory utilization: 0.92
• KV cache dtype: float16 / auto (not FP8)

Example:

vllm serve groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit \
  --served-model-name hivemind-32b-preview-autoround \
  --tensor-parallel-size 2 \
  --max-model-len 40960 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype auto \
  --dtype float16 \
  --quantization awq

On 24 GB GPUs, the tested 40,960-token profile requires two GPUs. A single 24 GB GPU loaded the weights but did not leave enough KV-cache memory for the full native context.

Notes

Review the upstream model's terms and intended-use guidance before deployment. This repository contains a quantized derivative and does not change the capabilities, limitations, or risks of the source checkpoint.