Downloads · 30 days
16
24% of all-time downloads
QWQ-32B-FP8
celertor-ai/QWQ-32B-FP8
QWQ-32B-FP8 is a text generation model from celertor-ai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
All-time downloads
66
Public
Parameters
32.8B
34.3 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors34.3 GB · 100%
Parameter types
How the weights are stored.
F8_E4M331.2B · 95%
Try a prompt
Base models
Model card
From the Hugging Face model README
Model Overview
Description
FP8 Quantized QwQ-32B.
Evaluation
The test results in the following table are based on the MMLU benchmark.
In order to speed up the test, we prevent the model from generating too long thought chains, so the score may be different from that with longer thought chain.
In our experiment, the accuracy of the FP8 quantized version is almost the same as the BF16 version, and it can be used for faster inference.
| Data Format | MMLU Score |
|---|---|
| BF16 Official | 61.2 |
| FP8 Quantized | 61.2 |
| Q8_0 (INT8) | 59.1 |
| AWQ (INT4) | 53.4 |
Quickstart
We recommend using the Chitu inference framework(https://github.com/thu-pacman/chitu) to run this model.
Here provides a simple command to show you how to run QwQ-32B-FP8.
torchrun --nproc_per_node 1 \
--master_port=22525 \
-m chitu \
serve.port=21002 \
infer.cache_type=paged \
infer.pp_size=1 \
infer.tp_size=1 \
models=QwQ-32B-FP8 \
models.ckpt_dir="your model path" \
models.tokenizer_path="your model path" \
dtype=float16 \
infer.do_load=True \
infer.max_reqs=1 \
scheduler.prefill_first.num_tasks=100 \
infer.max_seq_len=4096 \
request.max_new_tokens=100 \
infer.use_cuda_graph=True
Contact
- Task
- Text Generation
- Library
- transformers
- Type
- qwen2
- License
- apache-2.0
- Created
- Nov 16, 2025
- Updated
- Nov 16, 2025