Downloads · 30 days
11
10% of all-time downloads
Ishant86/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-awq-int4
FuseO1-DeepSeekR1-QwQ-SkyT1-32B-awq-int4 is a machine learning model from Ishant86. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This is a 4-bit quantized version of FuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview using AWQ (Activation-aware Weight Quantization) technique via the compressed-tensors method. This is a fusion model that combines ca…
Downloads · 30 days
11
10% of all-time downloads
All-time downloads
115
Public
Parameters
5.7B
19.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors19.2 GB · 100%
How the weights are stored.
I3231.2B · 95%
From the Hugging Face model README
This is a 4-bit quantized version of FuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview using AWQ (Activation-aware Weight Quantization) technique via the compressed-tensors method. This is a fusion model that combines capabilities from multiple reasoning models including DeepSeek-R1, QwQ, and SkyT1, built on the Qwen2 architecture. The model has been optimized for efficient inference while maintaining high performance.
This quantized model is designed for:
from transformers import AutoModelForCausalLM, AutoTokenizer
# Replace with your HuggingFace model ID
model_id = "Ishant86/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-AWQ-4bit"
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Example inference
prompt = "What is the meaning of life?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
from vllm import LLM, SamplingParams
# Replace with your HuggingFace model ID
model_id = "Ishant86/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-AWQ-4bit"
llm = LLM(
model=model_id,
quantization="compressed-tensors",
dtype="auto"
)
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=100)
outputs = llm.generate("What is the meaning of life?", sampling_params)
print(outputs[0].outputs[0].text)
import sglang as sgl
# Replace with your HuggingFace model ID
model_id = "Ishant86/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-AWQ-4bit"
runtime = sgl.Runtime(
model_path=model_id,
quantization="compressed-tensors"
)
response = runtime.generate("What is the meaning of life?", max_tokens=100)
print(response["text"])
The model was quantized using the following configuration:
quant_stage:
quant_modifiers:
GPTQModifier:
config_groups:
group_0:
targets: [Linear]
weights:
num_bits: 4
type: int
symmetric: true
group_size: 128
strategy: group
actorder: static
observer: minmax
ignore: [lm_head]
block_size: 128
dampening_frac: 0.01
| Property | Value |
|---|---|
| Model Format | SafeTensors |
| Quantization Format | pack-quantized |
| Bits per Weight | 4 |
| Group Size | 128 |
| RoPE Theta | 1,000,000 |
| RMS Norm Epsilon | 1e-05 |
| Attention Dropout | 0.0 |
| Tie Word Embeddings | False |
| Use Sliding Window | False |
config.json - Model configurationtokenizer.json - Tokenizer vocabulary and mergestokenizer_config.json - Tokenizer configurationchat_template.jinja - Chat template for conversation formattinggeneration_config.json - Default generation parametersmodel-*.safetensors - Quantized model weights (4 shards)model.safetensors.index.json - Weight shard mappingquantization_log.json - Quantization process metadatarecipe.yaml - Quantization recipe configurationspecial_tokens_map.json - Special token mappingsFuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview is a fusion model that combines:
This fusion approach aims to combine the strengths of multiple reasoning-focused models into a single, powerful model with enhanced reasoning capabilities.
If you use this quantized model, please cite the original models and the quantization methodology:
@misc{fuseai-fuseo1,
title={FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview},
author={FuseAI},
year={2025},
howpublished={\url{https://huggingface.co/FuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview}}
}
@article{qwen2,
title={Qwen2 Technical Report},
author={Qwen Team},
journal={arXiv preprint},
year={2024}
}
@article{compressed-tensors,
title={Compressed Tensors: Efficient Model Compression},
author={Neural Magic},
year={2024}
}
This model inherits the license from the original FuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview model. Please refer to the original model's licensing terms on HuggingFace.
For issues or questions regarding this quantized model, please:
Compressed-Tensors Version: 0.12.2
Transformers Version: 4.56.2