Downloads · 30 days
19
18% of all-time downloads
Ishant86/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-compressed-tensors-int4
FuseO1-DeepSeekR1-QwQ-SkyT1-32B-compressed-tensors-int4 is a machine learning model from Ishant86. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
This is a 4-bit INT4 quantized version of FuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview using the compressed-tensors format.
Downloads · 30 days
19
18% of all-time downloads
All-time downloads
108
Public
Parameters
5.7B
19.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors19.2 GB · 100%
How the weights are stored.
I3231.2B · 95%
From the Hugging Face model README
This is a 4-bit INT4 quantized version of FuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview using the compressed-tensors format.
{
"quant_method": "compressed-tensors",
"format": "pack-quantized",
"bits": 4,
"group_size": 128,
"symmetric": true,
"type": "int"
}
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"Ishant86/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-compressed-tensors-int4",
device_map="auto",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
"Ishant86/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-compressed-tensors-int4"
)
prompt = "Explain the concept of quantum entanglement in simple terms."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
from vllm import LLM, SamplingParams
llm = LLM(
model="Ishant86/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-compressed-tensors-int4",
quantization="compressed-tensors",
tensor_parallel_size=2, # Use 2 GPUs
gpu_memory_utilization=0.85,
)
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=200)
outputs = llm.generate(
["Explain the concept of quantum entanglement in simple terms."],
sampling_params
)
print(outputs[0].outputs[0].text)
FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview is a fusion model that combines capabilities from multiple strong reasoning models. This quantized version preserves the reasoning and instruction-following capabilities while significantly reducing the memory footprint.
Same as the base model: MIT License
Quantized using llm-compressor with symmetric INT4 quantization and a group size of 128 for optimal balance between model size and quality preservation.
Note: For optimal performance, use this model with vLLM and appropriate tensor parallelism settings based on your available hardware.