Downloads · 30 days
56
16% of all-time downloads
DeepSeek-V3-4bit-4layers
tflsxyy/DeepSeek-V3-4bit-4layers
DeepSeek-V3-4bit-4layers is a machine learning model from tflsxyy. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This is the first 4 layers of DeepSeek-V3 with GPTQ quantization style. - Layer 4's all routed experts (256 experts) are quantized to 2-bit - All other Linear layers are quantized to 4-bit (including MLA, dense layer…
All-time downloads
353
Public
Parameters
9.5B
7.8 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors7.8 GB · 100%
Parameter types
How the weights are stored.
I327.6B · 80%
Base models
Model card
From the Hugging Face model README
This is the first 4 layers of DeepSeek-V3 with GPTQ quantization style.
- Layer 4's all routed experts (256 experts) are quantized to 2-bit
- All other Linear layers are quantized to 4-bit (including MLA, dense layer ffn, and shared expert)
To load and run this model:
from transformers import AutoModelForCausalLM, AutoTokenizer
from gptqmodel import GPTQModel, QuantizeConfig, get_best_device
pretrained_model_id = "/root/dataDisk/DeepSeek-V3-bf16-4layers"
quantized_model_id = "/root/dataDisk/DeepSeek-V3-4bit-4layers"
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_id, use_fast=True)
device = get_best_device()
model = GPTQModel.load(quantized_model_id, device=device, trust_remote_code=True)
print(tokenizer.decode(model.generate(**tokenizer("gptqmodel is", return_tensors="pt").to(model.device), max_new_tokens=10)[0]))
- Type
- deepseek_v3
- Created
- Mar 8, 2025
- Updated
- Mar 8, 2025