Downloads · 30 days
5
12% of all-time downloads
fbaldassarri/meta-llama_Llama-3.2-3B-Instruct-auto_round-int4-gs64-sym
meta-llama_Llama-3.2-3B-Instruct-auto_round-int4-gs64-sym is a text generation model from fbaldassarri. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.2.
Quantized version of meta-llama/Llama-3.2-3B-Instruct using torch.bfloat16 for quantization tuning. - 4 bits (INT4) - group size = 64 - Symmetrical Quantization - Method: WoQ — SignRound (AutoRound algorithm)
Downloads · 30 days
5
12% of all-time downloads
All-time downloads
42
Public
Parameters
796M
3.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors3.1 GB · 99%
How the weights are stored.
F32394M · 50%
From the Hugging Face model README
Quantized version of meta-llama/Llama-3.2-3B-Instruct using torch.bfloat16 for quantization tuning.
Fast and low memory, 2-3X speedup (slight accuracy drop at W4G64)
Quantization framework: Intel AutoRound v0.13.1
Note: this INT4 version of Llama 3.2 3B Instruct has been quantized for inference on Intel CPU, Intel iGPU (Arc) via intel-extension-for-pytorch, Intel NPU (AI Boost on Core Ultra series) via OpenVINO.
This is an instruct / chat model — use its built-in chat template rather than raw text prompts:
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "fbaldassarri/meta-llama_Llama-3.2-3B-Instruct-auto_round-int4-gs64-sym"
model = AutoModelForCausalLM.from_pretrained(repo, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(repo)
messages = [{"role": "user", "content": "Hello!"}]
inputs = tokenizer.apply_chat_template(
messages, return_tensors="pt", add_generation_prompt=True
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
The recommended way to reproduce this exact quantization is via the auto-round-pipeline — the same orchestration tool that produced this artifact.
Set up a dedicated conda environment using the pipeline's setup.sh. Any of the install modes below produces an environment that can reproduce this quantization; pick the one that matches your goals:
git clone https://git.epicdynamic.com/auto-round-pipeline
cd auto-round-pipeline
# Pinned PyPI wheel (fastest; matches what this pipeline used by default):
bash setup.sh --pip-version 0.13.1
# Or build from intel/auto-round at the same tag (byte-identical reproducibility):
bash setup.sh --source-tag v0.13.1
# Intel Arc iGPU acceleration (e.g. Core Ultra 185H) — append to either of the above:
# ... --intel-xpu
# NVIDIA / AMD opt-in: --cuda / --rocm
The script prints the resulting conda env name (something like auto-round-pipeline-v0.13.1[-src][-xpu|-cuda|-rocm]) at the end.
Activate the env that setup.sh created, then invoke the runner with the same job filters that produced this artifact:
conda activate <env-name-printed-by-setup.sh>
python runner.py \
--model 'meta-llama/Llama-3.2-3B-Instruct' \
--quant 'INT4-gs64' \
--format auto_round \
--no-upload # drop this to also push to HuggingFace Hub
If you'd rather call auto-round directly without the orchestration wrapper, this is the exact call the pipeline made:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round import AutoRound
model_name = "meta-llama/Llama-3.2-3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_name)
bits, group_size, sym = 4, 64, True
autoround = AutoRound(
model, tokenizer,
bits=bits, group_size=group_size, sym=sym,
device_map="cpu",
nsamples=128, iters=200, seqlen=512, batch_size=4,
)
autoround.quantize_and_save("./AutoRound/meta-llama_Llama-3.2-3B-Instruct-auto_round-int4-gs64-sym", format="auto_round")
Recorded by the auto-round-pipeline at quantization time:
| Field | Value |
|---|---|
| Intel auto-round version | 0.13.1 |
| transformers version | 4.55.3 |
| torch version | 2.12.1+cpu |
| torch_dtype (load) | torch.bfloat16 |
| calibration device | cpu |
| calibration samples | 128 |
| tuning iterations | 200 |
| calibration seq len | 512 |
| calibration batch size | 4 |
| quantization duration | 20279.2s (338.0 min) |
| completed at (UTC) | 2026-07-09T23:39:53.679110+00:00 |
This quantized model comes with no warranty. It has been developed only for research purposes.