Downloads · 30 days
8
16% of all-time downloads
soyrsoyr/TinyLlama-1.1B-Chat-HIGGS-W6.0avg
TinyLlama-1.1B-Chat-HIGGS-W6.0avg is a machine learning model from soyrsoyr. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Mixed-precision quantized model using HIGGS (Heuristic ILP-Guided Grouped Scheme) with ModelFreePtqConverter integration.
Downloads · 30 days
8
16% of all-time downloads
All-time downloads
51
Public
Parameters
1.1B
1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1 GB · 100%
How the weights are stored.
I32969M · 88%
From the Hugging Face model README
Mixed-precision quantized model using HIGGS (Heuristic ILP-Guided Grouped Scheme) with ModelFreePtqConverter integration.
pack-quantized (compressed-tensors)HIGGS used integer linear programming (ILP) to automatically assign optimal quantization precision per layer based on MSE sensitivity:
This achieves a 6.0 bit average while preserving quality on sensitive layers.
from vllm import LLM, SamplingParams
model = LLM(model="soyrsoyr/TinyLlama-1.1B-Chat-HIGGS-W6.0avg")
prompts = ["What is the capital of France?"]
outputs = model.generate(prompts, SamplingParams(temperature=0.7, max_tokens=128))
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"soyrsoyr/TinyLlama-1.1B-Chat-HIGGS-W6.0avg",
device_map="auto",
torch_dtype="auto"
)
tokenizer = AutoTokenizer.from_pretrained("soyrsoyr/TinyLlama-1.1B-Chat-HIGGS-W6.0avg")
This model was created to validate the integration between HIGGS and ModelFreePtqConverter (llm-compressor PR #2935 + PR #3028).
from llmcompressor.entrypoints.higgs import get_higgs_config
from llmcompressor.entrypoints.model_free import ModelFreePtqConverter
from compressed_tensors.entrypoints.convert import convert_checkpoint
# Step 1: HIGGS ILP optimization
config = get_higgs_config(
model_stub="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
candidate_schemes=["W4A16", "W8A16"],
targets="Linear",
ignore=["lm_head", "re:.*embed_tokens"],
target_avg_bitwidth=6.0,
device="cuda:0",
)
# Step 2: Apply quantization with ModelFreePtqConverter
converter = ModelFreePtqConverter(config=config)
convert_checkpoint(
model_stub="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
save_directory="./TinyLlama-HIGGS-W6.0avg",
converter=converter,
)
Same as base model: Apache 2.0
@software{higgs2025,
title={HIGGS: ILP-Based Mixed-Precision Quantization},
author={HDCharles and contributors},
year={2025},
url={https://github.com/vllm-project/llm-compressor}
}