Downloads · 30 days
1
5% of all-time downloads
TheHouseOfTheDude/L3.3-70B-Animus-V12.4_Compressed-Tensors
L3.3-70B-Animus-V12.4_Compressed-Tensors is a text generation model from TheHouseOfTheDude. Use it when you need the model to write or continue text. It is set up for vllm. The card lists the license as other.
This repository provides quantized runtime builds of Darkhn/L3.3-70B-Animus-V12.4 (a Llama-3.3-70B–based finetune), repackaged for vLLM using the compressed-tensors format.
Downloads · 30 days
1
5% of all-time downloads
All-time downloads
22
Public
Repo size
190 GB
Likes
0
Public
Click a slice to open those files.
.md6.5 KB · 81%
From the Hugging Face model README
This repository provides quantized runtime builds of
Darkhn/L3.3-70B-Animus-V12.4 (a Llama-3.3-70B–based finetune), repackaged for vLLM using the compressed-tensors format.
TL;DR
- Quantized with W4A16 (INT4 weights / A16 activations) and W8A16 (INT8 weights / A16 activations)for vLLM via
--quantization compressed-tensors.- Same calibration recipe as the recent cards: 512 chat samples, 2048 max sequence length, dataset
neuralmagic/LLM_compression_calibration(messages rendered with the model’s chat template).- Weight-only AWQ, group size 128, symmetric quant;
lm_headkept high-precision; exported withsave_compressed=True.
The
mainbranch is a landing page (model card + links). Runnable artifacts live in per-quant branches.
Quick links
*.safetensors) + index (model.safetensors.index.json)config.json with compressed-tensors metadata (weight_format, quantization, quantization_config, etc.)tokenizer.json, tokenizer.model, merges/vocab as applicable)chat_template.jinja (inherits the finetune’s chat style)Exact file lists may differ between branches — see Files and versions for each revision.
Method / flow
llmcompressor oneshot pipeline with an AWQModifier (weight-only quantization).Targets / exclusions
lm_head (kept high-precision).Weights / grouping
num_bits=4, type="int", symmetric=True)"group" with group_size=128 (Marlin-friendly)Calibration dataset & preprocessing
neuralmagic/LLM_compression_calibration, split trainmessages list is rendered via tokenizer.apply_chat_template(..., tokenize=False), then tokenized with:
max_length=2048, truncation=True, padding=False, add_special_tokens=FalseCompression call
oneshot(..., max_seq_length=2048, num_calibration_samples=512, tokenizer=tokenizer) on the preprocessed datasetExport for vLLM
save_compressed=True so vLLM loads the compressed-tensors runtime layout directlyThis mirrors the group size, sequence length, sample count, and dataset parameters used in the previous quantization script you referenced.
Install vLLM (recent version recommended):
pip install vllm
Serve (adjust to your hardware):
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
vllm serve TheHouseOfTheDude/L3.3-70B-Animus-V12.4_Compressed-Tensors \
--quantization compressed-tensors \
--tensor-parallel-size 8 \
--max-model-len 2048 \
--gpu-memory-utilization 0.70 \
--dtype bfloat16
Example Chat Completions:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "TheHouseOfTheDude/L3.3-70B-Animus-V12.4_Compressed-Tensors",
"messages": [
{"role":"system","content":"You are Animus — helpful, precise, and safe."},
{"role":"user","content":"Draft a character-driven opening scene in under 250 words."}
],
"max_tokens": 512,
"temperature": 0.7,
"top_p": 0.95
}'
Note:
compressed-tensorsis a vLLM runtime format. Loading directly with vanilla 🤗 Transformers is not supported.
For Transformers, use a compatible quant (e.g., GPTQ/AWQ export) or the full-precision finetune.
This package follows the finetuned parent’s chat conventions. If a chat_template.jinja is present, libraries that support apply_chat_template will automatically format messages.
Guidelines:
This quantization:
Apply appropriate content filters / policies for your deployment context.
--max-model-len and batch size.