Downloads · 30 days
9
3% of all-time downloads
BareMetalAI/Llama-3.1-Nemotron-Nano-8B-W4A16-AWQ
Llama-3.1-Nemotron-Nano-8B-W4A16-AWQ is a text generation model from BareMetalAI. Use it when you need the model to write or continue text. It is set up for tensorrt-llm. The card lists the license as other.
Downloads · 30 days
9
3% of all-time downloads
All-time downloads
259
Public
Parameters
4.6B
5.8 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors5.8 GB · 100%
How the weights are stored.
U83.5B · 76%
From the Hugging Face model README
Built with Llama.
A 4-bit (W4A16-AWQ) quantization of nvidia/Llama-3.1-Nemotron-Nano-8B-v1, produced for the BareMetalRT consumer inference engine.
This is a modified, quantized derivative. The weights have been AWQ 4-bit
quantized; everything else (architecture, tokenizer, chat template) is unchanged
from the base model. It loads on the TensorRT-LLM PyTorch (_torch) backend,
which reads hf_quant_config.json and dispatches the W4A16-AWQ kernels — no
engine build step.
lm_head kept in higher precision.modelopt 0.37) INT4_AWQ_CFG,
AWQ scale search calibrated on cnn_dailymail, exported via export_hf_checkpoint.quant_algo: W4A16_AWQ).This model is a derivative of Llama-3.1-Nemotron-Nano-8B-v1 and is governed by the
NVIDIA Open Model License and the Llama 3.1 Community License Agreement
("Built with Llama"). See LICENSE, LICENSE.llama, and NOTICE. You must retain
attribution and the notice that the weights were modified (quantized).