Downloads · 30 days
129
7% of all-time downloads
NotoriousH2/kanana-2-30b-a3b-instruct-2601-awq-w4a16
kanana-2-30b-a3b-instruct-2601-awq-w4a16 is a text generation model from NotoriousH2. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This is the AWQ quantized version of kakaocorp/kanana-2-30b-a3b-instruct-2601.
Downloads · 30 days
129
7% of all-time downloads
All-time downloads
1.9K
Public
Parameters
4.5B
16.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors16.6 GB · 100%
How the weights are stored.
I3230.1B · 97%
From the Hugging Face model README
This is the AWQ quantized version of kakaocorp/kanana-2-30b-a3b-instruct-2601.
This model was quantized using LLM Compressor with the following configuration:
lm_head: Output layermlp.gate: MoE router gatesmlp.shared_expert_gate: Shared expert gatesfrom llmcompressor.modifiers.awq import AWQModifier
recipe = [
AWQModifier(
ignore=["lm_head", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$"],
scheme="W4A16",
targets=["Linear"],
),
]
from vllm import LLM, SamplingParams
model = LLM(
model="NotoriousH2/kanana-2-30b-a3b-instruct-2601-awq-w4a16",
trust_remote_code=True,
)
sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
output = model.generate("고혈압 환자의 식이요법에 대해 설명해주세요.", sampling_params)
print(output[0].outputs[0].text)
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"NotoriousH2/kanana-2-30b-a3b-instruct-2601-awq-w4a16",
torch_dtype="auto",
device_map="auto",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
"NotoriousH2/kanana-2-30b-a3b-instruct-2601-awq-w4a16",
trust_remote_code=True,
)
messages = [
{"role": "user", "content": "고혈압 환자의 식이요법에 대해 설명해주세요."}
]
input_ids = tokenizer.apply_chat_template(
messages,
return_tensors="pt",
add_generation_prompt=True,
).to(model.device)
output = model.generate(input_ids, max_new_tokens=512)
print(tokenizer.decode(output[0], skip_special_tokens=True))