Downloads · 30 days
8
2% of all-time downloads
nagix999/gemma-3-27b-it-gptq-ko-calibration
gemma-3-27b-it-gptq-ko-calibration is a text generation model from nagix999. Use it when you need the model to write or continue text. The card lists the license as gemma.
This repository provides a quantized version of the google/gemma-3-27b-it model. This quantization was performed to optimize inference efficiency and reduce VRAM usage while maintaining high performance, especially fo…
Downloads · 30 days
8
2% of all-time downloads
All-time downloads
387
Public
Parameters
27.4B
17 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors17 GB · 100%
How the weights are stored.
I3225.6B · 93%
From the Hugging Face model README
This repository provides a quantized version of the google/gemma-3-27b-it model. This quantization was performed to optimize inference efficiency and reduce VRAM usage while maintaining high performance, especially for Korean language tasks.
The model was quantized using Generative Pre-trained Transformer Quantization(GPTQ) with a focus on preserving the linguistic nuances of Korean.
| Layer | Module | Loss | Damp | Time (s) |
|---|---|---|---|---|
| 61 | self_attn.k_proj | 0.0000883157 | 0.05000 | 13.501 |
| 61 | self_attn.q_proj | 0.0001169325 | 0.05000 | 13.603 |
| 61 | self_attn.v_proj | 0.0000929558 | 0.05000 | 13.684 |
| 61 | self_attn.o_proj | 0.0000056512 | 0.05000 | 1.266 |
| 61 | mlp.up_proj | 0.0004954723 | 0.05000 | 3.865 |
| 61 | mlp.gate_proj | 0.0004981519 | 0.05000 | 3.886 |
| 61 | mlp.down_proj | 0.0000036350 | 0.05000 | 8.101 |
You can easily deploy this model using the vllm-openai Docker image to serve an OpenAI-compatible API.
docker run --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model nagix999/gemma-3-27b-it-gptq-ko-calibration \
--served-model-name gemma3-27b-it-gptq
--quantization gptq_marlin \
--max-model-len 8192 \
--dtype bfloat16
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="gemma3-27b-it-gptq",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "한국어와 영어로 LLM에 대해 설명해줘."}
]
)
print(response.choices[0].message.content)
This model is a derivative of google/gemma-3-27b-it and is subject to the Gemma Terms of Use. By downloading or using this model, you agree to the terms and conditions specified by Google.
For more details, please visit the Gemma License Agreement.