Downloads · 30 days
3
1% of all-time downloads
elmat234/qwen3-8b-aqua-Q4_K_M-gguf
qwen3-8b-aqua-Q4_K_M-gguf is a machine learning model from elmat234. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for gguf.
Single-file GGUF Q4KM build of Qwen/Qwen3-8B with the LoRA adapter elmat234/qwen3-8b-aqua-lora merged into the base weights. Quantized for inference with llama.cpp.
Downloads · 30 days
3
1% of all-time downloads
All-time downloads
230
Public
Repo size
5 GB
Likes
0
Public
Click a slice to open those files.
.gguf5 GB · 100%
From the Hugging Face model README
Single-file GGUF Q4_K_M build of Qwen/Qwen3-8B with the LoRA adapter
elmat234/qwen3-8b-aqua-lora
merged into the base weights. Quantized for inference with
llama.cpp.
W_merged = W_base + (B @ A) * (alpha / rank)convert_hf_to_gguf.py --outtype f16 -> fp16 GGUFllama-quantize ... Q4_K_M -> this fileTokenizer files were copied verbatim from Qwen/Qwen3-8B so the BPE
pre-tokenizer hash matches what llama.cpp expects. The chat template comes
from the adapter (it's what the model was trained against).
llama-cpp-pythonfrom llama_cpp import Llama
llm = Llama(
model_path="qwen3-8b-aqua-Q4_K_M.gguf",
n_gpu_layers=-1,
n_ctx=4096,
)
out = llm.create_chat_completion(
messages=[{"role": "user", "content": "Hello!"}],
max_tokens=128,
)
print(out["choices"][0]["message"]["content"])
~4.68 GB. Fits on a 16 GB GPU (T4) with full offload.