Downloads · 30 days
13
14% of all-time downloads
EvanOLeary/laguna-xs2-dense-k8-cuda-sft-int4-hqq
laguna-xs2-dense-k8-cuda-sft-int4-hqq is a text generation model from EvanOLeary. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Quantized variant of EvanOLeary/laguna-xs2-dense-k8-cuda-sft. 4-bit weight quantization via HQQ (Half-Quadratic Quantization). Pure-PyTorch, data-free, no calibration set required.
Downloads · 30 days
13
14% of all-time downloads
All-time downloads
93
Public
Repo size
2.3 GB
Likes
0
Public
Click a slice to open those files.
.pt2.3 GB · 100%
From the Hugging Face model README
Quantized variant of
EvanOLeary/laguna-xs2-dense-k8-cuda-sft. 4-bit weight quantization via HQQ (Half-Quadratic Quantization). Pure-PyTorch, data-free, no calibration set required.
| bf16 (base) | HQQ 4-bit | |
|---|---|---|
| Weight file | 5.99 GB | 2.28 GB (38% of bf16) |
| VRAM (loaded) | 6.00 GB | 2.28 GB |
| Bits/param (effective) | 16 | ~4.5 (incl. group scales + zeros) |
302 tokens generated in 51.7s = 5.8 tok/s.
Prompt: "Write a CUDA kernel that computes ReLU (max(x, 0)) on a float array in-place. Include the kernel and a host-side launcher."
The model produced a complete torch::extension-style CUDA kernel with templated relu_kernel,
AT_DISPATCH_FLOATING_TYPES dispatch, a relu_forward host launcher, and a PYBIND11_MODULE
entrypoint suitable for torch.utils.cpp_extension.load_inline.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "EvanOLeary/laguna-xs2-dense-k8-cuda-sft-int4-hqq"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
from hqq.models.hf.base import AutoHQQHFModel
model = AutoHQQHFModel.from_quantized(repo, compute_dtype=torch.bfloat16, device="cuda")
msgs = [{"role":"user","content":"Write a CUDA kernel for elementwise sigmoid on a float array."}]
text = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
ids = tok(text, return_tensors="pt", add_special_tokens=False).input_ids.to(model.device)
out = model.generate(ids, max_new_tokens=400, do_sample=False, pad_token_id=tok.pad_token_id)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))
| Stage | Repo |
|---|---|
| Teacher MoE | poolside/Laguna-XS.2 |
| Dense recon (V1) | EvanOLeary/laguna-xs2-dense-k8-recon |
| CUDA-SFT (bf16 base for this quant) | EvanOLeary/laguna-xs2-dense-k8-cuda-sft |
| This: HQQ 4-bit | EvanOLeary/laguna-xs2-dense-k8-cuda-sft-int4-hqq |