Downloads · 30 days
824
31% of all-time downloads
coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF
Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF is a text generation model from coolthor. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as openmdw-1.1.
Hardware-tuned GGUF pair for running Poolside Laguna S 2.1 on a single 22 GB RTX 2080 Ti with CPU/GPU hybrid offload:
Downloads · 30 days
824
31% of all-time downloads
All-time downloads
2.7K
Public
Repo size
73.2 GB
Likes
4
Public
Click a slice to open those files.
.gguf73.2 GB · 100%
From the Hugging Face model README
Hardware-tuned GGUF pair for running Poolside Laguna S 2.1 on a single 22 GB RTX 2080 Ti with CPU/GPU hybrid offload:
This is a modified quantized redistribution, not a fine-tune and not an official Poolside release. The target's existing Q4_K, Q6_K, router, norm, and bias tensors were byte-preserved; only the F16 attention tensors were changed.
| File | Size | Tensor layout | Purpose |
|---|---|---|---|
laguna-s-2.1-attnQ8.gguf | 67.6 GiB | 239 Q4_K, 48 Q6_K, 240 Q8_0, 287 F32 | Layer-aware target |
laguna-s-2.1-DFlash-Q4_K_M.gguf | 622 MiB | 45 Q4_K, 4 Q6_K, 27 F32 | Speculative-decoding draft |
The original official target GGUF is about 75 GB. Converting its 240 F16 attention tensors to Q8_0 reduced tensor payload by 2,506 MiB while preserving every already-quantized tensor without requantization.
Laguna S 2.1 is a 118B-A8B MoE. Its official Q4_K_M GGUF deliberately leaves attention in F16, while experts use Q4_K/Q6_K and sensitive router/norm/bias tensors remain F32. On a 22 GB Turing card, target weights are mostly CPU-resident, so reducing attention traffic lets llama.cpp fit more target data on GPU.
The target conversion was intentionally narrow:
AI Lab test system: EPYC 7402, 128 GB eight-channel DDR4 ECC, one modified RTX 2080 Ti 22 GB for Laguna, Poolside's laguna llama.cpp branch, 262K context, one slot.
| Prompt language | Official Q4_K_M | attention-Q8 | Change |
|---|---|---|---|
| English | 19.640 tok/s | 21.048 tok/s | +7.2% |
| Traditional Chinese | 19.733 tok/s | 21.259 tok/s | +7.7% |
| Prompt language | Official target + Q4 draft | attention-Q8 + Q4 draft | Change |
|---|---|---|---|
| English | 27.812 tok/s | 28.959 tok/s | +4.1% |
| Traditional Chinese | 18.906 tok/s | 19.498 tok/s | +3.1% |
Strict JSON, native tool-call safety, and executable min_window coding gates passed after target quantization. Laguna's known failure mode—spending the full completion budget in reasoning and returning no final content—was unchanged and should be handled at the agent layer.
Use Poolside's llama.cpp fork, laguna branch. Generic upstream llama.cpp does not yet contain the complete Laguna DFlash decoder contract required by this pair.
./build/bin/llama-server \
-m laguna-s-2.1-attnQ8.gguf \
-md laguna-s-2.1-DFlash-Q4_K_M.gguf \
--spec-type draft-dflash \
--spec-draft-n-max 4 \
-ngld 99 \
-c 262144 \
-ctk q4_0 -ctv q4_0 \
-ctkd f16 -ctvd f16 \
-t 24 -tb 24 -td 12 -tbd 12 \
-fa on \
-fit on -fitt 7680 -fitc 262144 \
--parallel 1 \
--jinja \
--reasoning on --reasoning-format deepseek \
--host 0.0.0.0 --port 8095
Keep draft KV in F16. Q4_0 draft KV reduced proposal acceptance below 1% and decode to roughly 5–6 tok/s. Q4_K_M draft weights and Q4_0 runtime KV are independent quantizations; their errors do not cancel.
Tune --spec-draft-n-max for the hardware and draft precision. Poolside's official BF16 example uses n=15. With this Q4_K_M draft on the tested Turing hybrid-offload system, n=4 was the best measured point. Larger values reduced acceptance enough to lose throughput.
The target KV can remain Q4_0; the precision warning applies specifically to the DFlash draft K/V cache (-ctkd and -ctvd).
This package was validated at 262,144 tokens. The native target was trained up to 1M context; see Poolside's official model card for the 1M YaRN override and quality caveats.
For tool/coding action lanes, consider disabling thinking or adding an empty-final repair. A short max_tokens cap alone can consume the whole budget inside reasoning_content before a final answer is emitted.
Distributed under the OpenMDW-1.1 License, inherited from Laguna S 2.1. A complete copy is included as LICENSE.md. The files in this repository were modified through quantization; Poolside does not endorse or maintain this redistribution.
☕ If this saved you GPU hours, you can buy me a coffee.
Made by Thor Lin. Full benchmarks, setup notes and the runs that failed are on ai-muninn.com. More quantized models: huggingface.co/coolthor.
This section documents release ea6cd8f8fd070f48a8c9bdbb8ee96838d1639cfd, inspected on 2026-11-14. Where the exact upstream commit used at build time is unknown, it is listed as not recorded rather than backfilled with today's upstream HEAD.
| Source repository | Revision used for this build | Role |
|---|---|---|
| poolside/Laguna-S-2.1-GGUF | not recorded | Immediate official imatrix Q4_K_M target GGUF |
| poolside/Laguna-S-2.1-DFlash | not recorded | Immediate DFlash draft input |
| poolside/Laguna-S-2.1 | not recorded | Original target lineage |
Tools and versions: Poolside llama.cpp fork, laguna branch; exact build-time commit/build not recorded.
For the target model, only the 240 F16 attention tensors were converted to Q8_0; the 239 Q4_K, 48 Q6_K and 287 F32 tensors were copied without changes. The draft model was converted to Q4_K_M. The published target recipe is copied below; the exact draft-conversion command is not recorded.
The existing records cover quantization and format conversion only; they don't mention any fine-tuning or merging by coolthor. That reflects the documentation, not an independent weight diff.
Recorded conversion command (published recipe; not executed for this provenance review):
llama-quantize \
--token-embedding-type Q4_K \
--output-tensor-type Q6_K \
--tensor-type '^blk\.[0-9]+\.attn_(q|k|v|output|gate)\.weight$=Q8_0' \
--tensor-type '^blk\.(0|1|2|3|4|5|8|11|14|17|20|23|26|29|32|35|38|41|42|43|44|45|46|47)\.ffn_down=Q6_K' \
--tensor-type 'ffn_down=Q4_K' \
--tensor-type 'ffn_gate=Q4_K' \
--tensor-type 'ffn_up=Q4_K' \
laguna-s-2.1-Q4_K_M.gguf laguna-s-2.1-attnQ8.gguf Q8_0 24
The values below are Hugging Face LFS object SHA-256 values for the pinned release above. File sizes and digests agree with pinned tree metadata. They identify full file contents, not LFS pointer text or Git blob IDs. Calibration imatrices are included when present.
| File | Bytes | SHA-256 |
|---|---|---|
laguna-s-2.1-DFlash-Q4_K_M.gguf | 652160160 | 872be33424231b9e70ad95caf5c080825d110a0a4411bd66fa13c379651b56b9 |
laguna-s-2.1-attnQ8.gguf | 72545160800 | 2d3d6f5a0a4fa534aad7bd1b67b00da59c2961a6c8bbe364ab8dbb376e842b54 |
Download the desired files from revision ea6cd8f8fd070f48a8c9bdbb8ee96838d1639cfd and compare sha256sum <file> (Linux) or shasum -a 256 <file> (macOS) with the table. If cloning with Git LFS, hash the downloaded weight files, not the LFS pointer files. A matching hash confirms your files match this release; it does not prove the weights came only from the listed base model with no other changes.
Checking how the weights were derived requires the exact upstream commit, tool versions, recipe and any calibration or training data, then re-running the build and diffing the weights with a format-aware tool (or comparing file hashes where serialization is deterministic). Because of the gaps listed below, this build can't be reproduced exactly today. Quantized files are expected to differ numerically from BF16 sources, and a raw file hash cannot establish a base-to-quant weight diff. No quantization rerun or weight-diff audit was performed for this provenance review.
Known gaps: Target/draft build-time upstream commits; Exact quantizer commit/build; Complete DFlash conversion invocation.
Sources: existing model card and HF metadata.