Downloads · 30 days
32
21% of all-time downloads
P4pps3n/Laguna-S-2.1-exl3-2.65bpw-hq-DFlash-8bpw
Laguna-S-2.1-exl3-2.65bpw-hq-DFlash-8bpw is a text generation model from P4pps3n. Use it when you need the model to write or continue text. It is set up for exllamav3. The card lists the license as openmdw-1.1.
This repository contains a low-VRAM EXL3 deployment bundle for poolside/Laguna-S-2.1, paired with a separately quantized EXL3 version of poolside/Laguna-S-2.1-DFlash.
Downloads · 30 days
32
21% of all-time downloads
All-time downloads
152
Public
Parameters
20B
41.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors41.2 GB · 100%
How the weights are stored.
I1619.5B · 97%
From the Hugging Face model README
This repository contains a low-VRAM EXL3 deployment bundle for poolside/Laguna-S-2.1, paired with a separately quantized EXL3 version of poolside/Laguna-S-2.1-DFlash.
The tested deployment target is one concurrent sequence with a 160,000-token Q8 cache on exactly two RTX 3090 GPUs. It uses ExLlamaV3 native tensor parallelism, DFlash model-draft speculative decoding, and no CPU MoE offload.
| Component | Requested | Measured safetensors bpw | Size | Output head |
|---|---|---|---|---|
| Laguna S 2.1 target | 2.65 bpw HQ | 2.724383 | 40,035,486,872 bytes | 6.01 bpw |
| Laguna S 2.1 DFlash | 8.0 bpw | 8.0172 | 1,117,378,345 bytes | N/A |
The target contains 11 safetensors shards. The DFlash checkpoint is stored in
draft/ and uses draft_mode: model; it is not an integrated MTP checkpoint.
For Hugging Face schema compatibility, root config.json reports integer
quantization_config.bits: 3. The exact nominal target, allocation, and packed
tensor metadata remain in quantization_config.json. The model is not a 3-bpw
conversion; its measured whole-safetensors rate is 2.724383 bpw.
Laguna requires softplus per-head attention gating. ExLlamaV3 1.4.1 omits
gate_softplus from SlidingAttention.tp_export(), causing TP-imported modules
to silently disable the gate and generate repetitive garbage even though model
loading succeeds.
Apply runtime-patches/exllamav3-laguna-tp-gate-softplus.patch to the exact
ExLlamaV3 1.4.1 checkout before serving with native tensor parallelism. The
patch preserves self.gate_softplus in the exported constructor arguments.
The fix was isolated using a known coherent Laguna 3.00-bpw checkpoint:
Do not treat a successful load as sufficient verification. Run a deterministic generation test before deployment.
The provided tabby_config.example.yml uses:
CUDA_VISIBLE_DEVICESSet model_dir and draft_model_dir to match your download location. The draft
is inside this repository under draft/.
Hardware: two RTX 3090 GPUs connected through PCIe PHB topology, without an active NVLink path.
| Test | Result |
|---|---|
| Three cold starts | 3/3 passed |
| Deterministic tool calls | 2/2 passed |
| Short coherent generation | 106.89 tok/s |
| Short DFlash acceptance | 91 / 196, 46.43% |
| 154,057-token prompt | Passed, exact CONTEXT_OK response |
| Long prefill | 954.34 tok/s, 162.02 s end to end |
| Peak RTX 3090 #1 | 24,106 MiB used, 18 MiB free |
| Peak RTX 3090 #2 | 23,033 MiB used, 1,091 MiB free |
| HumanEval pass@1 | 76.83% (126 / 164) |
HumanEval used the official OpenAI 164-problem evaluator, one deterministic sample per problem, temperature 0, thinking disabled, and a 1024-token output limit. Generated code was evaluated with 8 workers and a 10-second test timeout.
The following values come from TabbyAPI server metrics after a fresh server restart. Each row is one cold-prefix run with temperature 0 and thinking disabled. Decode speed includes the separately quantized DFlash draft.
| Encoded context | Client TTFT | Prefill | Decode | DFlash acceptance |
|---|---|---|---|---|
| 1,015 | 2.32 s | 622.70 tok/s | 31.60 tok/s | 6.50% |
| 7,978 | 8.65 s | 1,053.90 tok/s | 38.79 tok/s | 9.71% |
| 31,838 | 33.99 s | 1,037.41 tok/s | 26.23 tok/s | 7.56% |
| 63,650 | 70.84 s | 974.28 tok/s | 20.82 tok/s | 7.52% |
| 127,242 | 143.26 s | 924.12 tok/s | 21.30 tok/s | 13.89% |
| 149,484 | 168.55 s | 908.33 tok/s | 14.29 tok/s | 8.46% |
DFlash acceptance is prompt dependent. A separate short coherent coding test reached 46.43% acceptance and 106.89 tok/s; the repetitive long-context corpus above produced substantially lower acceptance.
This exact 160K Q8 target-plus-DFlash profile passed the full operational prompt and all three cold starts, but GPU 0 reached only 18 MiB free at peak. This is a very small safety margin. Other drivers, display attachment, CUDA allocator behavior, or background GPU consumers can cause OOM. Stop unrelated workloads before loading, and reduce context if your placement differs.
TabbyAPI allocates cache in pages and reserves generation/speculation capacity. A request with nearly 160,000 prompt tokens can therefore be rejected even when the configured cache pool is 160,000 tokens. The validated operational prompt contained 154,057 encoded tokens and allocated the full configured pool.
Use the ExLlamaV3 1.4.1 CUDA wheel matching your Python, CUDA, PyTorch, and ABI,
plus a compatible TabbyAPI checkout. Apply the included patch to the ExLlamaV3
source checkout and ensure that checkout is first on PYTHONPATH.
git clone --branch v1.4.1 https://github.com/turboderp-org/exllamav3.git
cd exllamav3
git apply /models/Laguna-S-2.1-exl3-2.65bpw-hq-DFlash-8bpw/runtime-patches/exllamav3-laguna-tp-gate-softplus.patch
export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
Restrict CUDA visibility to the intended two RTX 3090 devices before starting TabbyAPI. Prefer GPU UUIDs rather than numeric indexes.
export CUDA_DEVICE_ORDER=FASTEST_FIRST
export CUDA_VISIBLE_DEVICES="GPU-UUID-3090-A,GPU-UUID-3090-B"
export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"
python /path/to/tabbyAPI/main.py --config /models/Laguna-S-2.1-exl3-2.65bpw-hq-DFlash-8bpw/tabby_config.example.yml
poolside/Laguna-S-2.100af5a51782109b587a3b3bbf11875e566036fa7poolside/Laguna-S-2.1-DFlashb0486d1586daa0d56435c508108171fc1c8daff9The model materials retain the upstream OpenMDW-1.1 license. See LICENSE.md.
The one-line runtime patch is provided only to preserve the upstream model
architecture flag during native tensor-parallel import.