Gemory-26B-A4B-NVFP4-GGUF

Shadowplague/Gemory-26B-A4B-NVFP4-GGUF

Gemory-26B-A4B-NVFP4-GGUF is a text generation model from Shadowplague. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.

This is the first native GGUF release of UltimateIntent/Gemory-26B-A4B in NVIDIA Blackwell NVFP4 format (GGMLTYPENVFP4), accompanied by optimized Q80 and Q4KM Multi-Token Prediction (MTP) speculative decoding sidecars.

Downloads · 30 days

201

10% of all-time downloads

All-time downloads

2.1K

Public

Repo size

16.1 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.gguf16.1 GB · 100%

At a glance

Task: Text Generation
Library: gguf
License: apache-2.0
Access: Public
Created: Sep 6, 2026
Updated: Sep 23, 2026
SHA: 75af64a9

Try a prompt

Base models

UltimateIntent/Gemory-26B-A4B-GGUF

Model card

From the Hugging Face model README

Gemory-26B-A4B — Blackwell NVFP4 GGUF + Optimized MTP Sidecars (Q8_0 & Q4_K_M)

This is the first native GGUF release of UltimateIntent/Gemory-26B-A4B in NVIDIA Blackwell NVFP4 format (GGML_TYPE_NVFP4), accompanied by optimized Q8_0 and Q4_K_M Multi-Token Prediction (MTP) speculative decoding sidecars.

Designed specifically for local deployment in llama.cpp and kobold.cpp, this quantization achieves high mathematical fidelity and fine-grained micro-scaling (group size 16) while fitting strictly within 16 GB VRAM consumer GPUs (e.g., RTX 5060 Ti / RTX 5070 / RTX 5080) and scaling to blisteringly fast speculative decoding speeds on 24 GB+ cards (RTX 3090 / RTX 4090).

Model & Files in this Release

File	Size	Role	Description
`Gemory-26B-A4B-nvfp4.gguf`	15.3 GB	Primary LLM	Native Blackwell NVFP4 (E2M1 weights, `UE4M3` micro-block scales, BF16 routers).
`mtp-Q8_0.gguf`	441 MB	MTP Drafter (Recommended)	Highest accuracy drafter. Reaches 69.0% speculative acceptance and 44.8 t/s. Slashes memory by 46% vs stock BF16.
`mtp-Q4_K_M.gguf`	311 MB	MTP Drafter (Ultra-Lean)	Ultra-compact drafter for tight memory setups. Slashes memory by 64% vs stock BF16.

Empirical Speculative MTP Benchmark (RTX 5060 Ti 16GB)

Tested locally with Gemory-26B-A4B-nvfp4.gguf at 8,192 context (-fa on -ctk q4_0):

MTP Quant	File Size	Draft Tokens ($n$)	Acceptance Rate ($\alpha$)	Mean Streak	Net Generation Speed
🏆 `Q8_0`	`441 MB`	`2`	`69.0%`	`2.33 tokens`	`44.8 tokens/sec`
`Q8_0`	`441 MB`	`4`	`37.0%`	`2.43 tokens`	`29.7 tokens/sec`
`Q8_0`	`441 MB`	`3`	`18.8%`	`1.55 tokens`	`24.0 tokens/sec`
`Q8_0`	`441 MB`	`5`	`22.5%`	`2.06 tokens`	`21.3 tokens/sec`
`Q4_K_M`	`311 MB`	`2`	`35.0%`	`1.70 tokens`	`36.1 tokens/sec`
`Q4_K_M`	`311 MB`	`3`	`30.2%`	`1.89 tokens`	`32.5 tokens/sec`
`Q4_K_M`	`311 MB`	`5`	`18.6%`	`1.89 tokens`	`23.2 tokens/sec`
`Q4_K_M`	`311 MB`	`4`	`20.8%`	`1.79 tokens`	`8.4 tokens/sec`

Key Finding: Setting --spec-draft-n-max 2 is universally optimal. Predicting 1–2 tokens ahead achieves a 69% acceptance rate on Q8_0, while pushing to 3–5 tokens compounds prediction error and lowers overall throughput.

Why NVFP4 over Standard Q4_0?

Logarithmic Weight Representation (E2M1): Unlike linear integer quantization (Q4_0), FP4 non-linear spacing matches the normal Gaussian bell-curve of neural network weights, placing high density around zero.
Micro-Scaling (Group Size 16): 1 scale factor per 16 weights (compared to 32 in standard Q4_0), preserving emotional nuance, creative vocabulary, and character card adherence in SillyTavern without repetitive cliché loops.

How to Run

Mode 1: Standard Low-VRAM (No MTP)

Optimal for 16GB GPUs (RTX 5060 Ti, RTX 4070 Ti, RTX 4080) or maximum KV context depth:

llama-server \
  -m Gemory-26B-A4B-nvfp4.gguf \
  -ngl 99 \
  -c 8192 \
  --host 0.0.0.0 --port 8080

Mode 2: Speculative MTP Turbo Mode (Recommended)

Accelerated generation using the high-acceptance Q8_0 draft head:

llama-server \
  -m Gemory-26B-A4B-nvfp4.gguf \
  -md mtp-Q8_0.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -ngl 99 \
  -ngld 99 \
  -fa on \
  -ctk q4_0 -ctv q4_0 \
  -c 8192 \
  --host 0.0.0.0 --port 8080

Using in SillyTavern / Open WebUI

In SillyTavern, set API to Text Completion (LLaMA.cpp) or Chat Completion (OpenAI compatible) pointing to http://localhost:8080.

Acknowledgments & Credits

Base model training & fine-tuning: UltimateIntent
Quantized, benchmarked, and converted with custom patches by Shadowplague

Task: Text Generation
Library: gguf
License: apache-2.0
Languages: en
Created: Sep 6, 2026
Updated: Sep 23, 2026