Downloads · 30 days
82
5% of all-time downloads
MikeKuykendall/deepseek-moe-16b-q4-k-m-cpu-offload-gguf
deepseek-moe-16b-q4-k-m-cpu-offload-gguf is a text generation model from MikeKuykendall. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
This is a Q4KM quantization of DeepSeek's DeepSeek-MoE-16B model with MoE (Mixture of Experts) CPU offloading capability enabled via Rust bindings for llama.cpp.
Downloads · 30 days
82
5% of all-time downloads
All-time downloads
1.7K
Public
Repo size
10.9 GB
Likes
4
Public
Click a slice to open those files.
.gguf10.9 GB · 100%
From the Hugging Face model README
This is a Q4_K_M quantization of DeepSeek's DeepSeek-MoE-16B model with MoE (Mixture of Experts) CPU offloading capability enabled via Rust bindings for llama.cpp.
Tested on Lambda Cloud GH200 (96GB VRAM, 480GB RAM, CUDA 12.8) with shimmy v1.6.0:
| Configuration | VRAM Usage | VRAM Saved | Reduction |
|---|---|---|---|
| All GPU (baseline) | 11.10 GB | - | - |
CPU Offload (--cpu-moe) | 1.86 GB | 9.24 GB | 83.2% |
Mixture of Experts models activate only a subset of parameters per token (sparse activation). This quantization includes Rust bindings that expose llama.cpp's MoE CPU offloading feature, allowing inactive experts to reside in system RAM instead of VRAM.
Note: The core MoE CPU offloading algorithm was implemented in llama.cpp (PR #15077, August 2025). This release provides Rust language bindings and production integration for that functionality.
# Download the model
huggingface-cli download MikeKuykendall/deepseek-moe-16b-q4-k-m-cpu-offload-gguf \
deepseek-moe-16b-Q4_K_M.gguf --local-dir ./models
# Run with CPU offloading (uses ~1.9 GB VRAM)
shimmy serve \
--model-dirs ./models \
--cpu-moe \
--bind 127.0.0.1:11435
# Run without offloading (uses ~11 GB VRAM)
shimmy serve \
--model-dirs ./models \
--bind 127.0.0.1:11435
use llama_cpp_2::context::params::LlamaContextParams;
use llama_cpp_2::llama_backend::LlamaBackend;
use llama_cpp_2::model::params::LlamaModelParams;
use llama_cpp_2::model::LlamaModel;
fn main() {
let backend = LlamaBackend::init().unwrap();
// Enable MoE CPU offloading
let model_params = LlamaModelParams::default()
.with_cpu_moe_all(); // Offload all inactive experts to CPU
let model = LlamaModel::load_from_file(
&backend,
"deepseek-moe-16b-Q4_K_M.gguf",
&model_params
).unwrap();
let ctx_params = LlamaContextParams::default()
.with_n_ctx(2048);
let mut ctx = model.new_context(&backend, ctx_params).unwrap();
// ... tokenize and generate as normal
}
# Build llama.cpp with CUDA support
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release
# Run with CPU offloading
./build/bin/llama-cli \
-m deepseek-moe-16b-Q4_K_M.gguf \
-p "Explain quantum computing" \
--cpu-moe
DeepSeek-MoE-16B uses a sparse Mixture of Experts architecture with 16 billion parameters. Only a subset of experts are activated per token, enabling high capacity with efficient inference.
The --cpu-moe flag (or with_cpu_moe_all() in Rust) tells llama.cpp to:
This dramatically reduces VRAM usage with a manageable performance trade-off.
Prompt: "Explain quantum computing in simple terms"
Response: (Generated coherent explanation suitable for Q4_K_M quantization quality)
If you use this model in your work, please cite the original DeepSeek paper:
@article{deepseek-moe,
title={DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models},
author={DeepSeek-AI},
year={2024}
}
License: Apache 2.0 (inherited from base model)
Quantized by: MikeKuykendall
Date: October 2025