Downloads · 30 days
0
bare-corporation/bare-ai-qwen-176b-edge-harness
bare-ai-qwen-176b-edge-harness is a text generation model from bare-corporation. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
The 176B-parameter sparse MoE model that runs on a consumer desktop — no GPU cluster required.
Downloads · 30 days
0
Access
Public
Updated Aug 28, 2026
Repo size
759 KB
Likes
0
Public
Click a slice to open those files.
.pdf759 KB · 98%
From the Hugging Face model README
The 176B-parameter sparse MoE model that runs on a consumer desktop — no GPU cluster required.
This repository is the deployment harness and documentation for running a
1-bit quantized Qwen3.8-Flash-Next (176B Sparse Mixture-of-Experts) model
on a single consumer edge node, powered by llama-server with NVMe mmap
streaming.
We asked a question most people assumed had an obvious answer: Can a 176B parameter model run on a Ryzen 9 CPU with 68GB of RAM?
The conventional wisdom says no — you need multiple 80GB GPUs, NVLink, and a six-figure power budget. We proved otherwise by exploiting the two structural facts that define modern MoE architectures:
The result: 6.31 Tokens/second at 176B parameters on a CPU-only edge node — nearly 2× faster than a much smaller dense 27B model running on the same hardware.
| Component | Specification |
|---|---|
| CPU | AMD Ryzen 9 (consumer desktop-class) |
| RAM | 68 GB (total system limit) |
| Storage | PCIe Gen4 NVMe (mmap streaming target) |
| GPU | None required — CPU-only inference |
| OS | Linux (Bare-AI edge node, bare-dc fleet) |
The model weights are memory-mapped directly from NVMe (mmap) and
streamed into the CPU on demand. We never load the full 176B of weights into
RAM — we only pull the active experts' weights across the bus, per token.
On CPU inference, token generation is dominated by how fast weights can cross the memory bus. FLOPs are cheap; moving gigabytes of weights per token is expensive.
For a dense model, every token requires streaming all of the model's weights across the memory bus:
Dense 27B: 27B parameters × 1 byte/param (1-bit) ≈ 27 GB streamed per token
For a sparse MoE, the router only activates a small subset of experts per token. The Qwen3.8-Flash-Next router activates just 6B parameters per token:
Sparse MoE 176B: 6B parameters × 1 byte/param ≈ 6 GB streamed per token
| Model | Active params/token | Bus traffic/token | Tokens/sec |
|---|---|---|---|
| Dense 27B (1-bit) | 27B | ~27 GB | 3.76 TPS |
| Sparse MoE 176B (1-bit) | 6B | ~6 GB | 6.31 TPS |
The 176B MoE is ~1.7× faster than the 27B dense model, on the same CPU, same RAM, same NVMe.
Why? The dense model saturates the memory bus every token — it is bus-latency-bound. The sparse MoE streams only a fraction of its weights per token, so it keeps the bus far below saturation and wastes almost no time waiting on memory. On a memory-constrained CPU edge node, sparsity is not a nice-to-have — it is the difference between 3 TPS and 6 TPS.
A dense 70B would be catastrophic on this hardware: ~70 GB streamed per token would pin the memory bus at 100% and produce single-digit TPS if it even fit in RAM. The MoE gives us frontier-class parameter count with sub-dense-model bandwidth cost — the best of both worlds for edge inference.
Our benchmarks show a hard empirical floor for deep technical QA fidelity: models under ~3B parameters suffer severe hallucination on multi-step technical reasoning and domain-specific questions.
Below the parametric floor, the model does not have enough capacity to represent the long-range dependencies and structured reasoning that production-grade technical answers require — it confidently fabricates plausible-sounding nonsense instead.
This forces a strategic conclusion for edge agents:
Small dense models are a dead end for high-fidelity agents. The only viable path to frontier-quality reasoning on consumer hardware is a large sparse MoE, where total capacity is enormous but per-token bandwidth cost stays low.
The 176B MoE harness is the direct realization of that strategy: massive parameter capacity, hallucination-resistant reasoning, and a bandwidth profile that fits a single consumer CPU.
| Setting | Value |
|---|---|
| Quantization | 1-bit (Qwen3.8-Flash-Next quant) |
| Swap buffer | 16 GB |
| Context window | 131,072 tokens (131k) |
| KV cache | 8-bit quantized |
| Weight access | NVMe mmap streaming (llama-server) |
| Inference backend | llama-server (llama.cpp) |
| Compute | CPU-only (no GPU) |
# 1. Clone this harness (or download the quantized weights via huggingface_hub)
git clone https://huggingface.co/bare-corporation/bare-ai-qwen-176b-edge-harness
cd bare-ai-qwen-176b-edge-harness
# 2. Launch llama-server with NVMe mmap streaming
llama-server
--model qwen3.8-flash-next-1bit.gguf
--ctx-size 131072
--cache-type-k q8_0
--cache-type-v q8_0
--threads nproc
--no-mmap false
--host 0.0.0.0 --port 8080
# 3. Query
curl http://localhost:8080/v1/chat/completions
-H "Content-Type: application/json"
-d '{"model":"qwen3.8-flash-next-1bit","messages":[{"role":"user","content":"Explain the MoE vs dense tradeoff on CPU."}]}'
Tune for your node: if your NVMe is Gen3 or your CPU has fewer cores, drop
--ctx-sizeto 65536 and increase swap to 32 GB to preserve TPS.
| File | Purpose |
|---|---|
README.md | This model card |
upload_to_hf.py | One-shot uploader for this release directory |
llm_benchmark_detailed.py | Benchmark harness used to produce the TPS figures |
TPS figures were measured with llm_benchmark_detailed.py on the reference
node (Ryzen 9, 68 GB RAM, PCIe Gen4 NVMe) using identical context and sampling
parameters for both models. YMMV based on NVMe speed, memory speed (DDR4 vs
DDR5), and CPU core count.
Proof of Work video on you tube; https://youtu.be/IR8CooObuV8?si=FOa21DgdZEpp2lCK
Bare-AI — sovereign edge intelligence. No GPU clusters were harmed in the making of this model card.