Qwen3.6-35B-A3B-8bit-paged

Qwen3.6-35B-A3B-8bit-paged

Qwen3.6-35B-A3B, quantized to 8 bits and packaged so that a machine which cannot hold the whole model in memory can still run it.

The model is Qwen/Qwen3.6-35B-A3B. This repository changes only how the weights are laid out on disk, not what they are.

Output is bit-identical to the same weights in their ordinary layout — the same bits, not merely close. The comparison is made while the repository is built, when both layouts still exist, and its result is recorded inside the repository.

How it works

A mixture-of-experts model only uses a few experts for any given token, but a normal checkpoint keeps all of them in memory all the time. Here the expert weights are stored separately, in a layout that allows any single expert to be read on its own.

At load time the engine looks at how much memory the machine actually has:

• Enough for the whole model — the experts are read into memory and the model runs exactly as the original does. Nothing is streamed and nothing is different.
• Not enough — the experts stay on disk, and each layer keeps a bounded cache of the ones it is using. The rest are read as the routing asks for them.

There is nothing to configure and no separate variant to choose. One repository serves both cases.

The repository is no larger for it: the expert tensors were moved out of the safetensors files, not duplicated.

What it costs

Streaming experts from disk is slower than having them in memory. If your machine has room for the model you pay nothing at all; if it does not, this is the trade that makes the model runnable rather than impossible.

Processing a long prompt for the first time is where the difference is most visible. Continued conversations are much less affected, because the prompt cache reuses work already done.

Requirements

Mac

Linux (gbx-lm v0.8.1 or later)

On a DGX Spark (128 GB) the model fits in memory and nothing is paged.

Memory is not a fixed figure for a paged build, and that is the point of one: it fills what fits and streams the rest from disk. On a 512 GB Mac Studio with room to spare this model settles at about 35 GB resident. A smaller machine holds less and reads more from disk -- slower, but it runs.

How much slower depends on how far the machine is from holding the experts, and on how fast its disk is. Each token routes to a few experts; the ones already in memory cost nothing to reach, and the ones that are not have to be read before that token can finish. A machine holding most of them waits rarely, one holding few waits often. We have not measured this across machine sizes and will not guess a figure: what we can say is that the model answers either way, and that the wait is the SSD's, not the model's.

Install

# upgrading? clear the previous version's unpack directory first
rm -rf ~/.libra/cache/onefile/gbx_lm

# on a DGX Spark, use gbx_lm-linux-arm64.tar.gz in place of gbx_lm-darwin-arm64.tar.gz
curl -fL -o gbx_lm-darwin-arm64.tar.gz 'https://github.com/GreenBitAI/gbx-lm/releases/latest/download/gbx_lm-darwin-arm64.tar.gz' \
  && tar -xzf gbx_lm-darwin-arm64.tar.gz gbx_lm \
  && mkdir -p "$HOME/.local/bin" \
  && mv gbx_lm "$HOME/.local/bin/gbx_lm" \
  && chmod +x "$HOME/.local/bin/gbx_lm"

gbx_lm -h

The Mac build is signed with a Developer ID and notarised, so macOS runs it without the usual detour for a downloaded binary.

command not found -- $HOME/.local/bin is not on your PATH:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc && source ~/.zshrc     # zsh
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bash_profile && source ~/.bash_profile   # bash

Killed: 9 (Mac) -- a previous version's files are still in the unpack directory, and macOS refuses to mix two builds. Run the rm -rf line above, then try again.

Run

gbx_lm --model GreenBitAI/Qwen3.6-35B-A3B-8bit-paged

That serves an OpenAI-compatible API on port 11688, which is its default. The weights download on first use into ~/.libra/cache/models; set HF_HOME to put them elsewhere, and HF_TOKEN if you meet the Hub's rate limits for anonymous downloads.

curl http://127.0.0.1:11688/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"GreenBitAI/Qwen3.6-35B-A3B-8bit-paged","messages":[{"role":"user","content":"Hello"}]}'

GBX_PAGING=off disables streaming entirely, for a machine that has the memory and wants to be certain it is not being used.

Images

The model reads images as well as text. Send them as OpenAI image_url parts -- an http(s) URL or a base64 data: URL:

curl http://127.0.0.1:11688/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"GreenBitAI/Qwen3.6-35B-A3B-8bit-paged","messages":[{"role":"user","content":[
        {"type":"text","text":"What is in this picture?"},
        {"type":"image_url","image_url":{"url":"https://example.com/photo.jpg"}}]}]}'

From the command line, --images takes one or more files: gbx_lm generate --model GreenBitAI/Qwen3.6-35B-A3B-8bit-paged --images photo.jpg --prompt "What is in this picture?" --max-tokens 2048.

From the command line

From gbx_lm v0.7.1, the binary also runs the model without a server -- one prompt, or an interactive chat:

GBX_QWEN35_MTP=on gbx_lm generate --model GreenBitAI/Qwen3.6-35B-A3B-8bit-paged --prompt "Hello" --max-tokens 2048
GBX_QWEN35_MTP=on gbx_lm chat --model GreenBitAI/Qwen3.6-35B-A3B-8bit-paged --max-tokens 2048

gbx_lm generate -h and gbx_lm chat -h list their options. Keep --max-tokens generous: the default is 100 for generate and 256 for chat, and the model's reasoning before it answers counts against it. Leave out GBX_QWEN35_MTP=on to run without the draft head.

Coding agents

The server speaks three wire protocols on the same port, so the tools that expect a hosted API can be pointed at this one:

Codex -- a provider in ~/.codex/config.toml:

[model_providers.gbx]
name = "gbx-lm"
base_url = "http://127.0.0.1:11688/v1"
wire_api = "responses"

and a profile in ~/.codex/gbx.config.toml:

model_provider = "gbx"
model = "GreenBitAI/Qwen3.6-35B-A3B-8bit-paged"
model_context_window = 262144

Claude Code -- ~/.claude/gbx.settings.json:

{
  "env": {
    "ANTHROPIC_BASE_URL": "http://127.0.0.1:11688",
    "ANTHROPIC_AUTH_TOKEN": "local",
    "ANTHROPIC_MODEL": "GreenBitAI/Qwen3.6-35B-A3B-8bit-paged",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "GreenBitAI/Qwen3.6-35B-A3B-8bit-paged"
  }
}

Both clients ask for a small model for their own background work, so every name in the settings has to be one this server is serving.

Limits

One request at a time. The expert cache is per-model state, so concurrent requests would interfere with each other. The engine detects this and serves without streaming instead — which means a machine that depends on streaming cannot serve concurrent requests.

Files

config.json, tokenizer*     as published upstream, with two additions
model.safetensors           every weight that is not an expert
experts.bin                 the expert weights
experts_index.json          their layout, and the build-time verification record

The two additions to config.json tell the engine which weight format these are and how the expert projections are quantized — the latter would otherwise be inferred from tensors that now live in experts.bin.

The draft head

The mtp/ folder carries the model's own multi-token prediction head, so speculative decoding works from this repository alone. It is off unless asked for:

GBX_QWEN35_MTP=on gbx_lm --model GreenBitAI/Qwen3.6-35B-A3B-8bit-paged

Measured 2026-10-13 with gbx_lm v0.8.0 on a 512 GB Mac Studio (M3 Ultra), weights resident, greedy, thinking off, 256 tokens a reply, decode timed from the first token. Short prompts: three each of code, prose and chat, three runs each, the median per kind. Long contexts: a slice of source code and a question about it, the median of three runs.

Every token the head proposes is checked by the model itself, so the reply is the model's own either way; the head only saves passes over the weights.