North-Mini-Code-1.0-GGUF-Q6

North-Mini-Code-1.0 - GGUF

GGUF quantization of CohereLabs/North-Mini-Code-1.0 - a 30B-A3B (~3B active) sparse Mixture-of-Experts coding model from Cohere Labs.

Requires cohere2_moe llama.cpp support

This model uses the cohere2_moe architecture (leading dense FFN layer, explicit head_dim, hybrid full + 4096 sliding-window attention with a dense-first SWA phase). Mainline llama.cpp does not support it yet. Use the upstream pull request that adds full support - chat parser, tokenizer/EOG fixes, and native tool calls:

• llama.cpp PR #24260 (cohere2-moe). Build that branch until it merges.

Build:

git clone https://github.com/ggml-org/llama.cpp.git && cd llama.cpp
git fetch origin pull/24260/head:cohere2-moe && git checkout cohere2-moe
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build --config Release -j

On stock llama.cpp you will hit "wrong shape ... blk.0.attn_q" or "missing tensor 'blk.0.ffn_gate_inp.weight'".

Files

Run

./llama-server -m North-Mini-Code-1.0-Q6_K.gguf \
  --host 0.0.0.0 --port 8080 -c 40960 -ngl 99 --split-mode layer \
  --jinja --reasoning-format auto

Fits comfortably on 2x24 GB (e.g. dual RTX 3090). A single 24 GB card needs Q4/Q5.

Reasoning and tool use

This model is trained for interleaved thinking. With PR #24260's chat parser, --jinja exposes a separate reasoning_content field (think blocks) and native tool calls. For best agentic behaviour, feed both the thinking content AND tool calls back into the chat history on subsequent turns.

Sampling

Cohere's default is temperature=1.0, top_p=0.95 (tuned for chat). For long code generation that can fall into repetition loops, so for coding prefer:

--temp 0.3 --top-p 0.9 --repeat-penalty 1.1 --repeat-last-n 320
# plus DRY if your build has it:
--dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2

Notes

• This is an early / pre-release model (shared by Cohere for community testing). Expect rough edges in instruction following, tool calling, and long-session stability; quality will improve toward the official release.
• Architecture: decoder-only MoE - 128 experts, 8 active/token, 49 layers (layer 0 dense), hybrid attention.
• Parameters: 30B total, ~3B active. Context: base supports 256K; these quants validated at 40K.

License and credit

Apache-2.0, inherited from the base model. Full credit to Cohere and Cohere Labs for North-Mini-Code-1.0, and to the llama.cpp contributors behind PR #24260. Quantized with llama.cpp.