Ornstein3.6-35B-A3B-GGUF

Ornstein3.6-35B-A3B-GGUF

GGUF quantizations of DJLougen/Ornstein3.6-35B-A3B — a Qwen 3.6 MoE fine-tune (35B total, ~3B active).

Support This Work

I'm a PhD student in visual neuroscience at the University of Toronto who also happens to spend way too much time fine-tuning, merging, and quantizing open-weight models on rented H100s and a local DGX Spark. All training compute is self-funded — balancing GPU costs against a student budget. If my uploads have been useful to you, consider buying a PhD student a coffee. It goes a long way toward keeping these experiments running.

Support on Ko-fi

---

Model info

• Architecture: Qwen3_5MoeForCausalLM (linear + full attention interleaved, Gated Delta Net)
• Parameters: 34.66 B total / ~3 B active (256 experts, 8 active per token)
• Context: 262,144 tokens
• Hidden size / layers: 2048 / 40
• Vocab: 248,320 tokens

All sub-8-bit quants were produced with an importance matrix (imatrix) computed from a mixed-domain multilingual calibration corpus (eaddario/imatrix-calibration → combined_all_medium): 200 chunks × 512 tokens = 102,400 calibration tokens.

Quant index

Choose a quant that fits in your RAM/VRAM with room for context. For MoE models quality degrades more sharply at low bit widths than for dense models of similar size — prefer Q4_K_M or higher if you have the memory.

Usage

llama.cpp

# Interactive chat
llama-cli -m Ornstein3.6-35B-A3B-Q4_K_M.gguf -cnv

# Single prompt
llama-cli -m Ornstein3.6-35B-A3B-Q5_K_M.gguf -p "Write a haiku about MoE routing."

# OpenAI-compatible server
llama-server -m Ornstein3.6-35B-A3B-Q4_K_M.gguf --host 0.0.0.0 --port 8080 -c 8192

Other runners

LM Studio, Ollama (via a Modelfile), koboldcpp, and text-generation-webui all load these GGUFs provided their bundled llama.cpp supports Qwen3_5MoeForCausalLM with Gated Delta Net.

Reproducing the quants

# 1. Convert safetensors → BF16 GGUF
python llama.cpp/convert_hf_to_gguf.py <model_dir> \
    --outtype bf16 --outfile Ornstein3.6-35B-A3B-BF16.gguf

# 2. Importance matrix
llama-imatrix \
    -m Ornstein3.6-35B-A3B-BF16.gguf \
    -f calibration.txt \
    -o imatrix.dat \
    --chunks 200 -c 512 -b 512 -ngl 99

# 3. Quantize (example)
llama-quantize --imatrix imatrix.dat \
    Ornstein3.6-35B-A3B-BF16.gguf \
    Ornstein3.6-35B-A3B-Q4_K_M.gguf Q4_K_M

License

Apache 2.0 — inherited from the Qwen 3.6 base release.