Hy3-colibri-int4

Hy3 — colibrì int4 container (~142 GB)

Pre-converted weights for colibrì — a pure-C engine that runs huge MoE models on consumer hardware by keeping the dense stack resident in RAM and streaming routed experts from disk.

This repo is the first public Colibri-format checkpoint for Tencent Hy3. It lets you run Hy3's 295B-parameter MoE (21B active per token) on a machine with modest RAM and a fast local NVMe drive, instead of an 8×GPU datacenter setup.

⚠️ This is NOT a GGUF / AWQ / GPTQ / MLX / vLLM model. It uses colibrì's own safetensors container: for each quantized weight, name (packed int4/int8) + name.qs (F32 per-row scales), with packing math bit-identical to the engine's C kernels. It only works with the colibrì hy3 engine (see the Hy3 port in this conversion fork).

What we contribute

The upstream tencent/Hy3 weights ship for BF16 / FP8 inference stacks (vLLM, SGLang, Transformers). Community quants (GGUF, AWQ, etc.) target other runtimes and cannot be loaded by colibrì.

This release adds the missing disk-streaming Colibri container plus the runtime pieces needed to use it:

Bottom line: same model family as tencent/Hy3, but repackaged so a single NVMe + ~16–64 GB RAM machine can run it via expert streaming — the same idea as jlnsrk/GLM-5.2-colibri-int4, applied to Hy3.

About Hy3 (from the base model)

Hy3 is Tencent's 295B-parameter Mixture-of-Experts instruct model with 21B activated parameters per token, developed by the Tencent Hy Team. See the official model card for benchmarks, agent capabilities, and deployment recipes for GPU stacks.

Usage

1. Get the engine (Hy3 port)

git clone https://github.com/JustVugg/colibri.git   # or your Hy3 port fork with c/hy3.c
cd colibri/c
./setup.sh    # builds hy3 + self-test on hy3_tiny fixture

2. Download this repo

Store on a fast local disk (NVMe, native ext4). Avoid WSL /mnt/c (9p) or network mounts — random expert reads are disk-bound.

hf download <your-username>/Hy3-colibri-int4 --local-dir /path/to/hy3_i4

3. Chat

COLI_MODEL=/path/to/hy3_i4 ./coli chat --ram 56 --topp 0.7

Recommended sampling for int4 streaming (differs from the official GPU defaults of temperature=0.9, top_p=1.0 on tencent/Hy3):

• --temp 0.7 --topp 0.7 — colibrì defaults tuned to reduce quantization tail noise

4. OpenAI-compatible API

COLI_MODEL=/path/to/hy3_i4 ./coli serve --ram 56 --host 127.0.0.1 --port 8000

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="hy3-colibri",
    messages=[{"role": "user", "content": "Who are you?"}],
    temperature=0.7,
    top_p=0.7,
)
print(response.choices[0].message.content)

Requirements

• Linux or WSL2 with native ext4 model path
• gcc + OpenMP, AVX2
• ≥16 GB RAM (32–64 GB recommended for a comfortable expert cache)
• ~150 GB free on a fast NVMe SSD
• Optional: CUDA build for future GPU-pinned expert tier (make hy3 CUDA=1)

What's inside

On-disk size: ~142 GB (int4 container). Resident RAM (dense only): ~4.1 GB at load time; peak RSS depends on context length, expert cache cap, and --ram budget.

Convert it yourself

If you prefer to reproduce the container locally instead of downloading:

cd colibri/c
pip install torch safetensors huggingface_hub numpy
./coli convert --repo tencent/Hy3-FP8 --model /path/to/hy3_i4

The converter downloads one FP8 shard at a time, dequants, requantizes to the Colibri layout, and deletes the source shard — the full FP8 checkpoint never has to exist on disk at once.

Why not use existing Hy3 quants?

Expected performance (honest)

Hy3 is smaller per expert than GLM-5.2 (~9 MB vs ~19 MB int4), with 192 experts/layer and 79 MoE layers — cold decode still moves ~5–6 GB of expert data per token from disk. Throughput is disk-bound on a cold cache (often 0.1–0.5 tok/s class on consumer NVMe, highly setup-dependent). Warm LRU / pinned hot experts improve this substantially.

This is not a replacement for vLLM / SGLang deployment on 8× datacenter GPUs. It is a way to run the same model class on hardware you already own.

Status & limitations (early port)

• Beta: the Hy3 hy3.c engine is a new port (validated on oracle fixtures; full-model chat is still being stabilized).
• MTP: weights are included; native speculative decode in hy3.c is not implemented yet.
• CUDA expert tier: planned (make hy3 CUDA=1); not production-ready for Hy3 yet.
• KV disk persistence: not yet ported from GLM's MLA path (GQA layout differs).
• Not loadable in transformers.AutoModelForCausalLM — use colibrì only.

Provenance & license

• Base weights: tencent/Hy3-FP8 and tencent/Hy3 (Apache 2.0).
• Architecture & benchmarks: see tencent/Hy3.
• Engine & converter: colibrì (MIT) + Hy3 port (hy3.c, convert_hy3.py).
• This derivative checkpoint remains under the Apache 2.0 terms of the Hy3 base model.

Citation

If you use the base Hy3 model in academic work, cite Tencent's Hy3 release. If you use this Colibri container, please also credit the colibrì project.