GLM5.3-Flash-CIRU-STRIX-IU4

GLM5.3 Flash CIRU STRIX IU4

320 billion parameters. Two Strix Halo PCs. Entirely local inference.

GLM5.3 Flash CIRU STRIX IU4 brings GLM-5.3-Flash to a pair of AMD Strix Halo systems for coding, reasoning, and tool-using applications. Hybrid 4-bit weights, native integer GPU kernels, DFlash2 speculative decoding, and a direct USB4 connection work together to put this large mixture-of-experts model on desktop-class hardware—with your prompts and generated content staying on your own machines.

The result is approximately 25 generated tokens per second on the recorded coding workload, 402 prompt tokens per second at a 2K-token input, and an OpenAI-compatible interface for chat and automatic tool calling. 128K is the default context, with a 256K profile available.

This is a complete deployment package: execution-ready weights, the custom ROCm runtime, compiled kernels and their source, service installers, and CiruStrixLink to manage the connection. You can use StrixLink or bring an existing compatible USB4 setup.

The model at a glance

The included runtime is required. These weights are already arranged for the two-machine GPU execution path, not a generic Transformers or GGUF download. Stock vLLM, Transformers, and llama.cpp cannot load this package directly. The DFlash2 draft is downloaded separately; see installation and licensing.

Performance

Prompt processing and generation

The measurements below used two Strix Halo systems, ROCm 10, USB4, and DFlash2. Prompt processing is the rate at which the model reads input. Generation speed counts new tokens after the first token, excluding prompt processing. Time to first token includes the initial request work.

The current coding sample generated 6,382 tokens across ten requests in 4 minutes 11 seconds, including prompt processing. All ten answers stopped naturally and passed their tests. DFlash2 accepted 69.4% of proposed draft tokens in that sample. Acceptance and output speed depend on the workload; these are real generated answers, not forced-acceptance or synthetic throughput results.

One additional local inference workload was resident during this HumanEval run. The 26.10 tok/s figure is therefore an observed workload result, not an isolated-system ceiling. The 10/10 correctness result is unaffected.

The two shorter fixed-prompt runs used the 64K configuration and the 131K input used 128K. The 2K run had prefix caching enabled, the 20.5K run disabled it, and the 131K result used an uncached input. The current coding sample used the 256K profile with caching disabled. These are individual recorded runs, not a single matched context-length sweep. An earlier 768-token input measured 485 prompt tokens/s; short-input speed should not be extrapolated to long prompts. Blank cells make no generation-speed claim for that row.

Coding and tool use

The HumanEval sample used one response per problem, the model's chat template, temperature 1.0, top-p 0.95, maximum reasoning effort, and an 8,192-token output allowance. There were no cap hits, loop aborts, retries, or best-of selection. It is a ten-problem sample—not the full 164-task HumanEval benchmark.

ToolEval used tool-eval-bench 2.1.0 with a local adapter for the two-machine API. Each scenario receives two points for a pass and one for a partial; 87% and 83% are rounded point scores, not pass-only percentages. Safety and boundary handling, and choosing among large toolsets, were weaker categories. Applications should enforce tool permissions outside the model.

These measurements come from several runs of the released weight configuration, with runtime and cache settings that varied during development. The benchmark reports document those settings and versions alongside the saved results.

Quantization quality against BF16

The hybrid weights were compared with the official BF16 model on 767 matched next-token positions from WikiText, scoring the full 154,880-token vocabulary at every position.

This measures how closely the output distribution tracks BF16 on one 768-token block. It is a useful quantization check, not a full WikiText evaluation, a broad capability score, or a ranking against other quants. Exact results and methodology.

Built for Strix Halo

Hybrid weights and native integer acceleration

The routed experts—the bulk of the model—use symmetric 4-bit weights with scales shared across groups of 128. Sensitive components retain higher precision, while selected dense projections have their own optimized quantized buffers. The complete target therefore averages approximately 4.46 bits per parameter, rather than exactly four.

The stored weights are already split between the two machines and arranged for the GPU kernels. Execution combines native 4-bit integer paths, 8-bit dot-product paths, and higher-precision operations where appropriate. Different kernels handle prompt processing, single-token decoding, and multi-token speculative verification. Expert routing, output reduction, and alignment also have specialized implementations.

The design tunes the weight representation and execution together, rather than relying on compression alone. The compiled kernels and their source are included; exact versions and layout details are in the runtime package notes.

DFlash2: more output per target-model iteration

DFlash2 proposes five draft tokens for the target model to check together. When several are accepted, one target-model iteration produces multiple output tokens. The supplied configuration uses probabilistic draft sampling and standard rejection sampling.

This pairs weight compression with speculative weight reuse: the large target evaluates a small batch of proposed tokens instead of doing all its work for only one new token. Actual gains depend on acceptance, which is why the performance section reports real workloads rather than only an ideal speculative ceiling.

Two machines, one application endpoint

The target uses two-way tensor parallelism, with both machines participating in inference. Direct USB4 accesses the Native Host Interface (NHI) through the packaged adapter for the eight-position verification workload; other communication uses the configured network collectives.

CiruStrixLink helps configure, inspect, and prepare that connection. Its optional Launch page can also select a context profile, load both ranks, show host-wide unified-memory use, report whether Fast mode is actually ready or in use, and unload the pair. The included API frontend presents a single endpoint to applications and sends matching requests to both workers. You can keep your own transport tooling, but generation requests must still go through the frontend—not directly to just one worker.

Direct GPU transport in microseconds

The direct NHI adapter exports dedicated uncached HIP allocations as DMA-BUF pools, exchanges them through USB4 hardware rings, then validates sequence and epoch footers and sums the received BF16 partials on the GPU. Its exact [8,4096] / 64-KiB all-reduce component gate measured 109.897–110.807 µs median, versus 340.904–342.745 µs for the matched RCCL socket control: 67.5–67.9% lower completion latency with bit-exact output. All 80 exchanges per rank completed without timeouts, protocol errors or driver failures.

This is completed GPU-operation latency, not TCP ping or a claim of 3× model speed. The published source contains the synchronous M8 bridge; deferred retirement and compressed activation transfers are separate development results. See transport details and measured scope and the full microsecond performance report.

Hardware and software

• Two Strix Halo systems with 128 GiB unified memory each, connected over USB4. The GPU target is gfx1151.
• ROCm 10 and the bundled vLLM runtime. The tested deployment uses NixOS and Linux 7.2.2 with USB4STREAM/NHI support. A generic USB4-capable kernel alone does not provide the direct-NHI path.
• Local NVMe on each machine for approximately 83.25 GiB of target weights, the separate draft, runtime, and cached prompts. Allow substantial additional storage for disk caching.
• Enough available unified memory for the selected context profile. The 128K default allocates 12 GiB of GPU KV cache and 8 GiB of host prefix staging per machine; other applications share the same physical memory.

The compiled runtime wheels were built on Ubuntu 24.04 with Python 3.14. Compatible non-NixOS requirements are documented, but other distributions are not yet tested deployment targets for this complete setup.

Installation

The installation guide gives the complete two-machine procedure: dependencies, downloads, link setup, service configuration, and paired startup.

Install CiruStrixLink 0.3.3

Version 0.3.3 removes host-specific checks of unrelated applications while keeping paired GLM control. It changes no model weights or inference runtime. The model repository includes the qualified static Linux amd64 release. Install the same version on both model hosts:

HF_ENV="${XDG_DATA_HOME:-$HOME/.local/share}/ciru-hf"
python3 -m venv "$HF_ENV"
"$HF_ENV/bin/pip" install -U huggingface_hub
HF="$HF_ENV/bin/hf"

VERSION=0.3.3
mkdir -p /tmp/ciru-strixlink-release
"$HF" download jcbtc/GLM5.3-Flash-CIRU-STRIX-IU4 \
  --include "tools/ciru-strixlink-${VERSION}-linux-amd64.tar.gz" \
            "tools/ciru-strixlink-${VERSION}-SHA256SUMS" \
  --local-dir /tmp/ciru-strixlink-model-release
(cd /tmp/ciru-strixlink-model-release/tools && \
  sha256sum -c "ciru-strixlink-${VERSION}-SHA256SUMS")
tar -xzf \
  "/tmp/ciru-strixlink-model-release/tools/ciru-strixlink-${VERSION}-linux-amd64.tar.gz" \
  -C /tmp/ciru-strixlink-release
sudo install -m 0755 \
  /tmp/ciru-strixlink-release/ciru-strixlink \
  /usr/local/bin/ciru-strixlink
ciru-strixlink version
ciru-strixlink prerequisites

The redesigned Connection Setup page walks through both hosts, while the third tab is an opt-in Launch page for this fixed TP2 deployment. Model control is disabled by default and requires a shared token, complementary fixed ranks, fixed USB4 IPv4 peers, a model frontend URL, a loopback console, and narrow root-owned helpers on both hosts.

On the peer model host:

ciru-strixlink agent \
  --token-file TOKEN_FILE \
  --model-control \
  --model-rank 1 \
  --model-peer RANK_0_USB4_ADDRESS

On the console model host:

ciru-strixlink ui \
  --peer RANK_1_USB4_ADDRESS \
  --token-file TOKEN_FILE \
  --model-url http://127.0.0.1:8083 \
  --model-control \
  --model-rank 0

From a desktop, forward the console's loopback port, then open http://127.0.0.1:7749/#/launch:

ssh -N -L 7749:127.0.0.1:7749 USER@CONSOLE_MODEL_HOST

The desktop needs neither the binary nor model-host privileges. When a helper permission is missing, the Launch page displays a copyable host-specific NixOS or generic Linux setup. The controller prevents the portable and NHI forms of this GLM deployment from running together, starts rank 0 before rank 1, verifies the exact context through the frontend, and rolls both ranks back if loading fails. Unload runs rank 1 before rank 0. Context can only change while the pair is unloaded.

See the complete StrixLink 0.3.3 instructions, model integration guide, and security model.

Download only the weight partition needed by each machine:

HF_ENV="${XDG_DATA_HOME:-$HOME/.local/share}/ciru-hf"
python3 -m venv "$HF_ENV"
"$HF_ENV/bin/pip" install -U huggingface_hub
HF="$HF_ENV/bin/hf"

sudo install -d -o "$USER" -g "$(id -gn)" /srv/llm/models
MODEL_ROOT=/srv/llm/models/GLM5.3-Flash-CIRU-STRIX-IU4
NODE_RANK=0  # use 1 on the other machine

"$HF" download jcbtc/GLM5.3-Flash-CIRU-STRIX-IU4 \
  --include "README.md" "*.sh" "LICENSE*" "THIRD_PARTY_NOTICES.md" \
            "config/*" "docs/*" "runtime/**" "systemd/*" \
            "context-profiles/*" "tools/*" "rank-${NODE_RANK}/*" \
  --local-dir "$MODEL_ROOT"

"$HF" download incoai/GLM-5.3-Flash-DFlash2 \
  --revision bf582e4eacc1810f76656d1811693ff6c6737d2a \
  --local-dir /srv/llm/models/GLM-5.3-Flash-DFlash2

Then:

1. Install the packaged runtime on both machines, including its NixOS module when applicable. Do not substitute a stock vLLM installation.
2. Install and prepare CiruStrixLink, or supply an existing compatible transport.
3. Install the NHI system service and choose the same context profile on both machines. Direct NHI requires CAP_SYS_RAWIO; the provided service uses a root-owned runtime and grants that capability to the model process.
4. Start both workers together, then start the API frontend on either machine.

A user systemd service is also included for portable transport. StrixLink is not a runtime dependency, and the frontend works without the app. Direct NHI uses the separate system service because an ordinary user unit cannot grant the required hardware-access capability.

Context profiles

128K is the default; a 256K profile is included for larger workloads. Choose the memory allocation that fits your application:

The limit covers input and generated output, so leave room for the reply. The additional 128 tokens provide runtime headroom above each nominal context. These profiles do not use RoPE scaling.

For the NHI service, select the same profile on both machines:

sudo glm53-nhi-context USER list
sudo glm53-nhi-context USER 2  # 1 = 64K, 2 = 128K, 3 = 256K

Replace USER with the local service account. For the portable user service, use glm53-context 2 instead. Selection updates configuration without restarting; apply it at the next coordinated two-machine restart.

The 256K/8 GiB profile has passed paired startup, exact 262,272-token frontend validation, HumanEval 0–9, and a 65,680-token recovery stress probe. That probe was designed to test a minimum generation floor after a very long prefill; it is not representative of ordinary short-prompt chat speed. The initial depth-comparison environment was not fully host-isolated, so its detailed report is retained as directional evidence rather than a production-speed claim. This is a bounded probe rather than a full 256K quality sweep. On the validated 128-GiB-per-host deployment, a 16 GiB KV allocation does not fit the remaining unified-memory headroom. No fixed free-memory reserve is promised.

DFlash and cache controls

The managed launcher reads the same two files on both ranks at startup. Change them only while the pair is stopped:

mkdir -p ~/.config/ciru-glm53-iu4
printf '5\n' > ~/.config/ciru-glm53-iu4/dflash-tokens
printf '0\n' > ~/.config/ciru-glm53-iu4/prefix-cache-enabled

dflash-tokens accepts integers from 0 through 7. A value of 0 is a true target-only launch: the speculative configuration is omitted. Invalid values stop startup with a clear error; missing files use the production defaults k=5 and cache-off. CiruStrixLink 0.3.1 and later report these settings only when both ranks agree.

Persistent disk prefix cache

Prefix caching is disabled by default. The filesystem tier is functional, but it is not safe as an unattended production default because it has no built-in byte quota or automatic eviction.

The 8 GiB setting limits RAM staging, not disk use. The filesystem cache can grow without a built-in limit. During the 256K investigation, retained cache trees reached 219 GiB on rank 0 and 266 GiB on rank 1. Keep PREFIX_CACHE_ENABLED=0 unless PREFIX_CACHE_ROOT is placed on a dedicated, quota-managed filesystem with enough headroom and monitored failure handling. Storage configuration.

Production settings and API

The launcher keeps the model, kernels, speculation, and serving configuration together. Use the packaged chat template and generation defaults rather than flattening a conversation into raw completion text.

The prompt batch budget is not the context limit. A large input is processed in chunks; increasing the batch budget is not required for long context. In a matched 20.5K-input test, increasing it to 8,192 did not help.

Send requests to the frontend, normally port 8083:

curl http://FRONTEND_HOST:8083/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM5.3-Flash-CIRU-STRIX-IU4",
    "messages": [
      {"role": "user", "content": "Write a Python LRU cache and explain its tradeoffs."}
    ],
    "temperature": 1.0,
    "top_p": 0.95,
    "reasoning_effort": "max",
    "chat_template_kwargs": {"clear_thinking": true},
    "max_tokens": 8192
  }'

max_tokens is this request's output allowance, not the context size. Adjust it to the task and remaining context. For automatic tool use, provide OpenAI-compatible tools and tool_choice fields; the runtime applies the template and parses tool calls. Do not insert GLM control tokens manually. Keep the endpoint private or add authentication before exposing it remotely.

Files and documentation

Scope and considerations

This is a two-system, hardware-specific, text-only deployment. Single-host, other-GPU, and concurrent-request throughput are not measured configurations. Long inputs take time to process, and configured context capacity should not be confused with demonstrated long-context answer quality.

The model can produce incorrect code or tool arguments. The stated coding and tool-use results are useful evidence, not a substitute for application testing, permission boundaries, or user confirmation for consequential actions.

The provided launcher requires the separate DFlash2 draft. A separately validated target-only launcher is not included.

License and credits

This independent Ciru release builds on:

• Z.AI, for GLM-5.3-Flash, its tokenizer, and configuration.
• wtdcode, for the AWQ conversion used to construct these weights.
• Inco AI, for DFlash2.
• vLLM, AMD AITER, Composable Kernel, ROCm, and Linux USB4, for the serving, GPU, and transport foundations.

Ciru provides the Strix-specific layouts and kernels, transport integration, serving configuration, frontend, and deployment packaging. The official GLM parent is MIT licensed; Ciru-authored runtime, frontend, and kernel components are Apache-2.0 where identified. Other components retain their notices.

The AWQ-derived weights are distributed with separate permission from wtdcode. That permission does not establish a blanket downstream license; see license scope and third-party notices. The separately downloaded DFlash2 draft is CC BY-NC-ND 4.0, so commercial use of the complete speculative setup requires appropriate rights. It is not relicensed here.

Third-party names and marks identify compatible hardware and software; they do not imply endorsement of this independent release.