Ling-3.0-Flash-CIRU-int4-Strix-native

Ling-3.0-Flash-CIRU-int4-Strix-native

Ling-3.0-Flash-CIRU-int4-Strix-native is CIRU's native vLLM/ROCm runtime distribution for InclusionAI's official packed-INT4 Ling 3.0 Flash checkpoint on AMD Strix Halo (gfx1151).

The model weights were not changed, requantized, merged, fine-tuned, or renamed internally. This release combines the original checkpoint at revision ca3ea63b0255d212c4fe6020db9e0a51ce136006 with a pinned vLLM fork, ROCm 7.15 build recipe, and launch profiles validated on an AMD Radeon 8060S with 128 GB unified memory.

Important engine update — August 9, 2026

Existing installations should apply the current engine update. The original ROCm 7.15 wheel runtime can leave one CPU core continuously busy while the server is idle, producing unnecessarily high CPU temperatures. This is a ROCr userspace-runtime issue; the Ling model weights are unaffected.

For the standard archive installation, stop the running Ling/vLLM server and run the following from the directory that contains your existing Ling-3.0-Flash-CIRU-int4-Strix-native folder:

curl -fLO https://huggingface.co/jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native/resolve/main/Ling-3.0-Flash-CIRU-int4-Strix-native.tar.gz
curl -fLO https://huggingface.co/jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native/resolve/main/SHA256SUMS.release
sha256sum --check --ignore-missing SHA256SUMS.release
tar -xzf Ling-3.0-Flash-CIRU-int4-Strix-native.tar.gz
cd Ling-3.0-Flash-CIRU-int4-Strix-native
bash install.sh --upgrade --install-root "$PWD" --install-host-deps

Extraction updates the packaged scripts in place while retaining the existing .venv, model, and generated environment file. If .venv is elsewhere, replace "$PWD" with the directory containing it. The upgrade rebuilds and verifies only the pinned ROCr library. It does not redownload the checkpoint, rebuild vLLM, or change model/configuration files. Restart the same service or launch again with the updated scripts/run-256k.sh when it completes. See Upgrade an existing CIRU engine for details and rollback instructions.

Packages downloaded before August 10 could incorrectly report a newly created, empty ROCr checkout as modified and stop before writing ling3-runtime.env. The current installer repairs that empty checkout automatically while still refusing to overwrite a non-empty checkout containing real local changes.

The August 10 installer also corrects two path/bootstrap failures found by a fresh Arch installation. Quoted agent path spellings such as '$HOME/runtime' and '${HOME}/runtime' are now resolved deliberately instead of creating a literal $HOME directory inside the checkout. ling3-runtime.env is written immediately after path resolution, before downloads or builds, so it remains available with the intended paths even when a later dependency or build step fails. Other unresolved $... path expressions are rejected.

The same release gate also found and corrected three retry/portability defects: the bundled vLLM branch is now fetched through a detached release ref so a second build does not fail on its checked-out branch; all direct build, download, and ROCr entry points resolve quoted $HOME/$PWD paths and reject unknown or surplus arguments; and virtual-environment command paths are fully quoted. Because Linux LD_PRELOAD uses whitespace and colons as separators, the runtime install root must not contain either character. The installer now rejects such a root immediately instead of producing a broken launch later.

The current launcher also identifies cold-cache compilation before vLLM starts. Messages containing tl.make_block_ptr is deprecated, TypedStorage is deprecated, Autotune Choices Stats, or AUTOTUNE mm(...) are emitted by the pinned Triton/PyTorch compiler path and are not startup failures. Wait for Application startup complete, then verify /health; a traceback, EngineCore failed, a nonzero process exit, or a failed health check is an actual failure. The required VLLM_ROCM_SAFE_MERGE_ATTN_STATES setting is now registered by the CIRU fork, so vLLM no longer mislabels it as an unknown environment variable.

A 97.5x native ROCm serving campaign: the first working upstream-compatible build decoded at 0.269 tok/s; this distribution reaches 26.23 tok/s with native MTP K1, while retaining the official vendor checkpoint, OpenAI-compatible serving, continuous batching, and Ling's structured tool parser.

What this release changes

The runtime is based on vLLM commit d35eb6c44071ea806018841c490f0d2f3219c485 plus a 17-commit CIRU branch ending at 838616875c5dd4913d9f753221bf05f64cb7a7ed. Its net source delta is 10 files, 477 insertions, and 28 deletions, including regression tests.

• Includes liminfei-amd's Wave32 LDS fix for the ROCm top-k merge launch, preserving the AMD author's commit credit.
• Adds an opt-in safe PyTorch attention-state merge for a gfx1151 Triton/HSA fault, including both supported MLA LSE layouts.
• Normalizes the upstream ROCm FlashAttention variable-length LSE result to vLLM's documented head-first layout before chunked MLA context accumulation.
• Routes causal uniform multi-token MLA verification blocks through Triton decode with correct per-token sequence lengths.
• Adds narrowly guarded W4A16 MoE decode/verification fast paths for Ling's exact small-token geometry.
• Adds a gfx1151 Triton SiLU-and-multiply kernel for Ling's exact eager W4A16 shapes.
• Rebuilds the wheel-pinned ROCr userspace runtime with the host GCC to remove the idle CPU busy-spin, then verifies and preloads the result automatically.

See CHANGELOG.md for the public file-level runtime diff.

Performance: 97.5x over the public starting point

The closest runnable public/upstream starting point was frozen vLLM commit d35eb6c44071ea806018841c490f0d2f3219c485 plus the publicly proposed five-line ROCm sampler fix required to launch on gfx1151. With the unchanged official model, that path produced 0.268944 tok/s in a warmed deterministic target-only decode test. This release reaches 26.2343 effective tok/s with the checkpoint's native K1 MTP: 97.55x the starting throughput, or approximately 100x.

The starting row is a warmed deterministic eight-token test; the final rows are medians from deterministic 32-token tests. The multiplier therefore describes the full optimization campaign on the same model and host, not a single-variable matched A/B. The final exact-shape kernel step was also tested separately against the already-optimized CIRU parent: target-only decode rose from 19.8993 to 21.4407 tok/s (+7.7457%), and K1 rose from 25.1348 to 26.2343 tok/s (+4.3743%). That smaller percentage is the final incremental step, not the overall release gain.

MTP acceptance remained 82.35% with 1.8235 mean tokens per step. Retained controls had exact token IDs/text and zero faults. These are local single-host measurements, not claims for other AMD architectures.

How 0.269 became 26.23 tok/s

The largest single problem was silent selection of vLLM's ROCm skinny-GEMM path on Wave32: disabling it alone improved identical-output decode by 28.16x. The remaining gains came from compilation, a coherent AMD software stack, Ling-shaped W4A16 MoE paths, repaired MTP verification routing, and the final exact-shape activation kernel.

Serving results

On exact 2K prompts with 256 forced output tokens, K1 MTP, simultaneous bursts, unique cache salts, and no prefix reuse, the released 256K/0.72/six-slot profile produced these medians across three measured repetitions:

All 18 measured runs were valid, with zero request failures, cache reuse, preemptions, post-warmup JIT, engine restarts, or device faults. Against the retained matched C1/C2/C4/C5 rows, every PP, TG, and wall result stayed within the 5% regression guard; PP improved at all four comparable concurrency levels. max_num_batched_tokens=8192 is part of this protocol and the packaged profile.

The runtime also sustained 540.19, 550.20, and 536.13 prefill tok/s at 2K, 8K, and 16K in clean long-prefill tests on an earlier retained build; treat those rows as a floor rather than a claim measured again on the final commit. The final stack completed CIRU's 20-scenario structured tool-call suite using Ling's native parser.

60K PP/TG comparison

This retained comparison uses one frozen 60,000-token prompt followed by 512 forced output tokens:

Native vLLM gives up 20.66% TG to Atomic at 60K, but its 57.87% PP lead completes the combined request 31.48% sooner. It leads ROCmFP4 by 68.25% PP, 4.88% TG, and 37.00% in combined wall time.

The native row predates the release's final SiLU kernel commit, so this table is a retained baseline rather than a final-commit maximum. See docs/BENCHMARKS.md for the exact protocol, source identities, and refresh rule.

Files in this package

• runtime/Ling-3.0-Flash-CIRU-int4-Strix-native-vllm.bundle: exact Git objects for the CIRU branch, based on the pinned upstream commit.
• runtime/Ling-3.0-Flash-CIRU-int4-Strix-native-vllm.patch: reviewable net source diff.
• install.sh: guided human installer plus a flag-driven interface for agents and automation.
• scripts/download-model.sh: downloads only the unchanged checkpoint files from this jcbtc repository, pinned to the verified model revision, into a parent directory or exact --model-path.
• scripts/install-host-deps.sh: distro-aware Linux/WSL host dependency installer.
• scripts/build-vllm-gfx1151.sh: reproducible build entry point for the pinned AMD wheel stack.
• scripts/install-rocr-idle-fix.sh: pinned, host-native ROCr rebuild that removes the idle CPU busy-spin seen with the original ROCm 7.15 wheel runtime.
• scripts/run-256k.sh: native 256K, native-MTP K1 server profile with six active sequence slots.
• scripts/run-1m-yarn-experimental.sh: experimental 4x YaRN 1M profile with two active sequence slots.
• MANIFEST.json: machine-readable identities and scope.

The named release archive is Ling-3.0-Flash-CIRU-int4-Strix-native.tar.gz. It contains the runtime distribution, image, and documentation—not a second 77 GB copy of unchanged weights.

Quick start

The exact validated environment is Python 3.12, Torch 2.13.0+rocm7.15.0a20260728, HIP 7.15.0, Triton 3.8.0+git4cff872c.rocm7.15.0a20260728, FlashAttention 2.8.3, and the CIRU vLLM source at 838616875.

For a clean runtime installation, download the 115 MB release archive instead of cloning this model repository. A normal Git LFS clone can also download the 77 GB checkpoint at the repository root.

mkdir -p ling3-ciru-release
cd ling3-ciru-release
curl -fLO https://huggingface.co/jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native/resolve/main/Ling-3.0-Flash-CIRU-int4-Strix-native.tar.gz
curl -fLO https://huggingface.co/jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native/resolve/main/SHA256SUMS.release
sha256sum --check --ignore-missing SHA256SUMS.release
tar -xzf Ling-3.0-Flash-CIRU-int4-Strix-native.tar.gz
cd Ling-3.0-Flash-CIRU-int4-Strix-native
sha256sum --check SHA256SUMS

For a guided install, run this from the repository root:

bash install.sh --interactive

The runtime path defaults to the current directory. The installer asks for the exact model directory, but the 77 GB checkpoint download is strictly optional and defaults to no. The installer writes a sourceable ling3-runtime.env before beginning the runtime build, then prints a command to download or provide the model later. Declining the model download still builds and validates the runtime without weights.

Use an install root without whitespace or :. This is a Linux LD_PRELOAD restriction for the checksum-verified ROCr override; model paths may contain spaces.

Agents and automation can avoid prompts:

bash install.sh \
  --install-root "$PWD" \
  --model-path "/path/to/Ling-checkpoint"

source "$PWD/ling3-runtime.env"
bash scripts/run-256k.sh

Add --install-host-deps when the supported distro packages are not already installed. Add --download-model only when you want this installer to fetch the checkpoint; omit it for a runtime-only install. Use bash install.sh --help for the complete agent interface.

Update an existing CIRU installation

From the updated package directory, rebuild the packaged fork and the corrected ROCr runtime while preserving the existing model:

bash install.sh \
  --install-root "$HOME/ciru-ling-runtime" \
  --model-path "/path/to/Ling-checkpoint" \
  --install-host-deps

Replace the install root with the directory that contains the existing .venv and use the existing checkpoint directory. This does not download, move, or modify the weights. It installs runtime head 838616875, builds the pinned ROCr userspace override that fixes the idle 100% CPU spin, and rewrites ling3-runtime.env with those same paths. Restart the server afterward.

install.sh --upgrade is intentionally ROCr-only. Use it only when the vLLM source already matches the package and just the idle-CPU override needs repair. For emergency ROCr rollback, launch once with CIRU_DISABLE_ROCR_IDLE_FIX=1.

The API advertises model ID Ling-3.0-Flash-CIRU-int4-Strix-native on http://127.0.0.1:18081/v1.

The default launcher provides the checkpoint's native 256K context (262144 tokens), six active sequence slots, gpu_memory_utilization=0.72, and max_num_batched_tokens=8192. Requests beyond six wait in vLLM's scheduler.

If you do not need that much simultaneous KV-cache capacity, lower the memory fraction at launch to leave more unified memory for the operating system and other applications. For example:

GPU_MEMORY_UTILIZATION=0.70 \
VENV="$HOME/ciru-ling-runtime/.venv" \
MODEL_PATH="$HOME/models/Ling-3.0-Flash-CIRU-int4-Strix-native" \
bash scripts/run-256k.sh

Lower values reduce KV-cache capacity. Even at the packaged 0.72 setting, six requests cannot all occupy the full 256K limit simultaneously; verify the reported KV-cache capacity at startup for your workload and host.

Cold-start compiler output

The optimized launcher deliberately uses vLLM compilation mode 3 with static decode sizes [1,2] and Inductor max autotuning. On an empty cache, model loading plus compilation can take several minutes and prints kernel comparisons such as AUTOTUNE mm(...). The two deprecation warnings named above are also expected with the pinned dependency versions. They occurred in the validated run before the API became healthy and returned HTTP 200.

Do not remove launcher flags merely to make these messages disappear. First run the packaged command unchanged and wait for Application startup complete:

source ling3-runtime.env
bash scripts/run-256k.sh

The launcher prints both the runtime head expected by the package and the head of the installed editable vLLM source. If they differ, update the CIRU fork by running the exact rebuild command printed there. That command reuses the model path and does not download weights; install.sh --upgrade is intentionally ROCr-only and is not a vLLM source update.

If a modified command is required on another distribution, retain the complete command, the first traceback or EngineCore failed block, and the final 100 log lines. Performance from a command with MTP, compilation, max autotuning, prefix caching, or the 8,192-token scheduler budget disabled is not comparable to the published benchmark profile.

For the experimental 1M profile instead:

VENV="$HOME/ciru-ling-runtime/.venv" \
MODEL_PATH="$HOME/models/Ling-3.0-Flash-CIRU-int4-Strix-native" \
bash scripts/run-1m-yarn-experimental.sh

That launcher applies a 4x YaRN extension to 1,048,576 tokens, uses gpu_memory_utilization=0.82, pins max_num_batched_tokens=8192, and admits two active sequences. It is not native 1M training; requests beyond two wait.

Recommended live sampling

Follow InclusionAI's official Ling 3.0 Flash model card for live generation: temperature=0.6, top_p=0.95, top_k=20, with enable_thinking=true for better performance. The CIRU launcher does not replace these request-level sampling choices with a temperature-zero preset.

curl -fsS http://127.0.0.1:18081/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Ling-3.0-Flash-CIRU-int4-Strix-native","messages":[{"role":"user","content":"Explain in one sentence why the sky is blue."}],"chat_template_kwargs":{"enable_thinking":true},"temperature":0.6,"top_p":0.95,"top_k":20,"max_tokens":256}'

Temperature 0 appears only in the retained benchmark documentation as a deterministic measurement control. It is not the recommended live-generation preset.

Choose the platform guide before building:

• Native Linux: Ubuntu/Debian, Fedora, Arch, and NixOS notes
• Windows 11 through WSL2 and AMD ROCDXG

The exact runtime has been validated on CIRU's native-Linux NixOS Strix Halo host. Mainstream native-Linux and Windows/WSL instructions are designed to be reproducible but are not yet CIRU clean-host validations. Native Windows vLLM is not supported; Windows users must use WSL2.

Context and concurrency

The official checkpoint is natively configured and trained for 256K context. run-256k.sh uses max_model_len=262144, max_num_seqs=6, gpu_memory_utilization=0.72, and max_num_batched_tokens=8192. On CIRU's 128 GB Strix Halo host, the validated profile reported a 1,457,313-token KV pool, or 5.56 full 262,144-token contexts.

Six is the packaged and validated active-request setting. The earlier MLA accumulator failure was root-caused to ROCm upstream FlashAttention returning LSE as [tokens, heads] while vLLM's chunk accumulator requires [heads, tokens]; it was not caused by a request occupying “slot 5.” Runtime commit 91fab4e6d normalizes that adapter output. A guarded three-repetition C1-C6 sweep then passed all 18 measured runs, and a separate production-profile C6 smoke passed 6/6 requests without restarting the server. Six active requests work, but the 5.56x full-context KV pool means six requests cannot all hold 256K tokens simultaneously. Requests beyond six wait in vLLM's scheduler.

The 1M profile is an experimental 4x YaRN extension, not native 1M training. It uses max_model_len=1048576, max_num_seqs=2, and gpu_memory_utilization=0.82; further requests wait in the vLLM scheduler.

Keep max_num_batched_tokens=8192 when launching either packaged MTP profile. If this value is omitted, vLLM may select a 2,048-token speculative-decoding scheduler budget and only report the choice in its startup log. That setting is functional, but it changes prefill/decode overlap and produced materially lower aggregate TG under the same C1-C6 protocol. Both packaged launchers pin 8,192; users constructing their own vllm serve command should pass --max-num-batched-tokens 8192 explicitly.

Does this work for other W4A16 models?

The fork remains a vLLM runtime and safely falls back to upstream paths when its guards do not match, but the measured speedups are mostly Ling-specific:

• The Wave32 top-k fix is broadly relevant to ROCm Wave32 GPUs.
• Safe attention merging and uniform verifier routing are relevant to compatible ROCm MLA workloads, independent of W4A16.
• The upstream ROCm FlashAttention LSE normalization is relevant to compatible ROCm MLA chunked-prefill workloads, independent of Ling's W4A16 geometry.
• The W4A16 fast assignment path currently requires 512 experts, top-k 8, block size 32, and one to three tokens.
• The fast reduction requires hidden size 2560 and top-k 8.
• The SiLU path requires exact (8,1536)->(8,768) or (16,1536)->(16,768) BF16 shapes on gfx1151.

Other W4A16 models will run only if already supported by this vLLM base; they should not be expected to receive these speedups unless their geometry matches. Generalizing those kernels is part of the upstream plan.

Credits and license

• Model and official INT4 weights: InclusionAI/Ling-3.0-flash-int4, MIT license.
• Inference engine: vLLM, Apache-2.0.
• Wave32 top-k merge fix: liminfei-amd, vLLM PR #46012. The release history retains the original AMD authorship.
• ROCm, Torch, and Triton packages: AMD and their respective upstream projects.
• Strix Halo kernel, TTM/GTT, and WSL/ROCDXG guidance: AMD ROCm documentation and ROCDXG.
• Runtime integration, profiling, kernels, and validation: CIRU.

The original model license continues to govern the weights. The vLLM patch and bundle retain vLLM's upstream licensing and file headers.