WAL-Ternary-27B-RC1

WAL-Ternary-27B-RC1

The model is the proof. The pipeline is the product.

WAL-Ternary-27B-RC1 is an independently constructed, language-only ultra-low-bit checkpoint derived from Qwen3.6-27B. It is the second full-scale demonstration of the WAL approach after WAL-Ternary-8B.

We are not claiming the best 27B ternary model. We are publishing an open, auditable path for constructing, physically packing, directly executing, and measuring one without large-scale quantization-aware retraining.

Русский · Қазақша · Model files on Hugging Face

New here? Follow Start here. To rebuild or port the method, use the step-by-step reproduction guide.

Release status

RC1 is a research release. It generates coherent text and preserves substantial functionality, but its knowledge quality and runtime speed remain below the trained low-bit frontier.

Physical model

The reported BPW covers the mandatory physical checkpoint payload. Repository documentation and the derived hardware cache are not counted in model BPW.

Representation

Each selected body matrix is approximated as:

W ≈ T3 + sparse + WALB2

• T3: strict {-1, 0, +1} base, stored as seekable losslessly compressed 2-bit symbols.
• Sparse lane: role- and depth-dependent k0/k2/k4/k8 corrections per group of 128 weights.
• WALB2: two packed binary low-rank residual paths on 152 causally selected matrices.
• Embedding: 75% INT3 rows and 25% causally selected INT4 rows.
• LM head: packed INT5.
• Small tensors: BF16.
• Body scales: a documented 24-bit float storage format decoded to FP32 in the derived compute cache. This does not make the model “FP24”; its large weight bodies remain ternary hybrid.

See docs/FORMAT.md for the storage ABI.

Measured quality

All public RC1 baseline values below were measured through the direct-packed reference-exact runtime, not a persistent materialized BF16 body.

These are baseline numbers, not a claim of parity with Qwen BF16 or Bonsai. The full generative and same-harness competitor battery is incomplete.

Measured runtime

Current research backend, batch 1 on an NVIDIA H200:

These numbers establish direct execution and memory behavior. They are not a production-speed claim. The released backend is validated for NVIDIA Hopper SM90; other GPUs may compile but are not certified.

Install

The cleanest environment is Python 3.11 with a recent CUDA-enabled PyTorch.

git clone https://github.com/AubakirovArman/WAL-Ternary-27B.git
cd WAL-Ternary-27B
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -e .
wal27 doctor

PyTorch wheels are platform-specific. If pip install -e . selects the wrong wheel, install the correct CUDA build of PyTorch first, then repeat the command.

Download and verify

The CLI downloads the Hugging Face repository automatically:

wal27 inspect armanibadboy/WAL-Ternary-27B-RC1
wal27 inspect armanibadboy/WAL-Ternary-27B-RC1 --verify

--verify reads and hashes the complete 8.32 GB payload.

Prepare the hardware cache

The canonical archive is compact and portable. Before inference, build a parent-attested SM90 compute cache:

wal27 prepare armanibadboy/WAL-Ternary-27B-RC1 \
  --cache /data/wal27-rc1-sm90-cache

The derived cache is approximately 10.54 GB. It keeps ternary and binary codes packed, expands 24-bit scales to FP32 and sparse positions to the layout used by the current kernels, and is excluded from canonical BPW. A cache whose parent manifest does not match RC1 is rejected.

Generate

Correctness-oriented direct-packed mode:

wal27 generate armanibadboy/WAL-Ternary-27B-RC1 \
  --cache /data/wal27-rc1-sm90-cache \
  --mode exact \
  --prompt "Explain ternary neural networks in simple terms." \
  --max-new-tokens 128

Experimental faster arithmetic:

wal27 generate armanibadboy/WAL-Ternary-27B-RC1 \
  --cache /data/wal27-rc1-sm90-cache \
  --mode fast \
  --prompt "Қазақстандағы жасанды интеллект туралы қысқаша жаз." \
  --max-new-tokens 128

exact is the canonical quality backend. fast passed strong diagnostic teacher-forced parity but has not completed the final preregistered generative acceptance protocol.

Repository map

src/wal27/                 public codec, cache builder and CUDA runtime
docs/                      concise format/runtime/parity reports
research/designs/          complete 27B design and result journal
research/experiments/      archival experiment implementations
research/lib/              shared public T3/WALB2 research code
reproduction/              machine-readable RC1 phase map
results/                   immutable summary receipts

The archival research scripts intentionally preserve historical experiment names and some machine-local default paths. The trilingual reproduction guides explain how to adapt them and which gates must remain frozen. They document the exact campaign; they are not the supported user CLI. Use wal27 for the released model.

What the campaign established

1. Full-body ternary hybrid conversion scales from 8B to 27B.
2. Uniform sparse density is inferior to causal role/depth allocation.
3. WALB2 helps when it is fitted to the residual of the chosen sparse base and allocated selectively.
4. Endpoint precision must be calibrated against the low-bit body.
5. Small aggregate calibration fields can improve average language and knowledge metrics while still breaking individual confident decisions.
6. Linearized safety does not guarantee safety after low-precision folding.
7. A physical low-bit checkpoint is not enough: direct-packed runtime parity is part of the model evidence.

Negative results are retained in the research ledger instead of being erased.

Comparison policy

Bonsai is an important trained low-bit reference and currently represents a stronger endpoint in quality, size, and runtime maturity. No Bonsai tensors were used. WAL explores a different regime: rapidly converting an existing BF16 checkpoint with limited calibration/fitting and exposing the entire process.

Do not compare vendor-reported and WAL scores as if they used one harness. A same-harness BF16/IQ2/Binary-Bonsai/Ternary-Bonsai comparison remains open.

Limitations

• Research release; API and cache ABI may change.
• H200/SM90 is the only validated GPU target.
• No 27B CPU or Apple Silicon backend yet.
• Decode and prefill are not production optimized.
• Long-context numerical behavior is validated through 4K, not the configured 262K maximum.
• MMLU remains materially below the desired trained low-bit frontier.
• cuda-fast is not yet the canonical quality evaluator.
• This is the language checkpoint only; vision and MTP training head are absent.

Citation

@software{aubakirov2026wal27,
  author = {Arman Aubakirov},
  title = {WAL-Ternary-27B: an open ultra-low-bit conversion and runtime laboratory},
  year = {2026},
  url = {https://github.com/AubakirovArman/WAL-Ternary-27B}
}

License

Apache License 2.0. See LICENSE and NOTICE. Users remain responsible for the source-model license, dataset licenses, and applicable law.