Downloads · 30 days
0
bong-water-water-bong/halo-sparse-bitnet-3-4-500m
halo-sparse-bitnet-3-4-500m is a text generation model from bong-water-water-bong. Use it when you need the model to write or continue text. The card lists the license as mit.
Research preview. 3:4 Sparse-BitNet pretrained from Qwen2.5-0.5B skeleton. Not production-ready; shipped as a validation checkpoint for the 3:4 vs 2:4 N:M sparsity question. The canonical production model on the 1bit-…
Downloads · 30 days
0
Access
Public
Updated Apr 24, 2026
Repo size
1.4 GB
Likes
1
Public
Click a slice to open those files.
.pt988 MB · 73%
From the Hugging Face model README
Research preview. 3:4 Sparse-BitNet pretrained from Qwen2.5-0.5B
skeleton. Not production-ready; shipped as a validation checkpoint
for the 3:4 vs 2:4 N:M sparsity question. The canonical production
model on the 1bit-systems stack remains
halo-1bit-2b (2 B params, PPL 9.16).
At 500 M params and 10 B training tokens (≈20 % of the paper gate of 50 B), on wikitext-103 test (145 chunks × 2048 tokens = 296 960 tokens scored):
| Model | PPL | NLL/tok | Notes |
|---|---|---|---|
| Qwen2.5-0.5B fp16 dense (teacher baseline) | 13.02 | 2.57 | reference upper bound, not trained by us |
| 3:4 Sparse-BitNet 500 M (this ckpt, step 9 600) | 95.13 | 4.55 | 7.3× PPL gap vs teacher — undertrained at 10 B tokens |
| 3:4 Sparse-BitNet 500 M (step 9 100 interim) | 94.99 | 4.55 | near-plateau, 500 steps earlier |
| 2:4 Sparse-BitNet 500 M (paired run, step 9 600) | 150.47 | 5.01 | 58 % worse than 3:4 at matched step |
Findings:
Takeaway: 3:4 is the pattern to scale at 2 B + full-budget. Published here so the ablation number is on record.
This is a raw PyTorch state_dict, not a packaged .h1b. Serve
path on the 1bit-systems stack requires a converter we haven't
written yet — tools/sparse-bitnet-to-h1b/ is queued for a future
release. Today, you load it in Python.
import torch
from llm_vendor.arch.model import Model, ModelArgs # see 1bit-systems repo
args = ModelArgs(
d_model=896, d_ffn=4864, head=14, kv_head=2,
n_layers=24, vocab_size=151936,
max_seq_len=2048, rope_theta=1000000.0, norm_eps=1e-6,
weight_tying=True, bitlinear=True,
use_weight_semi_sparse=True,
sparse_n=3, sparse_m=4,
)
model = Model(args).to("cuda").to(torch.bfloat16)
model.load_state_dict(torch.load("model.pt", weights_only=False), strict=False)
model.eval()
ppl_eval.py script pattern in the
1bit-systems
repo under benchmarks/.
model.pt # 943 MB, bf16 raw state_dict (training artifact)
halo-sparse-bitnet-3-4-500m.safetensors # 346 MB, 3:4 ternary 2-bpw packed + per-row bf16 scales (halo-ready)
pack_sparse_bitnet.py # the packer script — reproduces .safetensors from .pt
pack-summary.json # compression stats
meta.json # training step pointer
training-loss.csv # per-50-step loss trace
ppl-qwen25-fp16-baseline.json # teacher upper bound
ppl-step9600-final.json # our final PPL
ppl-step9100-interim.json # near-plateau interim
ppl-run5-baseline-for-comparison.json # 2:4 twin for reference
SHA256SUMS # sha256 of every artifact
Packed vs raw:
model.pt (943 MB bf16) is the exact training-loop output; use
this if you want to fine-tune or resume.halo-sparse-bitnet-3-4-500m.safetensors (346 MB) is the
halo-native layout: every linear is NAME.packed (uint8 2-bpw
ternary, 4 codes per byte: 00=0, 01=+1, 10=-1) + NAME.row_scale
(bfloat16 per-row scale). Norms, embeddings, and the tied
LM head stay bf16.Optimizer state (optim.pt, ~1.9 GB) not uploaded — if you want to
resume training, open an issue. We can move it.
.h1b packaged weights (conversion tool pending).1bit-server. Serve path needs Qwen2.5 arch port
halo-1bit-2b model stays the install default. This ckpt is
explicitly a research preview; the PPL gap between 95.13 and
the 2 B production model's 9.16 is scale, not method.scripts/pretrain_sparse_bitnet_qwen_0p5b.py.MIT, matching the 1bit-systems stack. Derived weights + fine-tunes welcome.