bert-base-uncased-hxq

BERT-base-uncased-HXQ

4.0x smaller from FP32. MLM accuracy 61.0%. First encoder-only model compressed with HXQ.

BERT-base-uncased compressed from 421 MB to 128 MB. Masked language modeling accuracy matches the dense baseline. Same codec that compresses Transformers, SSMs, Hybrids, MoEs, and vision models.

Install and Run

pip install "helix-substrate[hf]"

import helix_substrate
from transformers import BertForMaskedLM, BertTokenizer

model = BertForMaskedLM.from_pretrained("EchoLabs33/bert-base-uncased-hxq")
tokenizer = BertTokenizer.from_pretrained("EchoLabs33/bert-base-uncased-hxq")

inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")
outputs = model(**inputs)
pred = outputs.logits[0, 5].argmax()
print(tokenizer.decode(pred))  # paris

Downstream Benchmarks

Masked language modeling on WikiText-2 (500 randomly masked tokens):

Deltas within sampling noise. Task performance preserved after 4.0x compression.

Compression Benchmark

Verification Status

• Compression receipt: PASS -- 75 compressed, cos 0.999+
• Conversion receipt: PASS (Gate 1 + Gate 2)
• Downstream eval: PASS -- paired dense/HXQ on WikiText-2 MLM

Architecture Details

BERT-base-uncased is an encoder-only Transformer:

• 12 layers, hidden_size=768, 12 attention heads
• 110M parameters
• Trained on masked language modeling + next sentence prediction

All 75 linear layers (attention Q/K/V/O, MLP intermediate/output, pooler, classification head) are compressed. Embedding layers (word, position, token_type), layer norms, and biases are stored at full precision.

Why This Matters

BERT is the first encoder-only model compressed with HXQ. The same codec now covers:

Six architecture families. One codec. One pip install.

Citation

@software{helix_substrate_2026,
  title={Helix Substrate: Universal Weight Compression via HelixCode},
  author={EchoLabs},
  year={2026},
  url={https://github.com/echo313unfolding/helix-substrate}
}

License

Apache 2.0 (inherited from google-bert/bert-base-uncased).