llm-fix-min-affine-recoded-minimal-code-table-free

Affine-Recoded Minimal Code Table-Free Model

Research checkpoint for the paper:

Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes

Model variant

This repository contains the fully table-free affine-recoded minimal binary-code model.

The model does not use an input embedding table. Instead, token codes are computed directly from token IDs.

For each token ID t, the model computes:

c(t) = bin_16(t)

and then applies a fixed invertible affine recoding over GF(2):

c_tilde(t) = A c(t) xor b

where:

• A is an invertible binary matrix in GL(16, 2)
• b is a fixed binary shift vector

The resulting 16-dimensional binary code is tiled to model width 1024.

The model uses:

0 trainable input-embedding parameters
0 input embedding table

The output projection remains standard and trainable.

Architecture

• decoder-only Transformer
• vocabulary size: 65,536
• model width: 1024
• number of layers: 32
• number of attention heads: 32
• context length: 1024
• rotary positional embeddings
• GELU activations
• untied trainable output projection

Loading example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "Bochkov/llm-fix-min-affine-recoded-minimal-code-table-free"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

prompt = "Question: What is the capital of UK?\nAnswer:"
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)

with torch.no_grad():
    output_ids = model.generate(input_ids, max_new_tokens=3, do_sample=False)

print(tokenizer.decode(output_ids[0].tolist()))

Standardized base-model evaluation

The checkpoint was evaluated as a base causal language model with EleutherAI LM Evaluation Harness v0.4.10.

Evaluation protocol:

• Hugging Face backend: hf
• maximum context length: 1,024
• add_bos_token=False
• no chat template
• deterministic likelihood-based evaluation
• harness seeds: 0,1234,1234,1234
• base checkpoints only; no SFT or instruction checkpoints

The three paper checkpoints form the controlled architectural comparison. SmolLM2-135M and SmolLM2-360M are external reference models, not matched baselines: they use different architectures, tokenizers, training mixtures, and much larger pretraining budgets. SmolLM2-135M was trained on approximately 2T tokens and SmolLM2-360M on approximately 4T tokens, whereas the paper checkpoints saw approximately 16–17B tokens. Their scores therefore provide context for absolute capability and must not be interpreted as isolating the effect of the input parameterization.

Perplexity values should be interpreted especially cautiously across different tokenizers. The primary controlled comparison is among the three paper models, which share the same tokenizer, data pipeline, and architecture.

Input-interface audit

This checkpoint has no input embedding table. Token codes are generated algorithmically from token IDs, and the fixed affine matrix and shift are registered as non-trainable buffers.

import torch
from transformers import AutoModelForCausalLM

repo_id = (
    "Bochkov/"
    "llm-fix-min-affine-recoded-minimal-code-table-free"
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
).cpu().eval()

print("get_input_embeddings():", model.get_input_embeddings())
print(
    "input-code parameters:",
    [
        name
        for name, _ in model.named_parameters()
        if name.startswith("input_code.")
    ],
)
print(
    "input-code buffers:",
    [
        name
        for name, _ in model.named_buffers()
        if name.startswith("input_code.")
    ],
)

ids = torch.arange(model.config.vocab_size).unsqueeze(0)

with torch.no_grad():
    codes = model.input_code.encode_bits(ids)[0]

weights = 1 << torch.arange(model.config.code_bits)
packed = (codes.long() * weights).sum(dim=-1)

print("code shape:", tuple(codes.shape))
print("unique values:", torch.unique(codes).tolist())
print("unique codes:", torch.unique(packed).numel())
print("collisions:", model.config.vocab_size - torch.unique(packed).numel())

assert model.get_input_embeddings() is None
assert not hasattr(model, "token_embeddings")
assert not any(
    name.startswith("input_code.")
    for name, _ in model.named_parameters()
)
assert torch.all((codes == 0) | (codes == 1))
assert torch.unique(packed).numel() == model.config.vocab_size

Expected audit properties:

get_input_embeddings(): None
input-code parameters: []
input-code buffers: ['input_code.bit_positions', 'input_code.A_gf2', 'input_code.b_gf2']
code shape: (65536, 16)
unique values: [0.0, 1.0]
unique codes: 65536
collisions: 0

Intended use

This checkpoint is provided for reproducibility. It demonstrates that the fixed minimal-code input interface remains viable even when the canonical token-ID binary code is randomly recoded by an invertible affine transform.

Limitations

This model is a research checkpoint. It is not intended for deployment. It may produce incorrect, biased, unsafe, or nonsensical outputs.

Training data

The model was trained on the same FineWeb-Edu + Cosmopedia mixture used for the matched comparisons in the paper. Dataset terms and licenses are those of the original datasets.

---

🧑‍🔬 Citation & Concept

If you use this model or the underlying concepts in your research, please cite our work:

@misc{bochkov2026languagemodelstrainableinput,
      title={Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes}, 
      author={A. Bochkov},
      year={2026},
      eprint={2605.09751},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.09751}, 
}