comp-vs-like-1B-topdowncomp-multi-128k

comp-vs-like-1B-topdowncomp-multi-128k

A 1.2B-parameter Llama-style language model trained with the TopDownComp tokeniser, from Objective vs. Search: Decomposing What Makes a Good Tokeniser (EMNLP 2026).

The paper separates two axes that BPE and UnigramLM confound. This model occupies the cell:

This model: TopDownComp — compression objective, top-down pruning.

All models in the collection are identical except for the tokeniser, so differences in bits-per-byte and BLiMP are attributable to tokeniser design alone.

Training

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("AhmetcanYvz/comp-vs-like-1B-topdowncomp-multi-128k")
model = AutoModelForCausalLM.from_pretrained("AhmetcanYvz/comp-vs-like-1B-topdowncomp-multi-128k")

ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=20)[0]))

The tokeniser is bundled with the model, so no separate download is needed.

Caveats

This is a research model trained on 20B tokens to study tokenisation. It is not instruction-tuned, not filtered for safety, and not intended for deployment.

Code

https://github.com/Ahmetcanyvz/comp-vs-like

Citation

@inproceedings{yavuz2026objective,
  title     = {Objective vs. Search: Decomposing What Makes a Good Tokeniser},
  author    = {Yavuz, Ahmetcan and Meister, Clara and Pimentel, Tiago},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026}
}