Downloads · 30 days
0
fablog-tokenizer-16k
JugsMa/fablog-tokenizer-16k
fablog-tokenizer-16k is a machine learning model from JugsMa. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
Byte-level BPE tokenizer for machine-level logs (fab tool events, ATE/wafer-sort, burn-in/memory test, GPU telemetry). Trained with buildlogtokenizer.py on a synthetic fab-log corpus (17.7 MB, 4,000 documents) — retra…
Access
Public
Updated Sep 7, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json359 KB · 99%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Sep 7, 2026
- Updated
- Sep 7, 2026
- SHA
- 52291f53
Model card
From the Hugging Face model README
fablog-tokenizer-16k
Byte-level BPE tokenizer for machine-level logs (fab tool events, ATE/wafer-sort, burn-in/memory test,
GPU telemetry). Trained with build_log_tokenizer.py on a synthetic fab-log corpus (17.7 MB,
4,000 documents) — retrain on real data before production use.
Design
- Byte-level BPE, no unicode normalizer → encode→decode is byte-exact, no
<unk>ever - Digits pre-split in groups of 1–3 (Llama-3 style):
2026→202,6;12.04→12,.,04 =isolated so field names (pwr=,lot=) never fuse with values- Specials:
<|endoftext|>(id 0, document separator, also BOS),<|pad|>(id 1); vocab padded with<|reserved_k|>tokens to a multiple of 128 - Vocab: 2,944 (< 65,536 → tokens fit in uint16 for datatrove/Nanoset storage)
Measured fertility (tokens/byte on the training-domain sample)
| tokenizer | tok/byte | tok/line |
|---|---|---|
| HuggingFaceTB/SmolLM2-135M | 0.635 | 69.2 |
| bigcode/starcoder2-3b | 0.620 | 67.6 |
| gpt2 | 0.524 | 57.1 |
| this tokenizer | 0.475 | 51.7 |
(~29% of corpus bytes are digits/hex — entropy-limited floor.)
Usage
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("JugsMa/fablog-tokenizer-16k")
ids = tok("<REC> ts=2026-07-03T13:04:15.092Z tool=ETCH07 status=OK")["input_ids"]
assert tok.decode(ids) == "<REC> ts=2026-07-03T13:04:15.092Z tool=ETCH07 status=OK"
For nanotron/datatrove preprocessing, pass the Hub id directly to
tools/preprocess_data.py --tokenizer-name-or-path <your-namespace>/fablog-tokenizer-16k --eos-token "<|endoftext|>".
- Library
- transformers
- Created
- Sep 7, 2026
- Updated
- Sep 7, 2026