NoTokenLM-Gen-3.6

NoTokenLM-Gen-3.6

A 13.9-million-parameter, byte-level, tokenizer-free language model trained on an English + French text mix. No subword vocabulary, no BPE — just raw UTF-8 bytes in, raw UTF-8 bytes out.

This is part of the NoTokenLM family: a series of small models built around one guiding question — how much can a genuinely small model do, if the architecture and training are done carefully, without leaning on scale to cover for weak design?

If you're looking for a model that answers questions, does math, or holds a conversation — this isn't that, and this card will tell you exactly why not. If you're curious what a ~14M-parameter byte-level transformer does when it's trained on a broad mix of educational web text, encyclopedic text and simple stories — keep reading.

---

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO_ID = "omurberaisik/NoTokenLM-Gen-3.6"
model = AutoModelForCausalLM.from_pretrained(REPO_ID, trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(REPO_ID, trust_remote_code=True)

text = model.generate_bytes("Once upon a time, there was a little girl named Mia.",
                             max_new_tokens=200, temperature=0.5, top_k=40)
print(text)

trust_remote_code=True is required — this model ships its own architecture code (byte-level I/O, RoPE, RMSNorm, SwiGLU, plus a short input convolution, QK-norm, value residual and per-head output gates) alongside the weights. There is no subword tokenizer; AutoTokenizer here is a thin byte<->id mapping provided purely so the model loads the standard Hugging Face way.

An interactive Colab/Kaggle notebook with a prompt box and sampling controls is in this repo as notebook.ipynb.

Recommended sampling settings are in How to actually run this thing below.

What is this checkpoint, actually?

Gen-3.6 is a from-scratch pretraining run of a 13.87M-parameter byte-level transformer (12 layers, d_model=320). This card describes step 13,000, and every number in it comes from that checkpoint's EMA (exponential moving average) weights, which is the standard choice for evaluation at this scale.

Training volume: 13,000 steps × (batch 16 × gradient accumulation 4 × 1,024 bytes) ≈ 0.85 GB of text. The best validation loss recorded during the run was 0.8915 (nats per byte, ≈1.29 bits per byte, on a held-out split of the training mix); the released weights are the final step-13,000 EMA weights. Peak learning rate 1e-3, 500 warmup steps, weight decay 0.1. Pretraining only — no instruction tuning.

Unlike Gen-4.5, which was trained on narrow children's-story text only, Gen-3.6 was trained on a broad mix: educational web text, Wikipedia, simple stories and textbook-style text, in English and French (details in Training data). That difference drives most of what you'll see below — including where it is better than Gen-4.5 and where it is worse.

---

How well does it actually write? (1,000-prompt evaluation)

We generated 1,000 completions — 1,000 distinct prompts, one generation each, temperature 0.5, top-k 40, 35 new bytes — and every one of the 1,000 outputs was read individually and sorted into one of three categories. The prompt set, prompt order, ids and generation settings are identical to those used for Gen-4.5's evaluation. No filtering, no cherry-picking; everything below is the real distribution.

Grading criteria (same strict standard as prior NoTokenLM evaluations — "grammatically fine but says nothing coherent" does not count as a win):

Result: 78.0% fully coherent, 99.5% grammatically correct overall, on a test set with no repeated prompts and no simplification.

Roughly half of the middle category (112 of 215) is a pronoun/gender mismatch, covered in detail in the next section; the rest is fluent text whose content contradicts the scene.

The labels were assigned by a single annotator. The pronoun check and the benchmarks below do not depend on them.

The label for every one of the 1,000 outputs is in gen36_13000_1000_test_outputs.json under the category field.

Real examples, unedited, that landed in "fully coherent":

"On a quiet evening, Jack spent the whole morning at the park."
  -> " He had a big smile on his face and"

"One special day, Rosie spent the whole morning at the campsite."
  -> " She was so excited to see the firs[t]"

"One special day, Ben was playing in the attic when something strange happened."
  -> " He got scared and ran away. He ran"

"One sunny morning, Anna stopped to rest in the village."
  -> " She saw a big puddle on the ground"

Real examples that landed in "grammar OK, meaning breaks down" (also unedited):

"One sunny morning, Silas was looking for a bag of apples in the backyard."
  -> " She saw a big bag of apples and sh[e]"
  <- pronoun doesn't match the character

"On a rainy Tuesday, a gentle horse saw a wooden box lying in the attic."
  -> " The box was a big dog! The dog was"
  <- grammatically fine, but the box turns into a dog

"Lily was playing in the attic when something strange happened."
  -> " The hill was a little boy who was "
  <- fluent, but nothing in the scene supports it

"Without any warning, Tom was walking on the farm road and started to sing."
  -> "\n\n\"Wow, look at all the warning!\" T"
  <- picks up a word from the prompt and turns it into nonsense

A pronoun/gender mismatch a sentence or clause after a character is introduced is the single most common failure inside this category.

Real examples that landed in "grammar breaks down" (all five, unedited):

"During the summer, Nova was cleaning up in the treehouse after a long day."
  -> " After the summer were ready to sit"         <- subject/verb agreement collapses

"Not far from home, Tom had never seen a wooden spoon before."
  -> " He had far from his house, and he "        <- broken verb phrase

"Just before sunset, Grace was making a plan to find a wooden box."
  -> " All of his living room were not cl"        <- agreement collapses, then trails off

"Just before sunset, a small bird was walking in the kitchen and started to sing."
  -> " Just as Just before sunset, Just b"        <- repeats the prompt's opening phrase

"Without any warning, a gentle horse saw a paper boat lying by the lake."
  -> " The paper leaves the paper and the"        <- word-salad repetition

---

Pronoun agreement, measured automatically

A generation can be fluent and still give a named character the wrong pronoun. To measure this directly, we took every prompt in the 1,000-prompt set that has a single named character with a clearly gendered name, and checked the first he/she pronoun in the completion against that name. 522 of the 1,000 completions contained such a pronoun.

The female-name number hides a sharp split by name:

So the model has learned the gender of common names well, and defaults to "he" for less common female names. Each name appears roughly 15–30 times in the set. (Willow and June can also be unisex; even without them the pattern holds — Hazel 1/17, Nora 1/14, Poppy 1/12, Zoe 2/15.) Ambiguous names (Wren) and prompts with two named characters are excluded, because a pronoun can legitimately refer to either character.

---

Real-world benchmark results

The coherence grading above is our own methodology. To see how the model does on external benchmarks, we ran it on two established suites. Neither involves code or math. Sample sizes are noted, and the sampling error is not small — read differences of a few points as noise.

BLiMP (Benchmark of Linguistic Minimal Pairs)

BLiMP scores whether a model assigns higher likelihood to a grammatical sentence than to a minimally-different ungrammatical one, across 67 phenomena. We ran all 67 paradigms with 30 randomly sampled pairs each (2,010 pairs, fixed seed), and list the baselines from the BLiMP paper for reference.

Gen-3.6 lands in the LSTM / Transformer-XL range. Against Gen-4.5 on the same pairs it is ahead on 34 paradigms, behind on 23, and tied on 10 — a gap that is within sampling error overall. (Gen-4.5's own card reports 66.8% on a different sample of pairs.)

Strengths and weaknesses follow the same shape as the rest of this card: 19 of 67 paradigms score 90% or higher, mostly local agreement and licensing (determiner_noun_agreement_2, principle_A_case_1, sentential_negation_npi_licensor_present, wh_vs_that_no_gap_long_distance all 100%). Four paradigms score 20% or below, all requiring long-range or structurally complex dependencies: wh_vs_that_with_gap_long_distance (3.3%), coordinate_structure_constraint_complex_left_branch (10.0%), existential_there_quantifiers_2 (10.0%), matrix_question_npi_licensor_present (16.7%).

Full BLiMP re-run: all 1,000 pairs per paradigm (67,000 pairs total)

The 30-pair sample above was later re-run against the full 1,000-pair set for every paradigm (67,000 pairs total), scored with a from-scratch NumPy re-implementation of the forward pass against the official model.safetensors weights, and checked against the original BLiMP repo's raw baseline files (GPT-2-large, Transformer-XL, LSTM, 5-gram) and the original human/MTurk data.

This matches the ranking and approximate values reported in the original BLiMP paper (Warstadt et al., 2020), and the 68.3% here is consistent with the 68.9% 30-pair-sample figure above — both fall inside the same 66.8–70.9% confidence interval, so the difference is sampling noise, not a discrepancy.

Paradigms won, out of 67:

Gen-3.6 beats GPT-2-large — a model 56x larger — on 13 of 67 paradigms, almost all local morphological/syntactic patterns learnable from short-range statistics alone: ellipsis_n_bar_2 (+9.5), only_npi_scope (+8.2), regular_plural_subject_verb_agreement_2 (+5.7), irregular_past_participle_verbs (+3.9), determiner-noun agreement, and reflexive/principle-A binding, among others.

Where it falls furthest behind human performance is the mirror image: long-range dependencies and island constraints, e.g. matrix_question_npi_licensor_present (−88.0 pts), left_branch_island_simple_question (−85.7 pts), coordinate_structure_constraint_complex_left_branch (−70.0 pts) — exactly where a 1024-byte context and 13.9M parameters would be expected to run out of room.

The complete 67-paradigm, 6-model breakdown (including per-paradigm win/loss flags) is in blimp_all_models_comparison_en.csv and the full write-up in blimp_report_en.md.

LAMBADA (long-context final-word prediction)

LAMBADA gives a paragraph and asks for the final word. We scored 500 randomly sampled passages from the LAMBADA test split (fixed seed) with greedy decoding: the first word of the continuation, punctuation stripped, lowercase, exact match.

On identical passages Gen-3.6 gets 29 right that Gen-4.5 misses and misses 12 that Gen-4.5 gets right (sign test p ≈ 0.01). The likely reason is data, not architecture: LAMBADA passages are fiction, and Gen-3.6 saw web and encyclopedic text as well as stories. (Gen-4.5's card reports 12.0% on a 100-passage sample under its own scoring; the two numbers are not directly comparable.) In absolute terms this is still very far below even GPT-2-small.

HellaSwag was not evaluated for this checkpoint.

Bits per byte on held-out text

A model-agnostic measure of how well it predicts raw text (first 1,000 bytes of each passage; lower is better; not comparable across tokenizers or other corpora):

---

What it knows, and what it doesn't

We probed the model directly with short factual and pattern prompts (temperature 0.5, top-k 40, 4 seeds each):

The model does capture frequent surface patterns. Measured directly as next-item probability (no sampling):

Common counting patterns are picked up; the rule behind them is not (compare 2, 4, 6, 8 with 5, 10, 15, 20). Anything that needs a fact, a rule, or a step of reasoning is out of reach.

---

Language behavior

The training mix targets 58% English and 42% French by byte share, and the data pipeline has no language-ID filter. What that looks like in practice, from informal testing (a handful of prompts, not a benchmark):

• English prompts stay in English and in a narrative or expository register.
• French prompts produce French, but in a news/encyclopedic register rather than a story register, with fluent-looking sentences that don't hold together. French is not usable for story generation at this checkpoint.
• Prompts in other languages (Spanish, Portuguese, Italian, German, Dutch, Turkish, Hausa) are not reliably continued in the prompt's language: the model drifts into English, French, Italian-like text, or falls into repetition loops. Earlier in training the same prompts could produce fluent Hausa, so non-target-language text is clearly present in the training data and the model picks it up.
• Broken byte sequences ( , a UTF-8 double-encoding artifact) appear in 2 of the 1,000 evaluation outputs, which suggests some source text contains encoding errors.

---

What it's actually good at

• Well-formed English at the sentence level. Grammar is largely intact, and local syntactic agreement is strong (19 of 67 BLiMP paradigms at 90%+).
• Staying in a narrative register on story-style prompts. Given a concrete character and action, continuations are usually on-topic for a sentence or two.
• Predicting general text. At 1.89 bits per byte on held-out fiction, it models everyday English prose far better than a narrow story-only model would.
• Occasional multi-sentence stretches. In informal 240-byte generations it stays on topic for a couple of sentences before repeating or drifting; this has not been measured at scale, so treat it as a directional observation.

What it's not good at, and why

• No usable world knowledge. Despite seeing Wikipedia and educational web text, ~0.85 GB of training text at 13.9M parameters is far too little to store facts. It writes plausible-sounding text that is wrong.
• Pronoun and gender consistency. The most common failure, and very name-dependent: reliable for common names, close to random for less common female names (see above).
• Long-range structure. LAMBADA at 11.4% and near-chance BLiMP paradigms on long-distance dependencies point at the same gap.
• Semantic drift. Fluent continuations that contradict the scene: objects turning into other objects, inanimate things acting as people, characters addressing themselves.
• Register drift. Occasional jumps out of story mode into encyclopedic or expository text.
• Non-English output. See above; French in particular is not reliable.
• Rare malformed output. Repeating the prompt's opening phrase, broken verb inflection, or garbled bytes.

---

How to actually run this thing

Recommended: temperature 0.5, top-k 40. The 1,000-prompt evaluation and the probes above were produced at these settings (BLiMP is likelihood-based and LAMBADA is greedy). Greedy decoding on open-ended prompts tends to fall into repetition loops; much higher temperatures increase drift and malformed output.

---

Architecture details

Training data

Target byte shares (the loader normalizes and balances toward these):

Documents are kept if they are 200–20,000 bytes, at least half alphabetic characters, and under a non-ASCII byte cap (8% for English sources, 25% for French). There is no language-ID filter, no dedicated code or math corpus, and no instruction-following data.

---

Part of the NoTokenLM family — small models, built and evaluated honestly.

---

License

This project is licensed under the Apache License 2.0. Please refer to the LICENSE file for the full text of the license.