NoTokenLM-Gen-2.5

<div align="center"> <img src="Gen-2.5.png" alt="NoTokenLM Gen-2.5" width="800"/> </div>

NoTokenLM-Gen-2.5

A from-scratch, character-level decoder-only Transformer -- no tokenizer, no BPE, no subword vocabulary.

Usage

from transformers import AutoModelForCausalLM
import torch, json

model = AutoModelForCausalLM.from_pretrained(
    "omurberaisik/NoTokenLM-Gen-2.5", trust_remote_code=True
)
model.eval()

with open("vocab.json") as f: 
    vocab = json.load(f)
c_stoi, c_itos = vocab["c_stoi"], vocab["c_itos"]

prompt = "Once upon a time,"
ids = [c_stoi["<bos>"]] + [c_stoi.get(ch, c_stoi["<unk>"]) for ch in prompt]
out_ids = model.generate_raw(ids, max_new_tokens=150, temperature=0.8, top_k=40)
text = "".join(c_itos[i] for i in out_ids if c_itos[i] not in ("<pad>", "<bos>", "<eos>"))
print(text)

Overview

NoTokenLM-Gen-2.5 is an experimental preview release of an open-source research project exploring how much open-ended text generation ability can be reached with a very small parameter budget when the model operates directly on raw characters instead of a learned subword tokenizer.

• Parameters: 3,163,552 (~3.16M)
• Architecture: Decoder-only Transformer (GPT-style), 5 layers, 2 attention heads, d_model=224, d_ff=896, causal self-attention with KV-cache for generation, weight-tied embedding/output head.
• Vocabulary: 104 symbols, derived directly from the training corpus (raw characters + a small set of control tokens: pad/bos/eos/unk and chat-role markers for a planned instruction-tuning stage). There is no byte-pair encoding, no subword merges, and no fixed dictionary -- every output character is generated one at a time from the model's own character-level probability distribution.
• Training data: ~4.4GB of English text assembled from public-domain books (Project Gutenberg / GITenberg), Simple English Wikipedia, dialogue corpora (BNC, Twitter-style conversational data), and SQuAD-derived question/answer pairs, plus synthetic arithmetic examples.
• Training stage at this checkpoint: pretraining only (27,900 steps of next-character sentence continuation). Supervised fine-tuning (SFT) on chat-formatted QA/math data and a lightweight fluency reward loop are planned for a following checkpoint, not yet applied here.

Why this is worth looking at

Character-level, tokenizer-free language models at this parameter scale (low single-digit millions) are unusual -- the overwhelming majority of small/toy language models at this size either (a) target a narrow, closed task, or (b) don't attempt to produce grammatically real, varied words at all. This model targets open-ended continuation directly: give it a sentence start, and it generates new characters, one at a time, with no predefined word list to fall back on.

What it can currently do

• Produce grammatically well-formed sentence fragments with correct capitalization, quotation marks, apostrophes, and punctuation it was never explicitly told the rules for -- all of this was inferred purely from raw character statistics.
• Generate a wide and varied vocabulary of real, correctly spelled English words, without collapsing into repeating a single dominant word -- a failure mode earlier checkpoints of this project did exhibit.
• Pick up incidental artifacts of its book-derived training data, such as "[Illustration: ...]" markers from scanned public-domain texts, and occasional French phrases that slipped through the English-only filter.

What it cannot reliably do yet (honest limitations)

• Semantic coherence across a full passage. Individual phrases are often locally plausible; the passage as a whole usually is not. Sentences drift topic rapidly.
• Occasional invented/malformed words that look plausible at the character level but are not real English.
• No instruction-following or QA ability yet -- this checkpoint has not been through the SFT stage, so prompting it with a question will not reliably produce a relevant answer.
• No factual grounding. At 3.16M parameters, the model has nowhere near the capacity to store or recall real-world facts reliably.

This is explicitly a preview / experimental checkpoint, not a finished model. It is being shared to document the project's progress transparently, including its current limitations, rather than to claim a finished product.

20 sample generations

All samples below were produced with top-k=40 sampling, a repetition penalty, and a no-repeat-4-gram filter, at temperature 0.7 or 0.9 as noted. They are shown unedited, including their imperfections.

1. (temp 0.7) "The history of science shows that" -> ideal, while the subject home was often from Mexico; my soul and should demand, I objet this. There
2. (temp 0.9) "The history of science shows that" -> terms of Dominiisons, Internalization future, he thought occupying made a framing. [1]Illustrativen
3. (temp 0.7) "The weather today is" -> implesively advanced, are three flowers, the rubbius to think it or thy personal suveral words, cou
4. (temp 0.9) "The weather today is" -> confident to-day. 'You may require explain, she removed," He stikes the back wife, suddenly. But wh
5. (temp 0.7) "She walked into the room and" -> there shape, as Duch reached her blushed, she had another Roman, biggertted Human hardly. I know hims
6. (temp 0.9) "She walked into the room and" -> count pernished with night, out by doing longitude; prefinded. T exactly when May, grew over her
7. (temp 0.7) "Once upon a time," -> hence to exclaim, sir si bottreva, white me en cur gendry aspector so she stood. I have introduced
8. (temp 0.9) "Once upon a time," -> Puy inventure. Querit: N'y avait-ail. Di a Pouse!... Non, I sera diable No muste et Mu
9. (temp 0.7) "In the beginning, there was" -> no more currase, and, to be expectedly. This dark, how frequently took cover below, appeared to.[17
10. (temp 0.9) "In the beginning, there was" -> a puller's place found." {=D}IINCRESIGHTH, is impossibly, to delight-rushed bands. [=D]INCLUSTRATOR
11. (temp 0.7) "The most important thing to remember is" -> filled the wit to resume. His place took a week through that refused by an increase black. Harry,
12. (temp 0.9) "The most important thing to remember is" -> And hy. A numbest going: Letter, I give it only face, Too throwing On the feemasure that
13. (temp 0.7) "He looked at her and said" -> "I will not be a comely fair; you with help; anything generally, I desired to see--yes," she suppo
14. (temp 0.9) "He looked at her and said" -> "Not well 'ah!" "I remember if I very we do addition?" Edmundius.--The next day. Now, there was
15. (temp 0.7) "The old house stood" -> and introduced by heavy girls. They are vexplained, and, they have not missionable. To be ingratua
16. (temp 0.9) "The old house stood" -> by war, which might not less than their birth. he fully loss slipped, and thy tife face laws from hi
17. (temp 0.7) "My favorite book is about" -> ; and a cruims, beginning, of plonged, as a horse delight. [Illustration: That which they migh invit
18. (temp 0.9) "My favorite book is about" -> 1 replaces." But, had been distinguished. They have gone backward's soul-kept the sophowood,, to re
19. (temp 0.7) "When the sun rose," -> and think all that such composion. [Sidenote: The beginning which I have thrown in although, as i
20. (temp 0.9) "When the sun rose," -> nothing to have been perceding; Whether amounts of this time. Eleven my coach that could respect

Model checkpoint status

phase: sft (about to begin)
global_step: 27,900
pretrain_steps_done: 27,900
sft_steps_done: 0

This checkpoint represents the end of the pretraining phase, right before instruction/QA fine-tuning begins. A future checkpoint with SFT and fluency-reward tuning applied is expected to follow.

License / usage

Licensed under Apache License 2.0. Trained entirely on public-domain and freely licensed text sources. Provided for research and educational purposes; no warranty of fitness for any particular purpose. Given the current limitations described above, this model should not be relied on for factual, medical, legal, or safety-critical use.