nanochat-760M-backward

nanochat d20 backward (561M parameters)

A base language model trained right to left: it predicts the previous token given everything that comes after it. Give it the end of a text and it writes what came before.

It is a depth-20 nanochat GPT trained with onanchat, a fork that adds backward and bidirectional training. It shares its architecture, tokenizer, data and schedule with a matched forward model and with nanochat-760M-bidirectional.

Size note: the repository name says 760M, but the model has 560,988,160 parameters. The name is kept so existing links keep working.

How the reversal works

During training, nanochat streams documents into one token stream, each document starting with <|bos|>. It cuts the stream into 16,385-token chunks (8 rows × 2,048 tokens, plus one). For this model every chunk was reversed except its first token and then split into rows. The text therefore runs right to left, and each document's <|bos|> now appears after that document, marking where it began.

To use the model:

1. Tokenize the ending you want, reverse the token list and put <|bos|> in front.
2. Sample as usual. Each new token is the one that comes before the text so far.
3. A generated <|bos|> means the model has reached the start of a document, so stop there.
4. Reverse the generated tokens and put them in front of your ending.

Reverse tokens, not characters. The tokenizer is the ordinary forward tokenizer.

Usage

Requires a checkout of onanchat (tested at commit 0e99157), PyTorch and huggingface_hub. The Rust tokenizer trainer is not needed for inference.

# run from the root of an onanchat checkout, or put it on PYTHONPATH
import json, sys, types, torch
sys.modules.setdefault("rustbpe", types.ModuleType("rustbpe"))  # only needed to *train* tokenizers
from huggingface_hub import snapshot_download
from nanochat.gpt import GPT, GPTConfig
from nanochat.tokenizer import RustBPETokenizer

def load(repo, device="cuda"):
    path = snapshot_download(repo, allow_patterns=["model.pt", "meta.json", "tokenizer.pkl", "token_bytes.pt"])
    meta = json.load(open(f"{path}/meta.json"))
    with torch.device("meta"):
        model = GPT(GPTConfig(**meta["model_config"]))
    model.to_empty(device=device)
    model.init_weights()  # builds the rotary tables; weights are overwritten below
    state = torch.load(f"{path}/model.pt", map_location=device)
    state = {k.removeprefix("_orig_mod."): v.float() if device == "cpu" else v for k, v in state.items()}
    model.load_state_dict(state, strict=True, assign=True)
    return model.eval(), RustBPETokenizer.from_directory(path), meta

@torch.no_grad()
def sample(model, ids, n, stop=None, temperature=0.8, top_k=50):
    x = torch.tensor([ids], device=next(model.parameters()).device)
    out = []
    for _ in range(n):
        with torch.autocast("cuda", dtype=torch.bfloat16, enabled=x.is_cuda):
            logits = model(x)[0, -1].float() / temperature
        v, _ = logits.topk(top_k)
        logits[logits < v[-1]] = -float("inf")
        t = torch.multinomial(logits.softmax(-1), 1)
        if t.item() == stop:
            break
        out.append(t.item())
        x = torch.cat([x, t[None]], 1)
    return out

model, tok, meta = load("raghavt/nanochat-760M-backward")
bos = tok.get_bos_token_id()
ending = " and that is why the experiment had to be repeated the following year."
before = sample(model, [bos] + tok.encode(ending)[::-1], 80, stop=bos)
print(tok.decode(before[::-1]) + ending)

One sample (yours will differ):

…However, it was obvious that the growth rate had not changed over the previous year, and that is why the experiment had to be repeated the following year.

load also works with device="cpu", but it is slow.

Architecture

Training

• Data: karpathy/fineweb-edu-100b-shuffle, a re-sharded copy of FineWeb-Edu, with 100 shards downloaded. Following nanochat, the last shard is held out for validation.
• Optimizer: Muon for the transformer matrices (lr 0.02) and AdamW for the input embedding (lr 0.2) and unembedding (lr 0.004). No weight decay, gradient clipping at 1.0.
• Batch: 262,144 tokens per step (micro-batch 8 × 2,048).
• Schedule: 20,000 steps with no warmup and linear warmdown over the last 20%.
• Stopped early: the run stopped at step 19,000, partway through warmdown, so the learning rate never fully decayed. The matched forward and bidirectional runs completed all 20,000 steps.
• Wall-clock: about 20.6 h, recorded in meta.json.
• Run: backward_d20_main. The full configuration is in meta.json under user_config.

Intended use and limitations

• This is a research model for studying direction in language modelling and for proposing text that leads up to a given ending. It is not an assistant and has no safety tuning.
• It is a small model trained on about 5B tokens. Its text is often fluent but vague, and facts are unreliable.
• It inherits the content and biases of FineWeb-Edu. It is English-centric.
• Most bad outputs come from wrong direction handling: forgetting to reverse, reversing characters instead of tokens, or not stopping at a generated <|bos|>.
• Validation bits per byte measure the same held-out text read in each model's own direction. Matched forward and backward models are roughly comparable on this number, but not on any downstream benchmark.

Related models

• nanochat-760M-bidirectional: one model for both directions, chosen with a marker token.
• nanochat-760M-backward-sft: backward model after chat fine-tuning.

Citation

@misc{nanochat-d20-backward,
  author = {Raghav},
  title = {nanochat d20 backward: a right-to-left nanochat base model},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/raghavt/nanochat-760M-backward}
}

Built on nanochat by Andrej Karpathy (MIT).