nanochat-760M-bidirectional

nanochat d20 bidirectional (561M parameters)

A single base language model trained on both left-to-right and right-to-left text. A marker token picks the direction:

• <|forward|>: ordinary next-token prediction. Give it a beginning and it continues.
• <|backward|>: previous-token prediction. Give it an ending and it writes what came before.

It is a depth-20 nanochat GPT trained with onanchat. It shares its architecture, tokenizer, data and schedule with a matched forward model and with nanochat-760M-backward. Its training budget is the same as each single-direction model's, split between the two directions.

Size note: the repository name says 760M, but the model has 560,988,160 parameters. The name is kept so existing links keep working.

How directions were trained

nanochat streams documents into one token stream, each document starting with <|bos|>. It cuts the stream into 16,385-token chunks (8 rows × 2,048 tokens, plus one). This model alternated chunks:

• Even chunks: first token, <|forward|>, then the text in order.
• Odd chunks: first token, <|backward|>, then the rest of the chunk reversed.

The marker sits at the start of each chunk's first row only. The other seven rows begin mid-stream with no marker, so the model must also infer direction from the text itself. In backward chunks each document's <|bos|> appears after that document, marking where it began.

Prompt formats:

• Forward: [<|bos|>, <|forward|>] + tokens.
• Backward: [<|bos|>, <|backward|>] + reversed(tokens). Stop at a generated <|bos|>, then reverse the output.

Reverse tokens, not characters.

Usage

Requires a checkout of onanchat (tested at commit 0e99157), PyTorch and huggingface_hub.

# run from the root of an onanchat checkout, or put it on PYTHONPATH
import json, sys, types, torch
sys.modules.setdefault("rustbpe", types.ModuleType("rustbpe"))  # only needed to *train* tokenizers
from huggingface_hub import snapshot_download
from nanochat.gpt import GPT, GPTConfig
from nanochat.tokenizer import RustBPETokenizer

def load(repo, device="cuda"):
    path = snapshot_download(repo, allow_patterns=["model.pt", "meta.json", "tokenizer.pkl", "token_bytes.pt"])
    meta = json.load(open(f"{path}/meta.json"))
    with torch.device("meta"):
        model = GPT(GPTConfig(**meta["model_config"]))
    model.to_empty(device=device)
    model.init_weights()  # builds the rotary tables; weights are overwritten below
    state = torch.load(f"{path}/model.pt", map_location=device)
    state = {k.removeprefix("_orig_mod."): v.float() if device == "cpu" else v for k, v in state.items()}
    model.load_state_dict(state, strict=True, assign=True)
    return model.eval(), RustBPETokenizer.from_directory(path), meta

@torch.no_grad()
def sample(model, ids, n, stop=None, temperature=0.8, top_k=50):
    x = torch.tensor([ids], device=next(model.parameters()).device)
    out = []
    for _ in range(n):
        with torch.autocast("cuda", dtype=torch.bfloat16, enabled=x.is_cuda):
            logits = model(x)[0, -1].float() / temperature
        v, _ = logits.topk(top_k)
        logits[logits < v[-1]] = -float("inf")
        t = torch.multinomial(logits.softmax(-1), 1)
        if t.item() == stop:
            break
        out.append(t.item())
        x = torch.cat([x, t[None]], 1)
    return out

model, tok, meta = load("raghavt/nanochat-760M-bidirectional")
bos = tok.get_bos_token_id()
fwd, bwd = tok.encode_special("<|forward|>"), tok.encode_special("<|backward|>")

start = "The history of the bicycle"
print(start + tok.decode(sample(model, [bos, fwd] + tok.encode(start), 60)))

ending = " and that is why the experiment had to be repeated the following year."
before = sample(model, [bos, bwd] + tok.encode(ending)[::-1], 80, stop=bos)
print(tok.decode(before[::-1]) + ending)

One sample from each mode (yours will differ):

forward: The history of the bicycle is full of stories about inventors, inventors, and inventions. In a nutshell, an inventor is someone who comes up with a new idea or product…

backward: …In fact, after one year, the group that was used in the experiment did as well, and that is why the experiment had to be repeated the following year.

Architecture

Training

• Data: karpathy/fineweb-edu-100b-shuffle, a re-sharded copy of FineWeb-Edu, with 100 shards downloaded. The last shard is held out for validation.
• Optimizer: Muon for the transformer matrices (lr 0.02) and AdamW for the input embedding (lr 0.2) and unembedding (lr 0.004). No weight decay, gradient clipping at 1.0.
• Batch: 262,144 tokens per step (micro-batch 8 × 2,048).
• Schedule: 20,000 steps with no warmup and linear warmdown over the last 20%, completed.
• Wall-clock: about 21.0 h, recorded in meta.json.
• Run: backward_d20_bidir_main, model tag d20_bidirectional. The full configuration is in meta.json under user_config.

Intended use and limitations

• This is a research model for comparing directions inside one network. It is not an assistant and has no safety tuning.
• Its validation number is not comparable to the single-direction models' (0.8430 forward, 0.8647 backward), because it is scored on the mixed-direction stream.
• Each direction saw about half the tokens a single-direction model saw.
• It is a small model trained on about 5B tokens. Facts are unreliable, and it inherits the content and biases of FineWeb-Edu.

Related models

• nanochat-760M-backward: dedicated right-to-left base model.
• nanochat-760M-backward-sft: backward model after chat fine-tuning.

Citation

@misc{nanochat-d20-bidirectional,
  author = {Raghav},
  title = {nanochat d20 bidirectional: one nanochat base model for both directions},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/raghavt/nanochat-760M-bidirectional}
}

Built on nanochat by Andrej Karpathy (MIT).