nanochat-760M-backward-sft

nanochat d20 backward SFT (561M parameters)

The right-to-left nanochat model after chat supervised fine-tuning (SFT). It reads conversations in reverse. The most useful things it does:

• Given an assistant reply, it writes the user message that probably prompted it.
• Given what comes later in a conversation, it writes the assistant turn that came before.

It is a depth-20 nanochat GPT trained with onanchat. It was fine-tuned from the backward line of nanochat-760M-backward.

Size note: the repository name says 760M, but the model has 560,988,160 parameters. The name is kept so existing links keep working.

Conversation format

Conversations use nanochat's chat tokens:

<|bos|> <|user_start|> question <|user_end|> <|assistant_start|> reply <|assistant_end|> …

For this model every conversation was reversed except the leading <|bos|>. That reverses the order of turns, the tokens inside each turn, and the start/end markers. A one-turn exchange therefore becomes:

<|bos|> <|assistant_end|> reversed(reply) <|assistant_start|> <|user_end|> reversed(question) <|user_start|>

The training loss covered assistant tokens only, as in standard nanochat SFT. The model was trained to write assistant text right to left, conditioned on everything later in the conversation. Writing user turns, as in the reply → question example below, relies on what it learned in base pretraining plus the chat format. It works in practice but was not directly supervised.

Reverse tokens, not characters.

Usage

Requires a checkout of onanchat (tested at commit 0e99157), PyTorch and huggingface_hub. onanchat's BackwardChatEngine (nanochat/directional_chat_engine.py) wraps the same format for multi-turn use.

# run from the root of an onanchat checkout, or put it on PYTHONPATH
import json, sys, types, torch
sys.modules.setdefault("rustbpe", types.ModuleType("rustbpe"))  # only needed to *train* tokenizers
from huggingface_hub import snapshot_download
from nanochat.gpt import GPT, GPTConfig
from nanochat.tokenizer import RustBPETokenizer

def load(repo, device="cuda"):
    path = snapshot_download(repo, allow_patterns=["model.pt", "meta.json", "tokenizer.pkl", "token_bytes.pt"])
    meta = json.load(open(f"{path}/meta.json"))
    with torch.device("meta"):
        model = GPT(GPTConfig(**meta["model_config"]))
    model.to_empty(device=device)
    model.init_weights()  # builds the rotary tables; weights are overwritten below
    state = torch.load(f"{path}/model.pt", map_location=device)
    state = {k.removeprefix("_orig_mod."): v.float() if device == "cpu" else v for k, v in state.items()}
    model.load_state_dict(state, strict=True, assign=True)
    return model.eval(), RustBPETokenizer.from_directory(path), meta

@torch.no_grad()
def sample(model, ids, n, stop=None, temperature=0.8, top_k=50):
    x = torch.tensor([ids], device=next(model.parameters()).device)
    out = []
    for _ in range(n):
        with torch.autocast("cuda", dtype=torch.bfloat16, enabled=x.is_cuda):
            logits = model(x)[0, -1].float() / temperature
        v, _ = logits.topk(top_k)
        logits[logits < v[-1]] = -float("inf")
        t = torch.multinomial(logits.softmax(-1), 1)
        if t.item() == stop:
            break
        out.append(t.item())
        x = torch.cat([x, t[None]], 1)
    return out

model, tok, meta = load("raghavt/nanochat-760M-backward-sft")
bos = tok.get_bos_token_id()
s = {k: tok.encode_special(f"<|{k}|>") for k in ("user_start", "user_end", "assistant_start", "assistant_end")}

# 1. Reply -> the question that came before it
reply = "The capital of France is Paris."
prompt = [bos, s["assistant_end"]] + tok.encode(reply)[::-1] + [s["assistant_start"], s["user_end"]]
question = sample(model, prompt, 80, stop=s["user_start"])
print(tok.decode(question[::-1]))

# 2. A later user turn -> the assistant turn that came before it
later = "Thanks! And what about Germany?"
prompt = [bos, s["user_end"]] + tok.encode(later)[::-1] + [s["user_start"], s["assistant_end"]]
answer = sample(model, prompt, 80, stop=s["assistant_start"])
print(tok.decode(answer[::-1]))

One sample (yours will differ). For the reply "The capital of France is Paris.", the generated question was:

Your response should contain at least 3 sentences. In your response, the word "the" should appear at least 2 times.

What is the capital of France?

The constraint-style preamble comes from SmolTalk's instruction-following data.

The onanchat Colab notebook's chat() helper builds a plain-text User: … Assistant: prompt. That format doesn't match how this model was trained, so use the special tokens shown above instead.

Training

• Base pretraining: see nanochat-760M-backward (FineWeb-Edu, about 5B tokens, right to left).
• SFT mixture: onanchat/nanochat default, about 23K conversations:
  • ARC-Easy and ARC-Challenge train (3.4K)
  • GSM8K train (8K)
  • 10K SmolTalk conversations
  • 1K synthetic identity conversations
  • 600 spelling tasks
• SFT optimization: one epoch, 32 conversations per step, 700 steps, starting at 2% of the base learning rates. Every conversation was reversed as described above.

Evaluation

meta.json also records MMLU 26.0% and ARC-Easy 27.3%, both close to the 25% chance rate. They come from onanchat's chat evaluation, which presents questions left to right, so they test a format this model was never trained to read. They are not a measure of what it knows. For comparison, the matched forward SFT model scores about 32% and 40% on the same evaluation.

Intended use and limitations

• This is a research model for studying reverse-direction chat: which questions lead to a given answer, and what an assistant said before a later turn.
• It is not an assistant to deploy, and it has had no safety or preference training.
• It is small (561M parameters, about 5B pretraining tokens). Its outputs are often generic or wrong, and it inherits biases from FineWeb-Edu and the SFT data.
• Most bad outputs come from wrong direction handling: forgetting to reverse, reversing characters, putting markers in forward order, or not stopping at the right start marker.

Related models

• nanochat-760M-backward: the right-to-left base model.
• nanochat-760M-bidirectional: one base model for both directions.

Citation

@misc{nanochat-d20-backward-sft,
  author = {Raghav},
  title = {nanochat d20 backward SFT: a right-to-left nanochat chat model},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/raghavt/nanochat-760M-backward-sft}
}

Built on nanochat by Andrej Karpathy (MIT).