Downloads · 30 days
45
1% of all-time downloads
eastlondoner/nanochat-wasm-fused-preview-01
nanochat-wasm-fused-preview-01 is a text generation model from eastlondoner. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
A coprocessor-augmented language model that embeds a frozen WASM bytecode interpreter transformer inside a 4.9B-parameter language model (NanochatGPT d34), connected via trained cross-attention bridges. The LM and cop…
Downloads · 30 days
45
1% of all-time downloads
All-time downloads
4.5K
Public
Repo size
136 GB
Likes
2
Public
Click a slice to open those files.
.safetensors136 GB · 100%
From the Hugging Face model README
A coprocessor-augmented language model that embeds a frozen WASM bytecode interpreter transformer inside a 4.9B-parameter language model (NanochatGPT d34), connected via trained cross-attention bridges. The LM and coprocessor execute in parallel within a single forward pass; from LM layer 10 onward, cross-attention lets the LM read the coprocessor's completed state. The LM generates text and WASM instructions; the coprocessor executes them deterministically; results flow back into the LM — enabling the model to think in computation.
| Component | Details |
|---|---|
| Language Model (trained) | NanochatGPT d34: d_model=2176, 34 layers, 17 heads, ~4.9B params (all trained) |
| WASM Coprocessor (frozen) | WASM Interpreter Transformer: d_model=102, 23 layers, variable heads, ~840K params |
| Bridge (trained) | 24 cross-attention layers (L10–L33, 4 heads each), WasmTokenEmbedding (260→2176), logit bias |
| Total parameters | ~4.9B trained LM + ~840K frozen coprocessor ≈ 4.9B total |
| Vocab | 65536 text tokens (tiktoken BPE) + 523 WASM tokens (opcodes + operand bytes + feedback) |
The coprocessor is a hand-compiled transformer that executes WASM bytecode via real matrix multiplications. It was not trained — every weight was set by a compiler. It supports arithmetic, comparisons, memory, local variables, filesystem I/O, and loops with conditional branching.
| Parameter | Value |
|---|---|
| GPU | NVIDIA B200 (192GB HBM3e) |
| Optimizer | MuonAdamW (Muon for matrix params, AdamW for scalars) |
| Precision | FP8 (Blackwell native) with bf16 master weights |
| Phase | Supervised Fine-Tuning (SFT) |
| Data | WASM programs + text conversations (SmolTalk + MMLU) |
| Text ratio | 30% pure text, 70% WASM conversations |
| Gradient checkpointing | Enabled |
| Base model | karpathy/nanochat-d34 |
epoch_0/ — End of SFT epoch 0epoch_1/ — End of SFT epoch 1epoch_4_batch_6100/ — Latest SFT checkpoint (epoch 4, batch 6100)All checkpoints are loadable via AutoModelForCausalLM.from_pretrained with trust_remote_code=True.
Note: This is an early preview. WASM coprocessor triggering reliability varies across checkpoints and prompts. Conversational (non-math) prompts work reliably. Math/WASM execution may not trigger consistently depending on the checkpoint and prompt phrasing.
Important: This model was trained with integers in the prompt encoded as 4-byte WASM tokens (not as regular text). You must byte-encode numbers in your input and decode WASM output values back to integers for display.
import re
import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
REPO = "eastlondoner/nanochat-wasm-fused-preview-01"
WASM_OFFSET = 65536
BYTE_OFFSET = 264
# ── 1. Load model + tokenizer ───────────────────────────────────
config = AutoConfig.from_pretrained(REPO, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
REPO, config=config, trust_remote_code=True,
torch_dtype=torch.bfloat16,
subfolder="epoch_4_batch_6100",
)
model = model.to("cuda").eval()
tok = AutoTokenizer.from_pretrained(REPO, trust_remote_code=True, use_fast=False)
# ── 2. Encode prompt (byte-encode integers) ─────────────────────
# encode_chat() automatically byte-encodes integers with i32.const prefix
prompt_ids = tok.encode_chat("What is 15 + 27?")
# ── 3. Generate ─────────────────────────────────────────────────
eos_id = tok._special_token_ids["<|assistant_end|>"]
generated, wasm_outputs, trace = model.generate_chat(
prompt_ids,
max_new_tokens=1024,
temperature=0,
return_outputs=True,
eos_token_id=eos_id,
)
# ── 4. Decode response ──────────────────────────────────────────
# Filter text-only tokens for human-readable text
text_tokens = [t for t in generated[len(prompt_ids):] if 0 < t < WASM_OFFSET]
response = tok.decode(text_tokens)
print(f"Text: {response}")
# The coprocessor's computed results are in wasm_outputs (list of ints)
if wasm_outputs:
print(f"Answer: {wasm_outputs[-1]}") # → 42
Inputs: All integers in the user prompt must be byte-encoded before
tokenization. The tokenizer's encode_chat() method handles this
automatically — it finds integer literals via regex and replaces each with
an i32.const opcode token (65536) followed by 4 big-endian byte tokens
in the WASM token range (65800–66055).
For example, "What is 15 + 27?" becomes:
text("What is ") + [i32.const, 0x00, 0x00, 0x00, 0x0F] + text(" + ") + [i32.const, 0x00, 0x00, 0x00, 0x1B] + text("?")
where i32.const is token 65536 and each byte b maps to token 65536 + 264 + b.
Outputs: The model generates a mix of text tokens (< 65536) and WASM
tokens (≥ 65536). The WASM coprocessor executes the WASM trace and returns
integer results via wasm_outputs. These are plain Python integers ready
for display. The text portion of the response (e.g., "The answer is") can be
decoded normally; the actual numeric answer comes from wasm_outputs.
For prompts without numbers (e.g., "Hello, how are you?"), standard tokenization works fine — no byte encoding needed:
prompt_ids = tok.encode_chat("What is the capital of France?")
generated, _, _ = model.generate_chat(
prompt_ids, max_new_tokens=1024, temperature=0.8,
eos_token_id=eos_id,
)
text_tokens = [t for t in generated[len(prompt_ids):] if 0 < t < WASM_OFFSET]
print(tok.decode(text_tokens))
The model uses an extended vocabulary where tokens ≥ 65536 are WASM tokens:
| Token Range | Meaning |
|---|---|
| 0–65535 | Standard text tokens (tiktoken BPE) |
| 65536–65799 | WASM opcodes (e.g., i32.const=65536, i32.add=65537, output=65776, halt=65791) |
| 65800–66055 | Operand/value bytes: token = 65536 + 264 + byte_value (0–255) |
i32.const (0x00), i32.add (0x01), output (0xF0), halt (0xFF), etc.wasm_offset + 264 + byte_valueREPL_RESULT (261), BRANCH_TAKEN (262), BRANCH_NOT_TAKEN (263)Input numbers are byte-encoded in the question; WASM output values appear as 4-byte tokens after the output opcode:
[<|bos|>] [<|user_start|>] [text + byte-encoded integers] [<|user_end|>] [<|assistant_start|>]
[WASM inst] [feedback] [WASM inst] [feedback] ...
[output] [4 result bytes]
[halt]
[answer text tokens]
[<|assistant_end|>]
The frozen WASM coprocessor is a hand-compiled 23-layer transformer (8 functional layers + 15 FFN-only padding layers):
Supports 25 WASM operations including arithmetic, comparisons, memory, locals, filesystem I/O, and loops with conditional branching. 115/115 compliance tests pass at 100% accuracy.
i32.const, i32.add, i32.sub, i32.mul, i32.and, i32.or
i32.eq, i32.ne, i32.lt_s, i32.gt_s, i32.le_s, i32.ge_s
i32.load, i32.store, local.get, local.set, local.tee
fd_open, fd_read, fd_write, fd_close (4 fds, 32 bytes/file)
loop, end_loop, br_if (up to 256 iterations, nested loops)
output, halt
| File | Description |
|---|---|
config.json | Model configuration with auto_map |
configuration_nanochat.py | PretrainedConfig subclass |
modeling_nanochat.py | PreTrainedModel wrapping ComposedModel |
composed_model.py | Coprocessor-augmented ComposedModel with generation |
gpt.py | NanochatGPT language model (d34 architecture) |
wasm_transformer.py | Frozen WASM Interpreter Transformer |
wasm_ops.py | WASM opcodes, constants, instruction encoding |
compile_weights.py | Coprocessor weight compiler |
tokenization_nanochat.py | Custom AutoTokenizer-compatible tokenizer class |
handler.py | HF Inference API handler |
tokenizer.pkl | Tiktoken BPE tokenizer (65536 tokens) |
model.safetensors | Model weights in safetensors format |
MIT