Downloads · 30 days
0
american-stories-tokenizer
alanamarzoev/american-stories-tokenizer
american-stories-tokenizer is a machine learning model from alanamarzoev. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
BPE tokenizer trained on the American Stories dataset for use with nanochat models.
Access
Public
Updated Jan 28, 2026
Repo size
264 KB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt264 KB · 99%
At a glance
- License
- mit
- Access
- Public
- Created
- Jan 28, 2026
- Updated
- Jan 28, 2026
- SHA
- 1730c93a
Model card
From the Hugging Face model README
American Stories Tokenizer
BPE tokenizer trained on the American Stories dataset for use with nanochat models.
Usage
import torch
from huggingface_hub import hf_hub_download
# Download
tokenizer_path = hf_hub_download(repo_id="alanamarzoev/american-stories-tokenizer", filename="token_bytes.pt")
# Load (requires nanochat)
from nanochat.tokenizer import Tokenizer
tokenizer = Tokenizer.from_file(tokenizer_path)
# Encode/decode
tokens = tokenizer.encode("Hello world")
text = tokenizer.decode(tokens)
- License
- mit
- Created
- Jan 28, 2026
- Updated
- Jan 28, 2026