Downloads · 30 days
0
urls-250m
ks46/urls-250m
urls-250m is a machine learning model from ks46. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
246,087,680 parameters: 12 layers × 1280 wide, 20 heads, SwiGLU MLP, tied 8,192-entry byte-level BPE vocabulary (structural pre-split, hosts stored TLD-first), learned absolute positions reset at every <eos, so each U…
Access
Public
Updated Sep 6, 2026
Repo size
71 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.pt65 GB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Sep 6, 2026
- Updated
- Sep 6, 2026
- SHA
- f4630777
Datasets
Model card
From the Hugging Face model README
urls-250m — a URL language model for lossless compression
246,087,680 parameters: 12 layers × 1280 wide, 20 heads, SwiGLU MLP,
tied 8,192-entry byte-level BPE vocabulary (structural pre-split, hosts stored TLD-first),
learned absolute positions reset at every <eos>, so each URL is its own context.
Shipped as a 124.8 MiB int4-g64 artifact (.nurl v3: 34 bytes per 64 weights, f16 scales)
that the browser codec streams with exact integer dot products, making encode and decode bit-identical.
Objective: bits per character on held-out URLs (the compression ratio), decoded on the client in under a second.
Training
| data | ks46/urls-tokenized shards 0–39, 19.9B tokens ≈ 561M URLs, one pass, seeded order |
| optimizer | AdamW β(0.9, 0.95), weight decay 0.01, peak LR 0.0006, warmup 1000, linear decay over the last 10% |
| batch | 64 windows × 512 tokens × 8 GPUs = 262,144 tokens per iteration, 76,000 iterations |
| tail | 7,600 quantisation-aware iterations on the exact int4-g64 grid at LR 6e-05 |
| hardware | 8× RTX 5090 (PCIe, no NVLink), bf16 autocast, ~680k tokens/s |
| code | linklet training/ at git unknown |
Checkpoints
checkpoints/ckpt-i<iter>.pt are immutable snapshots every 4,000 iterations (1.05B tokens), stable-phase
weights with optimizer state, loadable by training/train.py (run.init=) for decay or fine-tuning.
ckpt-fp.pt is the end of the full-precision phase; ckpt.pt is the final model after the QAT tail.
Eval is bits/char on a fixed 2,000-URL held-out slice (shard 512, even-stride sample).
| file | tokens | eval bits/char |
|---|---|---|
| ckpt-i004000.pt | 1.05B | nan |
| ckpt-i020000.pt | 5.24B | nan |
| ckpt-i040000.pt | 10.49B | nan |
| ckpt-i060000.pt | 15.73B | nan |
| ckpt-i068000.pt | 17.83B | nan |
| ckpt-i072000.pt | 18.87B | nan |
| ckpt-i080000.pt | 20.97B | nan |
Final evaluation
fp32: 1.2629 bits/char over 99926 URLs (74 skipped)
kernel_sim(int4): 1.26372 bits/char
kernel: 1.2952 bits/char (rust/nurlcheck) over 500 URLs, artifact 124.8 MiB
model/linklet-target-base.nurl is the packed artifact; model/tokenizer/ the tokenizer.
Every number above is copied from manifest.json, the run's provenance record.
- License
- mit
- Languages
- en
- Created
- Sep 6, 2026
- Updated
- Sep 6, 2026