rwkv-hils-3-m10

rwkv-hils-3-m10

A 44.6M-parameter research language model. It interleaves RWKV-7 selective state-space layers (linear-time recurrence; Peng et al., RWKV-7) with HiLS layers — Hierarchical Landmark Sparse attention, which scores chunk summaries with an entropy-calibrated, low-rank-calibrated query, attends the Top-K chunks in full, and always attends a local window (Hu et al., Hierarchical Sparse Attention Done Right). Text-only, byte-level BPE, 4096-token vocabulary, 8192-token position budget.

Deviation from that paper: chunk summaries here use a shared learnable query instead of per-chunk landmark tokens (the paper's "w/o lmk, shared q_c" variant), because landmark tokens need extra tokens interleaved through every layer and that interacts badly with the recurrent blocks in this hybrid.

On the name. m10 is a tier label from this project's own ladder, keyed to depth and width rather than to parameters — this m10 is 44.6M parameters, not 10M. The ladder exists because the original tier labels were optimistic about parameter counts; the dimensions below are what is authoritative.

This is a research artifact, not a product. It has no instruction tuning and no safety tuning, and a couple of paragraphs of freeform generation are enough to find its failure modes — they are listed below rather than left for you to discover.

Files

Architecture

Two design details are worth knowing before you read the code:

• The vocabulary carries a synthetic task. Ids 0–255 are opening and closing bracket types for a Dyck-language task that shares the sequence with text, which is why a 4096-token text vocabulary sits inside a 4096 total. The model was trained on interleaved bracket and text windows.
• HiLS layers are not attention over the raw sequence. They attend over chunk summaries and a local window, which is what keeps the cost near-linear while still allowing exact retrieval inside the window. q_cal_rank is the low-rank calibration applied to the query before scoring summaries.

Measured behaviour

All numbers below are from the released checkpoint at step 10,000, measured locally (Arc A770, fp32) with the scripts in the repository.

The last three rows are the honest headline: retrieval in this model is specific to the retrieval task it was trained on. It solves the single-needle variant perfectly and does nothing at all on variants that never appeared in its data. That is a statement about the training data, not about the architecture — a newer, deliberately broader needle generator exists in the repository and is not what this checkpoint was trained with.

Known failure modes

• Repetition. Long freeform generations collapse into repeating phrases or into short list-like lines. Sampling-side penalties (DRY, a targeted continuation penalty, a frequency penalty) were implemented and measured: they move which attractor it lands in, not whether it lands in one. At this scale it is a property of the model, not of the sampler.
• No instruction following. Base model. Prompts are continuations, not requests.
• Short effective range for prose. Retrieval holds at 4K for the trained variant; general coherence over long documents was not measured here and should not be assumed.
• Incorrect/conflicting chat tokens The chat model was finetuned using tokens that were already present in its output due to an oversight in planning.
• Chat coherence is very weak The chat model is able to sometimes stay on topic, but factuality or general coherence is not effective.

Running it

Python — chat.py at the root of this repository is the entry point (interactive REPL and one-shot mode):

uv run chat.py Vibelet/rwkv-hils-3-m10 --prompt "The sea was" --max_new 200
uv run chat.py /path/to/local/dir --device cpu     # or xpu / cuda

# or install it and use the console script
uv pip install . && rwkv-hils-3 Vibelet/rwkv-hils-3-m10 --prompt "The sea was"

A repository id, a local directory holding model.safetensors + config.json, a .pt training checkpoint and a directory of checkpoints are all accepted. Runs on CUDA, Intel XPU and CPU; fp32 inference, ~180 MB resident. The fast kernels need triton, but a host without it still works: the model falls back to the reference recurrence, which produces identical tokens (verified by running with the triton import blocked) and only changes the speed.

Browser — the demo in the repository runs the model entirely client-side, with WebGPU when available and a CPU fallback:

uv run demo/web/serve.py     # serves with no-store, port 8123
# open http://127.0.0.1:8123/demo/web/index.html

?backend=cpu forces the reference path. The page can also point at any hosted copy of the weights with ?model=<base-url>, which is how the public demo site loads the fp16 export from this repository.

From the Hub directly — download the files and use web/ as the base URL; the browser runtime fetches one manifest and one flat blob, and views every tensor in place rather than copying them.

The fp16 export was checked against the fp32 weights in the browser runtime before shipping: max |Δlogit| 9.1e-3 over 24 decode steps, with the greedy token sequence identical. That is a constant quantisation offset and not a backend disagreement, so the demo's CPU-versus-WebGPU comparison still holds within either export.

Training

Training mixes three kinds of window: synthetic bracket sequences (curriculum over depth), real text, and needle-in-a-haystack examples rendered as text. The mix shifts over the run — bracket-heavy early, text-dominant later, with the needle task present from the point it becomes measurable.

Data. Public corpora, drawn through a cached token pipeline:

Each listed source was drawn to between 82.5% and 86.7% of its target epochs by final step, so the run consumed these corpora at close to one pass. The pipeline also renders synthetic bracket and retrieval windows. This release ships inference and the browser demo only; the training pipeline is not part of it, and the dimensions, step count, objective and wall clock above are what the released weights were produced with.

Caveat on the text-loss figure. Because the corpora were consumed at close to one epoch, the validation windows — drawn with a different seed from the same sources — are not strictly held out. At 44.6M parameters and under one epoch of exposure, memorisation of any specific window is unlikely, but the number should be read as an in-corpus loss rather than a generalisation measurement. A loss on text the run never saw is not reported here.

Content note: the corpus above is public web text and public-domain literary fiction. The model can produce text that is unsuitable for some audiences and has no filtering of any kind.

Chat conditioning

Data

The single data source for the fine tuning is the wildchat dataset and drew upon 260 million tokens from the dataset. Toxic discussions and non-english language text were filtered out.

Limitations and intended use

Intended for experimentation with hybrid recurrent/attention architectures, sparse-attention retrieval behaviour, and small-model training dynamics. It is not suitable for production, for factual applications, or for any use where outputs need to be correct, safe or attributable. Do not treat its outputs as information about the world.

The chat version is very confident but also almost always very wrong.

Licence

MIT. The training corpora keep their own licences, listed above.