rwkv7-168m-pile-hf

RWKV-7 "Goose" 168M (Pile) — converted for the in-tree transformers implementation

This is not a new model. It is a format conversion of BlinkDL/rwkv-7-pile → RWKV-x070-Pile-168M-20241120-ctx4096.pth into a transformers directory, so that the rwkv7 implementation can load it with from_pretrained. All credit for the weights belongs to BlinkDL / the RWKV project; they are redistributed here under the Apache-2.0 licence they were released under.

Why the Pile model rather than a World one

This is the checkpoint the rwkv7 docstring examples point at, so it has to be loadable by copy-and-paste — including its tokenizer, with no trust_remote_code.

The World models use RWKV's own vocabulary, whose tokenizer is greedy longest-match over raw bytes with no pre-tokenisation. There is no tokenizer.json for it, and the natural tokenizers encoding of that algorithm has no bound on its match window, which makes it quadratic in the input length. The Pile models were trained with the "20B tokenizer" (EleutherAI/gpt-neox-20b) instead, which is an ordinary fast tokenizer that ships as a single file.

RWKV-4 in transformers documents RWKV/rwkv-4-169m-pile for the same reason.

For actual use, prefer a World checkpoint — they are trained on far more data and are multilingual. This one exists so the documented example runs.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Hakureirm/rwkv7-168m-pile-hf")
model = AutoModelForCausalLM.from_pretrained("Hakureirm/rwkv7-168m-pile-hf")

inputs = tokenizer("The capital of France is", return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=24)[0]))
# The capital of France is the city of Paris.

At 168M parameters and greedy decoding it will loop after a sentence or two. That is the model, not the port.

What the conversion does

A prefix rename and nothing else. All 399 tensors in the source .pth are bit-identical here — verified tensor by tensor, not by spot check. The converted directory holds three tensors the source does not: layer 0's value-residual LoRA (v0/v1/v2), written as zeros so the state dict is rectangular. Layer 0 produces v_first rather than mixing towards it, so it never reads them.

The tokenizer files are the unmodified GPT-NeoX-20B ones, also Apache-2.0.

The embedding is padded past the tokenizer's vocabulary, which is how the source was trained. The extra rows are untrained; in practice they do not win the argmax, and the generation above was checked for ids the tokenizer cannot decode.