Downloads · 30 days
0
RWKV7-2.9B-midtrain50-docs-lora
Lambent/RWKV7-2.9B-midtrain50-docs-lora
RWKV7-2.9B-midtrain50-docs-lora is a machine learning model from Lambent. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Non-HF data sources: github.com/suttacentral/bilara-data, github.com/cbeta-org/xml-p5, github.com/OpenITI/RELEASE Also a private dataset based on personal Discord logs for indefinite context dialogue training.
Access
Public
Updated Aug 23, 2026
Repo size
755 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors1.5 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Aug 23, 2026
- Updated
- Aug 23, 2026
- SHA
- 768bf4f1
Base models
Model card
From the Hugging Face model README
Non-HF data sources: github.com/suttacentral/bilara-data, github.com/cbeta-org/xml-p5, github.com/OpenITI/RELEASE Also a private dataset based on personal Discord logs for indefinite context dialogue training.
Long documents with varying forms of long-range dependencies for the objective, replay data, and a dash of extra math and code thrown in.
Rank 128 LoRA adapter for long context training, 50.0M tokens, 180 steps, whole-document training with 16k gradient windows (docs ≤256k).
5e-5 learning rate, WSD schedule.
Flash attention path utilized, consistency with the standard modeling paths cross-checked.
Targets and removes pollution penalty to prediction from long state, tested at 56k/196KB. May also help with handling token-hungry languages and less continuous languages.
No particular tax on English harness benchmarks, at this size; strong reduction of state pollution at long contexts.
- License
- apache-2.0
- Created
- Aug 23, 2026
- Updated
- Aug 23, 2026