Downloads · 30 days
0
RWKV-x060-Japanese-11.2B
OpenMOSE/RWKV-x060-Japanese-11.2B
RWKV-x060-Japanese-11.2B is a machine learning model from OpenMOSE. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Access
Public
Updated May 1, 2024
Repo size
44.7 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.pth22.4 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Apr 28, 2024
- Updated
- May 1, 2024
- SHA
- 55cbcd2d
Model card
From the Hugging Face model README
RWKV-x060-Japanese-11.2B
RWKV Architecture "Finch" based 11.2B Parameters Model.
継続トレーニング中です。実験なので性能評価はしていません。
- "YORINOBU"
- Based on RWKV6-World v2.1 7b 53% Model, we have applied a layer expansion approach and tuned it as a 48-layer, 4096-dimensional model.
- I added 8 layers to the 40-layer model, froze layers 0 to 39, and continued pre-training layers 40 to 47, along with the Embedding and Head layers, using a Japanese corpus.
- Since it is an experimental approach, it may exhibit unpredictable behavior.
- RWKV6-World v2.1 7b 53% Modelをベースに、レイヤー拡張アプローチを適用し、48層4096次元モデルとしてチューニングしました。
- 40層モデルに8層を追加し、0から39レイヤーまでを凍結し、40から47、Emb、Head層を日本語コーパスで継続事前学習を行いました。
- 実験的アプローチなので、予測不可能な挙動をする可能性があります
Training
- using RWKV-LM-LISA Anarchy mode, Continuous Pre-traning
- https://github.com/OpenMOSE/RWKV-LM-LISA
- Single A6000 LISA 4layer training each step
2024 OpenMOSE
- License
- apache-2.0
- Languages
- ja
- Created
- Apr 28, 2024
- Updated
- May 1, 2024