Downloads · 30 days
5
7% of all-time downloads
arlowgpt-dummy-weights
yuchenxie/arlowgpt-dummy-weights
arlowgpt-dummy-weights is a text generation model from yuchenxie. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
- GQA - Silu - Flash Attention VarLen + manual QKV proj - cross attention (untrained, there for easy vision encoder incorporation) - model is decoder only, however, cross attention weights is there. - Custom RoPE - an…
All-time downloads
70
Public
Parameters
368M
736 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors736 MB · 99%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- arlow
- Access
- Public
- Created
- Apr 16, 2025
- Updated
- Apr 18, 2025
- SHA
- 5bd805f8
Try a prompt
Datasets
Model card
From the Hugging Face model README
DUMMY WEIGHTS ONLY !! NOT REAL MODEL !!
Specifications (Will be carried out to the main fully released model)
Architecture: arlow (Not supported by transformers right now, but beta version is out -> here)
Exact location: here
Will feature:
- GQA
- Silu
- Flash Attention VarLen + manual QKV proj
- cross attention (untrained, there for easy vision encoder incorporation)
- model is decoder only, however, cross attention weights is there.
- Custom RoPE
- and more!
Arlow architecture isn't officially supported by transformers but my implementation will be there if you want to try it.
- Task
- Text Generation
- Library
- transformers
- Type
- arlow
- License
- apache-2.0
- Languages
- en
- Created
- Apr 16, 2025
- Updated
- Apr 18, 2025