Downloads · 30 days
69
13% of all-time downloads
smol_llama-220M-GQA-32k-linear
Doctor-Shotgun/smol_llama-220M-GQA-32k-linear
smol_llama-220M-GQA-32k-linear is a text generation model from Doctor-Shotgun. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Experimental model meant to serve as a long-context speculative decoding model.
All-time downloads
545
Public
Repo size
1.3 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin435 MB · 99%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- llama
- Access
- Public
- Created
- Dec 24, 2023
- Updated
- Jan 10, 2024
- SHA
- 03be3e3e
Try a prompt
Model card
From the Hugging Face model README
smol_llama-220M-GQA-32k-linear
Experimental model meant to serve as a long-context speculative decoding model.
Created using BEE-spoke-data/smol_llama-220M-GQA and further pretraining at 32768 context length on togethercomputer/RedPajama-Data-1T-Sample.
This variant uses the linear rope scaling method for context extension.
Wikitext Perplexity (64 rows) as evaluated by exllamav2:
Base Model
2048: 20.2193
4096: 102.6928
8192: 235.5210
16384: 390.7198
32768: 515.8053
32k - Linear Rope Scale 16.0
2048: 25.7148
4096: 23.4461
8192: 22.3326
16384: 21.6744
32768: 21.4317
32k - Rope Theta 1000000.0
2048: 20.2158
4096: 18.3868
8192: 17.5976
16384: 17.1462
32768: 16.6989
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- apache-2.0
- Languages
- en
- Created
- Dec 24, 2023
- Updated
- Jan 10, 2024