Downloads · 30 days
64
0% of all-time downloads
Pallas-0.5-LASER-0.2
Mihaiii/Pallas-0.5-LASER-0.2
Pallas-0.5-LASER-0.2 is a text generation model from Mihaiii. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
This model has a LASER intervention on Mihaiii/Pallas-0.5-LASER-0.1 .
All-time downloads
40.3K
Public
Parameters
34.4B
88.2 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors68.8 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- other
- Model type
- llama
- Access
- Public
- Created
- Jan 1, 2024
- Updated
- Jan 2, 2024
- SHA
- 9119f34f
Try a prompt
Base models
Model card
From the Hugging Face model README
This model has a LASER intervention on Mihaiii/Pallas-0.5-LASER-0.1 .
Configs used:
- lnum: 58
- lnames: attn (meaning: ["self_attn.k_proj.weight", "self_attn.q_proj.weight", "self_attn.v_proj.weight", "self_attn.o_proj.weight"])
- rate: 9.0
- dataset: bigbench (subset: causal_judgement)
- intervention type: rank-reduction
| Name | Validation acc (higher is better) | Validation logloss (lower is better) | Test acc (higher is better) | Test logloss (lower is better) |
|---|---|---|---|---|
| Pallas-0.5 | 55.263 | 1.650 | 60.526 | 1.463 |
| Pallas-0.5-LASER-0.1 | 55.263 | 1.639 | 61.184 | 1.451 |
| Pallas-0.5-LASER-0.2 | 55.263 | 1.646 | 61.184 | 1.458 |
| Pallas-0.5-LASER-0.3 | 55.263 | 1.575 | 61.842 | 1.382 |
| Pallas-0.5-LASER-0.4 | 55.263 | 1.525 | 61.842 | 1.326 |
| Pallas-0.5-LASER-0.5 | 55.263 | 1.484 | 61.842 | 1.297 |
| Pallas-0.5-LASER-0.6 | 55.263 | 1.455 | 61.184 | 1.283 |
In order to replicate on a single A100, you can use my branch (the original code will throw OOM for 34b models).
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- other
- Created
- Jan 1, 2024
- Updated
- Jan 2, 2024