Downloads · 30 days
59
0% of all-time downloads
Pallas-0.5-LASER-0.1
Mihaiii/Pallas-0.5-LASER-0.1
Pallas-0.5-LASER-0.1 is a text generation model from Mihaiii. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
This is the first LASER intervention on Pallas-0.5 . More will follow using the previous results as base.
All-time downloads
47.7K
Public
Repo size
138 GB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.bin68.8 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- other
- Model type
- llama
- Access
- Public
- Created
- Dec 30, 2023
- Updated
- Jan 2, 2024
- SHA
- bc07f908
Try a prompt
Base models
Model card
From the Hugging Face model README
This is the first LASER intervention on Pallas-0.5 . More will follow using the previous results as base.
Configs used:
- lnum: 59
- lnames: attn (meaning: ["self_attn.k_proj.weight", "self_attn.q_proj.weight", "self_attn.v_proj.weight", "self_attn.o_proj.weight"])
- rate: 6.0
- dataset: bigbench (subset: causal_judgement)
- intervention type: rank-reduction
| Name | Validation acc (higher is better) | Validation logloss (lower is better) | Test acc (higher is better) | Test logloss (lower is better) |
|---|---|---|---|---|
| Pallas-0.5 | 55.263 | 1.650 | 60.526 | 1.463 |
| Pallas-0.5-LASER-0.1 | 55.263 | 1.639 | 61.184 | 1.451 |
| Pallas-0.5-LASER-0.2 | 55.263 | 1.646 | 61.184 | 1.458 |
| Pallas-0.5-LASER-0.3 | 55.263 | 1.575 | 61.842 | 1.382 |
| Pallas-0.5-LASER-0.4 | 55.263 | 1.525 | 61.842 | 1.326 |
| Pallas-0.5-LASER-0.5 | 55.263 | 1.484 | 61.842 | 1.297 |
| Pallas-0.5-LASER-0.6 | 55.263 | 1.455 | 61.184 | 1.283 |
In order to replicate on a single A100, you can use my branch (the original code will throw OOM for 34b models).
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- other
- Created
- Dec 30, 2023
- Updated
- Jan 2, 2024