Downloads · 30 days
0
mappingadapter_roberta_mistral
sade-adrien/mappingadapter_roberta_mistral
mappingadapter_roberta_mistral is a machine learning model from sade-adrien. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
MappingAdapter exact structure available in representationmapping.py
Access
Public
Updated Nov 29, 2023
Repo size
126 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pth126 MB · 99%
At a glance
- Access
- Public
- Created
- Nov 29, 2023
- Updated
- Nov 29, 2023
- SHA
- 771f8752
Model card
From the Hugging Face model README
MappingAdapter exact structure available in representation_mapping.py
Mapping "sentence-transformers/stsb-roberta-large"'s hidden representation to "mistralai/Mistral-7B-Instruct-v0.1"'s.
Training:
-
Steps: 114k
-
Gradient accumulation: 2
-
Batch size: 64
-
Warm-up steps: 100
-
Learning Rate: 3e-5 with linear scheduling
-
Eval steps: %8000
-
Training hours: ~98h
-
Eval hours: ~10h
-
Gradient updates: 57k
-
Train examples: 7.3M
-
Eval examples: 106k
-
Adapter: Decoder_dim (4096) → 4096 → LeakyRelu(.1) → Encoder_dim (1024)
- Created
- Nov 29, 2023
- Updated
- Nov 29, 2023