Downloads · 30 days
6
18% of all-time downloads
hydra-decoder-2l-vicuna-7b-v1.3
shiqihe/hydra-decoder-2l-vicuna-7b-v1.3
hydra-decoder-2l-vicuna-7b-v1.3 is a machine learning model from shiqihe. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
--- We currently support inference in the single GPU and batch size 1 setting, which is consistent with Medusa and is also the most common setup for local model hosting.
All-time downloads
33
Public
Repo size
1.7 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt1.7 GB · 100%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Dec 13, 2023
- Updated
- Dec 13, 2023
- SHA
- 203f8511
Model card
From the Hugging Face model README
Hydra Decoder (Codebase)
We currently support inference in the single GPU and batch size 1 setting, which is consistent with Medusa and is also the most common setup for local model hosting.
Model Weights. We uploaded three model weights for users to try.
| Base Model | Description | Hugging Face Repo |
|---|---|---|
| Vicuna-7b | Medusa Model (Original) | FasterDecoding/medusa-vicuna-7b-v1.3 |
| Vicuna-7b | Medusa Model - 3 Head (Ours) | Rango2000/medusa-3h-vicuna-7b-v1.3 |
| Vicuna-7b | Hydra Model - 3 Head - 1 Decoding Layer (Ours) | shiqihe/hydra-decoder-1l-vicuna-7b-v1.3 |
| Vicuna-7b | Hydra Model - 3 Head - 2 Decoding Layer (Ours) | shiqihe/hydra-decoder-2l-vicuna-7b-v1.3 |
Inference. You can use the following command to launch a CLI interface:
# optional
CUDA_VISIBLE_DEVICES=0
# run cli
python -m medusa.inference.cli --model [path/repo of hydra decoder]
license: mit
- Library
- transformers
- Created
- Dec 13, 2023
- Updated
- Dec 13, 2023