Downloads · 30 days
6
4% of all-time downloads
Breeze-7B-PTT-Chat-v1_lora
Riyuechang/Breeze-7B-PTT-Chat-v1_lora
Breeze-7B-PTT-Chat-v1_lora is a text generation model from Riyuechang. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
Riyuechang/Breeze-7B-PTT-Chat-v1所使用的，未與主模型MediaTek-Research/Breeze-7B-Instruct-v10合併的lora模型
All-time downloads
171
Public
Repo size
89.5 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors89.5 MB · 100%
At a glance
- Task
- Text Generation
- Library
- peft
- License
- apache-2.0
- Access
- Public
- Created
- Sep 2, 2024
- Updated
- Sep 9, 2024
- SHA
- 40428d2c
Try a prompt
Base models
Model card
From the Hugging Face model README
簡介
Riyuechang/Breeze-7B-PTT-Chat-v1所使用的，未與主模型MediaTek-Research/Breeze-7B-Instruct-v1_0合併的lora模型
注意!!
此Lora模型有使用Dora技術，Dora能讓模型有更好的學習效率
代價就是會讓訓練和推理花費的時間大幅上升，尤其是推理的速度會非常慢
建議把此Lora模型跟主模型合併後在進行推理
設備
- Ubuntu 22.04.4 LTS
- NVIDIA GeForce RTX 3060 12G
Lora參數
r=8,
lora_alpha=32,
lora_dropout=0.1,
task_type="CAUSAL_LM",
target_modules="all-linear",
bias="none",
use_dora=True,
use_rslora=True
訓練參數
per_device_train_batch_size=28,
gradient_accumulation_steps=1,
num_train_epochs=3,
warmup_ratio=0.1,
learning_rate=2e-5,
bf16=True,
save_strategy="steps",
save_steps=500,
save_total_limit=10,
logging_steps=10,
output_dir=log_output,
optim="paged_adamw_8bit",
gradient_checkpointing=True
結果
- loss: 1.1035
- Task
- Text Generation
- Library
- peft
- License
- apache-2.0
- Created
- Sep 2, 2024
- Updated
- Sep 9, 2024