Downloads · 30 days
14
24% of all-time downloads
furkancekic/turkish-finance-qwen7b-v2-lora
turkish-finance-qwen7b-v2-lora is a text generation model from furkancekic. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
14
24% of all-time downloads
All-time downloads
59
Public
Repo size
2.9 GB
Likes
0
Public
Click a slice to open those files.
.pt1.9 GB · 59%
From the Hugging Face model README
axolotl version: 0.13.0.dev0
base_model: Qwen/Qwen2.5-7B-Instruct
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true
load_in_8bit: false
load_in_4bit: false
datasets:
- path: furkancekic/turkish-finance-instructions-v2
type: alpaca
chat_template: chatml
split: train
dataset_prepared_path: last_run_prepared
val_set_size: 0.02
output_dir: ./outputs/qwen7b-turkish-finance-v2
adapter: lora
lora_r: 32
lora_alpha: 64
lora_dropout: 0.05
lora_target_linear: true
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj
sequence_len: 1024
sample_packing: true
pad_to_sequence_len: true
micro_batch_size: 4
gradient_accumulation_steps: 4
num_epochs: 2
learning_rate: 1e-4
lr_scheduler: cosine
warmup_ratio: 0.05
weight_decay: 0.01
optimizer: adamw_torch
gradient_checkpointing: true
bf16: auto
tf32: true
logging_steps: 10
save_strategy: steps
save_steps: 200
save_total_limit: 3
eval_steps: 200
wandb_project: turkish-finance-slm-v2
wandb_run_id:
wandb_log_model:
train_on_inputs: false
special_tokens:
pad_token: "<|endoftext|>"
</details><br>
This model is a fine-tuned version of Qwen/Qwen2.5-7B-Instruct on the furkancekic/turkish-finance-instructions-v2 dataset. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss | Active (gib) | Allocated (gib) | Reserved (gib) |
|---|---|---|---|---|---|---|
| No log | 0 | 0 | 6.1174 | 21.3 | 21.3 | 21.42 |
| 1.1934 | 0.1741 | 200 | 1.2171 | 21.94 | 21.94 | 26.14 |
| 1.0438 | 0.3482 | 400 | 1.0703 | 21.94 | 21.94 | 25.91 |
| 0.979 | 0.5223 | 600 | 0.9918 | 21.94 | 21.94 | 25.91 |
| 0.9547 | 0.6964 | 800 | 0.9400 | 21.94 | 21.94 | 25.91 |
| 0.9103 | 0.8705 | 1000 | 0.9004 | 21.94 | 21.94 | 25.91 |
| 0.8156 | 1.0444 | 1200 | 0.8716 | 21.94 | 21.94 | 25.91 |
| 0.799 | 1.2185 | 1400 | 0.8502 | 21.94 | 21.94 | 25.91 |
| 0.7954 | 1.3926 | 1600 | 0.8339 | 21.94 | 21.94 | 25.91 |
| 0.7699 | 1.5667 | 1800 | 0.8228 | 21.94 | 21.94 | 25.91 |
| 0.7713 | 1.7408 | 2000 | 0.8165 | 21.94 | 21.94 | 25.91 |
| 0.7638 | 1.9149 | 2200 | 0.8145 | 21.94 | 21.94 | 25.91 |