Downloads · 30 days
0
0% of all-time downloads
philipperen55/Qwen2.5-14B-style-SFT-v3
Qwen2.5-14B-style-SFT-v3 is a machine learning model from philipperen55. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
0
0% of all-time downloads
All-time downloads
10
Public
Repo size
5 GB
Likes
0
Public
Click a slice to open those files.
.bin551 MB · 49%
From the Hugging Face model README
axolotl version: 0.5.0
# =============================================================================
# PHASE 2 V3 : SFT Réécriture - FORMAT COMPLETION (sans template Alpaca)
# =============================================================================
base_model: philipperen55/Qwen2.5-14B-style-MERGED-v2
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true
load_in_8bit: false
load_in_4bit: false
# =============================================================================
# DATASET : Format completion pré-formaté (pas de template ajouté par Axolotl)
# =============================================================================
datasets:
- path: philipperen55/dataset40FT
data_files: dataset40FT.jsonl
type: completion
field: text
dataset_prepared_path: /workspace/prepared_data
val_set_size: 0.05
output_dir: /workspace/output
# =============================================================================
# SÉQUENCE - SANS PACKING
# =============================================================================
sequence_len: 2048
sample_packing: false
pad_to_sequence_len: true
# =============================================================================
# TOKEN EOS EXPLICITE
# =============================================================================
special_tokens:
eos_token: "<|endoftext|>"
# =============================================================================
# LoRA
# =============================================================================
adapter: lora
lora_r: 32
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj
# =============================================================================
# TRAINING - 33K PAIRES, 2 EPOCHS, LR 1.5e-5
# =============================================================================
gradient_accumulation_steps: 2
micro_batch_size: 8
num_epochs: 2
learning_rate: 1.5e-5
lr_scheduler: cosine
warmup_ratio: 0.05
optimizer: adamw_torch
# =============================================================================
# PRÉCISION
# =============================================================================
bf16: true
fp16: false
tf32: true
# =============================================================================
# OPTIMISATIONS
# =============================================================================
flash_attention: false
gradient_checkpointing: true
# =============================================================================
# LOGGING ET SAUVEGARDES
# =============================================================================
logging_steps: 50
save_steps: 500
eval_steps: 500
save_total_limit: 8
# =============================================================================
# WEIGHTS & BIASES
# =============================================================================
wandb_project: Qwen2.5-14B-SFT-phase2-v3
# =============================================================================
# HUGGING FACE HUB
# =============================================================================
hub_model_id: philipperen55/Qwen2.5-14B-style-SFT-v3
push_to_hub: true
hub_strategy: every_save
</details><br>
This model is a fine-tuned version of philipperen55/Qwen2.5-14B-style-MERGED-v2 on the None dataset. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| No log | 0.0005 | 1 | 1.6796 |
| 1.4216 | 0.2557 | 500 | 1.4063 |
| 1.3818 | 0.5114 | 1000 | 1.3881 |
| 1.3878 | 0.7671 | 1500 | 1.3792 |
| 1.3816 | 1.0228 | 2000 | 1.3741 |
| 1.3631 | 1.2784 | 2500 | 1.3711 |
| 1.3541 | 1.5341 | 3000 | 1.3693 |
| 1.3653 | 1.7898 | 3500 | 1.3685 |