Downloads · 30 days
2
3% of all-time downloads
philipperen55/Qwen2.5-14B-v3-3690-dataset52SFT_chat_3epochs
Qwen2.5-14B-v3-3690-dataset52SFT_chat_3epochs is a machine learning model from philipperen55. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
2
3% of all-time downloads
All-time downloads
70
Public
Repo size
3.3 GB
Likes
0
Public
Click a slice to open those files.
.bin551 MB · 49%
From the Hugging Face model README
axolotl version: 0.5.0
base_model: philipperen55/Qwen2.5-14B-style-MERGED-BF16-v3-3690
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true
load_in_8bit: false
load_in_4bit: false
datasets:
- path: philipperen55/dataset52SFT_chat # ou chemin local si tu ne l’as pas push sur HF
ds_type: json
data_files: dataset52SFT_chat.jsonl
type: messages
field_messages: messages
message_field_role: role
message_field_content: content
train_on_inputs: false
chat_template: chatml
train_on_inputs: false
add_eos_token: false
dataset_prepared_path: /workspace/prepared_data
val_set_size: 0.01
output_dir: /workspace/output
sequence_len: 2048
sample_packing: false
pad_to_sequence_len: false
group_by_length: true
special_tokens:
eos_token: "<|endoftext|>"
pad_token: "<|endoftext|>"
adapter: lora
lora_r: 32
lora_alpha: 64
lora_dropout: 0.05
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj
micro_batch_size: 2
gradient_accumulation_steps: 32 # 32/2 = 100802MiB / 143771MiB et 16/4 plante out of memory !!!
num_epochs: 3
learning_rate: 2e-5
lr_scheduler: cosine
warmup_ratio: 0.03
optimizer: adamw_torch_fused
weight_decay: 0.0
max_grad_norm: 1.0
bf16: true
fp16: false
tf32: true
flash_attention: true
sdp_attention: false
overrides_of_model_config:
attn_implementation: flash_attention_2 # forcer FA2 jusqu’au modèle
use_cache: false
gradient_checkpointing: false
# --- Reproductibilité ---
seed: 42
logging_steps: 5
save_steps: 50
eval_strategy: steps
eval_steps: 50
save_total_limit: 1
wandb_project: Qwen2.5-14B-v3-3690-dataset52SFT_04
hub_model_id: philipperen55/Qwen2.5-14B-v3-3690-dataset52SFT_chat_3epochs
push_to_hub: true
hub_strategy: every_save
</details><br>
This model is a fine-tuned version of philipperen55/Qwen2.5-14B-style-MERGED-BF16-v3-3690 on the None dataset. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| No log | 0.0129 | 1 | 1.2171 |
| 1.1498 | 0.6465 | 50 | 1.0827 |
| 1.0955 | 1.2929 | 100 | 1.0620 |
| 1.1058 | 1.9394 | 150 | 1.0588 |
| 1.0979 | 2.5859 | 200 | 1.0582 |