dolphin-qwen2-72b-lainbear

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>

<details><summary>See axolotl config</summary>

axolotl version: 0.13.0.dev0

# Axolotl config for finetuning Dolphin 2.9.2 Qwen2-72B
# Optimized for QLoRA on single H100 SXM 80GB on Vast.ai

base_model: cognitivecomputations/dolphin-2.9.2-qwen2-72b
# Or use local path if you have it downloaded:
# base_model: ./dolphin-2.9.2-qwen2-72b

# Model loading configuration
trust_remote_code: true
load_in_8bit: false
load_in_4bit: true  # QLoRA requires 4-bit quantization

# Dataset configuration - Discord DMs in chat format
chat_template: chatml  # Dolphin uses ChatML format
datasets:
  - path: discord_training_data.jsonl  # Your Discord DM training data
    type: chat_template
    field_messages: conversations  # Field containing the conversation list
    message_property_mappings:
      role: from
      content: value

dataset_prepared_path:
val_set_size: 0.05
output_dir: ./outputs/dolphin-qwen2-72b-finetune

# Sequence and packing configuration
sequence_len: 2048  # Adjust based on your data (max 32768 for Qwen2)
sample_packing: true  # Efficient training by packing multiple samples
eval_sample_packing: true

# LoRA configuration
adapter: qlora
lora_model_dir:
lora_r: 32  # Rank - higher = more parameters (8, 16, 32, 64)
lora_alpha: 64  # Typically 2x lora_r
lora_dropout: 0.05
lora_target_linear: true  # Target all linear layers

# Weights & Biases logging (optional)
wandb_project:  # Set to your W&B project name
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

# Training hyperparameters
gradient_accumulation_steps: 4  # H100 SXM can handle more, reduced from 8
micro_batch_size: 2  # H100 80GB has enough VRAM for batch size 2
num_epochs: 6  # Total 6 epochs (already did 2)
optimizer: adamw_torch_fused  # Fast optimizer
lr_scheduler: cosine
learning_rate: 0.0002  # Conservative for large models

# Precision settings
bf16: auto  # Use bfloat16 if available
tf32: true  # Enable TF32 for faster training on Ampere+ GPUs

# Memory optimization
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
flash_attention: true  # Faster and more memory efficient attention

# Training schedule
warmup_ratio: 0.1
evals_per_epoch: 4
saves_per_epoch: 2  # Increased for Vast.ai - save more frequently in case of interruption
logging_steps: 1
weight_decay: 0.0

# Checkpointing
resume_from_checkpoint:  # Can't resume from LoRA, will train from scratch
# save_first_step: true  # Uncomment to validate checkpoint saving

# IMPORTANT FOR VAST.AI: Upload to HuggingFace Hub for persistence
# Vast.ai storage is not persistent after instance stops!
# REPLACE "your-username" with your actual HuggingFace username:
hub_model_id: LainBear/dolphin-qwen2-72b-lainbear
hub_strategy: every_save  # Auto-upload after each checkpoint
hub_private_repo: false  # Public repo

# Special tokens (uncomment if needed)
# special_tokens:
#   bos_token: "<|im_start|>"
#   eos_token: "<|im_end|>"
#   pad_token: "<|im_end|>"

</details><br>

dolphin-qwen2-72b-lainbear

This model is a fine-tuned version of cognitivecomputations/dolphin-2.9.2-qwen2-72b on the discord_training_data.jsonl dataset. It achieves the following results on the evaluation set:

• Loss: 5.5116
• Memory/max Active (gib): 49.84
• Memory/max Allocated (gib): 49.84
• Memory/device Reserved (gib): 70.01

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 56
• training_steps: 564

Training results

Framework versions

• PEFT 0.17.1
• Transformers 4.57.0
• Pytorch 2.7.1+cu126
• Datasets 4.0.0
• Tokenizers 0.22.1