Qwen2-1.5B-Medical

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

Qwen2-1.5B-Medical-Instruct

This model is a pretrain+fine-tuned+rlhf version of Qwen2-1.5B-Instruct on the medical dataset

Model description

Detailed model description can be seen on the base model Qwen/Qwen2-1.5B-Instruct page. This model has been pre-trained on medical industry, then fine-tuned all lora_target, and finally trained with reinforcement learning.

Training and evaluation data

This model is a pretrain+fine-tuned+rlhf version of Qwen2-1.5B-Instruct on the medical dataset

Training procedure

Pre-train

pre-trained on medical industry

SFT

fine-tuned on sft data

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 8
• total_train_batch_size: 512
• total_eval_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 4

Training results

DPO

human feedback reinforcement learning on medical reward data

{
    "epoch": 3.764705882352941,
    "eval_logits/chosen": -1.19295334815979,
    "eval_logits/rejected": -0.7887511253356934,
    "eval_logps/chosen": -150.47561645507812,
    "eval_logps/rejected": -75.58721160888672,
    "eval_loss": 0.6550262570381165,
    "eval_rewards/accuracies": 1.0,
    "eval_rewards/chosen": 0.03167621046304703,
    "eval_rewards/margins": 0.13228271901607513,
    "eval_rewards/rejected": -0.10060650110244751,
    "eval_runtime": 1.805,
    "eval_samples_per_second": 55.403,
    "eval_steps_per_second": 3.878
}

Framework versions

• PEFT 0.12.0
• Transformers 4.44.2
• Pytorch 2.4.0
• Datasets 2.21.0
• Tokenizers 0.19.1