distily_bench_obj_cross_v2.4

distily_bench_obj_cross_v2.4

This student model is distilled from the teacher model roneneldan/TinyStories-33M using the dataset (unspecified).

The Distily library was used for this distillation.

It achieves the following results on the evaluation set:

• eval_enwikippl: 49928.3438
• eval_frwikippl: 60082.1211
• eval_zhwikippl: 75499.0547
• eval_tinystoriesppl: 44922.0352
• eval_loss: 6.1235
• eval_runtime: 13.0526
• eval_samples_per_second: 76.613
• eval_steps_per_second: 9.577

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed -->

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
• train_embeddings: True
• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 1.0

Resource Usage

Peak GPU Memory: 8.0568 GB

Eval-Phase Metrics

Framework versions

• Distily 0.2.0
• Transformers 4.44.0
• Pytorch 2.3.0
• Datasets 2.21.0