distily_bench_obj_cross_v2.8

distily_bench_obj_cross_v2.8

This student model is distilled from the teacher model roneneldan/TinyStories-33M using the dataset (unspecified).

The Distily library was used for this distillation.

It achieves the following results on the evaluation set:

• eval_enwikippl: 13782.8369
• eval_frwikippl: 65787.1094
• eval_zhwikippl: 58004.7852
• eval_tinystoriesppl: 6904.7764
• eval_loss: 5.9830
• eval_runtime: 6.5013
• eval_samples_per_second: 76.908
• eval_steps_per_second: 9.69

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed -->

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
• train_embeddings: True
• learning_rate: 0.004
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 1.0

Resource Usage

Peak GPU Memory: 6.6058 GB

Eval-Phase Metrics

Framework versions

• Distily 0.2.0
• Transformers 4.44.0
• Pytorch 2.3.0
• Datasets 2.21.0