distily_bench_obj_cross_v2.10

distily_bench_obj_cross_v2.10

This student model is distilled from the teacher model roneneldan/TinyStories-33M using the dataset (unspecified).

The Distily library was used for this distillation.

It achieves the following results on the evaluation set:

• eval_enwikippl: 107.6398
• eval_frwikippl: 10204.3643
• eval_zhwikippl: 49954.8242
• eval_tinystoriesppl: 6.6903
• eval_loss: 0.7036
• eval_runtime: 13.0602
• eval_samples_per_second: 76.568
• eval_steps_per_second: 9.571

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed -->

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
• train_embeddings: True
• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 1.0

Resource Usage

Peak GPU Memory: 6.6064 GB

Eval-Phase Metrics

Framework versions

• Distily 0.2.0
• Transformers 4.44.0
• Pytorch 2.3.0
• Datasets 2.21.0