DistilLlamaV1

Overview

This model is a distilled version of LLaMA 2, containing approximately 80 million parameters. It was trained using a mix of OpenWebText and WikiText Raw V1 datasets. Knowledge distillation was employed to transfer knowledge from a larger "teacher" model—Meta’s 7B LLaMA 2—to help this smaller model mimic the behavior of the teacher. This version is the latest version of DistilLlama, which has gone through 5 days of training using two Nvidia A100 80G GPU.

Update

30 out of 300 checkpoints were examined, and the one with the best performance in semantic and factual accuracy has now been updated in this repository.

Model Architecture

The architecture is based on LLaMA 2, with the following parameters:

Evaluation Metrics

1. Cosine Similarity using Word Embeddings

   • Description: Measures semantic similarity by mapping words/phrases to vectors.
   • Equation: Cosine Similarity = ( A • B ) / ( ||A|| ||B|| )
   • Example: "The dog chased the cat." vs. "A canine pursued a feline." (High similarity)

2. Exact Match (EM)

   • Description: Checks if critical keywords are present.
   • Example:
     • Expected: "Paris"
     • Response: "The capital of France is Paris." (EM = 1)

3. ROUGE Score

   • Description: Measures the overlap of the longest common subsequences between reference and response texts.
   • Equation:
     • Precision = Precision = LCS(R, C) / Length of C
     • Recall = Recall = LCS(R, C) / Length of R

Model Evaluation Summary

Note: CodeCarbon was used to track carbon emission. Allocated 80GB memory, 32 cores, Intel(R) Xeon(R) Gold 6448H for the evaluation

GitHub Repositories

• Training Repo: DistilLlama Training Repository
• Evaluation Repo: Knowledge Distillation Evaluation Repository

Reference

@misc{timiryasov2023babyllamaknowledgedistillation, title={Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty}, author={Inar Timiryasov and Jean-Loup Tastet}, year={2023}, eprint={2308.02019}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2308.02019}, }

Note: The repository will be updated as training progresses. Last update 2024-11-06