DistillLens-TinyLlama-1.1B

DistillLens-TinyLlama-1.1B

paper | code

DistillLens-TinyLlama-1.1B is a TinyLlama-1.1B model distilled from Llama (7B) on databricks-dolly-15k.

Method

<p align='left'> <img src="architecture.png" width="1000"> </p>

Note: DistillLens requires a TinyLlama-1.1B-intermediate-step-1431k-3T for initilization to perform the distillation.

Citation

@article{dhakal2026distilllens,
  title={DistillLens: Symmetric Knowledge Distillation Through Logit Lens},
  author={Dhakal, Manish and Jinadu, Uthman and Budathoki, Anjila and Sunderraman, Rajshekhar and Ding, Yi},
  journal={arXiv preprint arXiv:2602.13567},
  year={2026}
}