KD-Teachers

KD-Teachers

Pre-trained teacher models used for Knowledge Distillation with Mixup augmentation on CIFAR-10 and CIFAR-100. These checkpoints are the teacher component of the KD-Mixup framework.

All models were fine-tuned from ImageNet pre-trained weights on CIFAR-10 and CIFAR-100 using SGD with momentum, ReduceLROnPlateau scheduling, and mixed precision training (float16).

---

Models

---

Performance

CIFAR-10

CIFAR-100

---

File Structure

cifar10/
├── best_resnet152v2.keras
├── best_convnexttiny.keras
├── best_convnextlarge.keras
└── best_vitbase.keras
cifar100/
├── best_resnet152v2.keras
├── best_convnexttiny.keras
├── best_convnextlarge.keras
└── best_vitbase.keras

---

Usage

Download a checkpoint and place it in your local checkpoints/teachers/{dataset}/ folder:

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="josemedina/KD-Teachers",
    filename="cifar100/best_resnet152v2.keras"
)

Then load it with Keras:

import tensorflow as tf

model = tf.keras.models.load_model(path)

The expected checkpoint path for the KD-Mixup training script is:

checkpoints/teachers/{dataset}/best_{teacher_name}.keras

---

Training Details

• Input size: 224 × 224 × 3
• Batch size: 250
• Optimizer: SGD (momentum=0.9, lr=1e-4)
• LR schedule: ReduceLROnPlateau (patience=3, factor=0.9, min_lr=1e-5)
• Max epochs: 500 (best checkpoint saved by val accuracy)
• Augmentation: Random crop, horizontal flip
• Precision: Mixed float16
• ViT-B/16 normalization: ImageNet mean/std ([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

---

Citation

If you use these checkpoints in your research, please cite:

@misc{medina2025kdmixup,
  author    = {Medina, Jos{\'e} and Hadachi, Amnir and Honeine, Paul and Bensrhair, Abdelaziz},
  title     = {Beyond Dark Knowledge: Mixup-Based Knowledge Distillation Under Vicinal Teacher Distributions},
  year      = {2025},
  publisher = {University of Tartu},
  url       = {https://github.com/JoseLMedinaC/KD-Mixup}
}