TinyBERT-SarcOji-KD-v2

TinyBERT-SarcOji-KD-v2

tinybert student fine-tuned on SarcOji with sarcasm-aware emoji embeddings injected directly into the token-embedding matrix, distilled from Vandita/Bert-finetuned-Sarc.

Results

SarcOjiTest2 is ~75/25 negative-skewed, so its accuracy is not comparable to Test1's — MCC and ROC-AUC are the meaningful columns there.

Distillation

Hinton et al. (2015): L = α·T²·KL(teacher‖student) + (1−α)·CE, with T = 3, α = 0.5. The T² factor is applied, so soft-target gradients do not shrink as T rises.

Emoji handling

1444 emoji tokens added to the tokenizer (vocab 30522 → 31959). Each new row of the 312-d embedding matrix was initialised from a 300-d sarcasm-aware emoji vector using concat-mean. Remaining dimensions hold the mean of the original vocabulary embeddings rather than zeros, so new tokens are not anchored to the origin.

Because the emoji are part of the vocabulary, this is a standard HuggingFace model — no custom architecture, no custom loader.

Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Vandita/TinyBERT-SarcOji-KD-v2")
model = AutoModelForSequenceClassification.from_pretrained("Vandita/TinyBERT-SarcOji-KD-v2")

Data

SarcOji, 80:20 stratified split. Duplicates retained; identical texts were kept on one side of the split so no text appears in both train and validation. Evaluated on SarcOjiTest1 and SarcOjiTest2 in full.