Anchor-DistilIndoBERT

Anchor-DistilIndoBERT for Emotion Detection (Criticism vs Appreciation)

Anchor-DistilIndoBERT is a fine-tuned variant of DistilBERT-base-Indonesian developed for detecting emotional tone in Indonesian-language comments. The model focuses on distinguishing between Criticism (Kritik) and Appreciation (Apresiasi), especially from YouTube comments discussing government performance.

---

1. Model Idea

This work adapts the concept proposed in
"Improving multilabel text emotion detection with emotion interrelation anchors"
(DOI: 10.1016/j.nlp.2025.100170).

The referenced paper introduces AnchorBERT, which enhances multilabel emotion detection by representing each emotion as a learned anchor vector derived from samples that purely express that emotion.
Each text embedding interacts with these emotion anchors via a multi-head attention mechanism, enabling emotion interrelation modeling before final classification.

In this adaptation, Anchor-DistilIndoBERT applies the same concept to Indonesian text:

• Uses DistilIndoBERT as the encoder.
• Represents Criticism and Appreciation as mean anchor embeddings.
• Employs a multi-label classification head for binary emotion prediction.

---

2. Model Details

---

3. Dataset

This dataset is exploratory and aims to demonstrate how anchor-based emotion representation can enhance Indonesian emotion classification.

---

4. Training Configuration

---

5. Performance

• Validation micro-F1: ~0.84

Training: 100%|██████████| 11/11 [01:19<00:00,  7.27s/it]
Epoch 1 | loss=0.6490 | val_micro-F1=0.7767
Training: 100%|██████████| 11/11 [01:20<00:00,  7.35s/it]
Epoch 2 | loss=0.5834 | val_micro-F1=0.8333
Training: 100%|██████████| 11/11 [01:17<00:00,  7.06s/it]
Epoch 3 | loss=0.5430 | val_micro-F1=0.8364

(Performance may improve with larger, balanced datasets.)

---

6. Example Usage

import torch
from transformers import AutoTokenizer, AutoModel
from huggingface_hub import hf_hub_download
# === Custom Model ===
class AnchorBERT(torch.nn.Module):
    def __init__(self, base_model_name="cahya/distilbert-base-indonesian", num_labels=2, hidden_dropout=0.1, attn_heads=8):
        super().__init__()
        self.encoder = AutoModel.from_pretrained(base_model_name)
        hidden_size = self.encoder.config.hidden_size
        self.mha = torch.nn.MultiheadAttention(embed_dim=hidden_size, num_heads=attn_heads, batch_first=True)
        self.dropout = torch.nn.Dropout(hidden_dropout)
        self.classifier = torch.nn.Sequential(
            torch.nn.Linear(hidden_size * 2, hidden_size),
            torch.nn.ReLU(),
            torch.nn.Dropout(hidden_dropout),
            torch.nn.Linear(hidden_size, num_labels)
        )

    def forward(self, input_ids, attention_mask, anchors_per_batch):
        outputs = self.encoder(input_ids=input_ids, attention_mask=attention_mask, return_dict=True)
        if hasattr(outputs, "pooler_output") and outputs.pooler_output is not None:
            cls = outputs.pooler_output
        else:
            last = outputs.last_hidden_state
            mask = attention_mask.unsqueeze(-1)
            cls = (last * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1e-9)

        query = cls.unsqueeze(1)
        attn_out, _ = self.mha(query=query, key=anchors_per_batch, value=anchors_per_batch)
        attn_out = attn_out.squeeze(1)

        enriched = torch.cat([cls, attn_out], dim=-1)
        logits = self.classifier(enriched)
        return logits

# === Config ===
REPO_ID = "BillyCemerson/Anchor-DistilIndoBERT"
MODEL_NAME = "cahya/distilbert-base-indonesian"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

# === Load Tokenizer ===
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

# === Download model weights ===
model_path = hf_hub_download(repo_id=REPO_ID, filename="pytorch_model.bin")

model = AnchorBERT(base_model_name=MODEL_NAME, num_labels=2).to(DEVICE)
state_dict = torch.load(model_path, map_location=DEVICE)
model.load_state_dict(state_dict)
model.eval()

# === Load anchors ===
anchors_path = hf_hub_download(repo_id=REPO_ID, filename="mean_anchors.pt")
mean_anchors = torch.load(anchors_path, map_location=DEVICE)

# === Dummy helper (for batch anchors) ===
def mean_anchors_for_batch(anchors, batch_size):
    return anchors.unsqueeze(0).repeat(batch_size, 1, 1)

# === Inference ===
text = "Mantap kinerja pak Prabowo, Gibran ganti aja"
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True).to(DEVICE)

with torch.no_grad():
    # Access the tensor within the dictionary
    anchors_batch = mean_anchors_for_batch(mean_anchors[1], inputs["input_ids"].shape[0]).to(DEVICE)
    logits = model(inputs["input_ids"], inputs["attention_mask"], anchors_batch)
    probs = torch.sigmoid(logits).cpu().numpy()

print(f"Text: {text}")
print(f"Kritik={probs[0][0]:.3f} | Apresiasi={probs[0][1]:.3f}")

Expected output:

Text: Mantap kinerja pak Prabowo, Gibran ganti aja
Kritik=0.644 | Apresiasi=0.581

---

7. Citation

If you use this model or build upon it, please cite the following:

@article{anchorbert2025,
  title={Improving multilabel text emotion detection with emotion interrelation anchors},
  author={Zhou, M. and others},
  journal={Natural Language Processing Journal},
  year={2025},
  doi={10.1016/j.nlp.2025.100170}
}

@misc{anchordistilindobert2025,
  title={Anchor-DistilIndoBERT for Emotion Detection (Criticism vs Appreciation)},
  author={Billy Cemerson},
  year={2025},
  note={Fine-tuned model for multi-label emotion classification in Indonesian text.}
}

---

8. License

This model is released under the MIT License, allowing free use, modification, and redistribution with attribution.