finetune_cls_vs_content_12class_v5

finetune_cls_vs_content_12class_v5

TL;DR

A multi-task Vietnamese web content classifier built on PhoBERT. Given a text segment extracted from a Vietnamese website, the model simultaneously predicts:

• Task 1 (Binary): Whether the content constitutes a copyright violation (0 = safe, 1 = violation) — F1 macro 0.983
• Task 2 (12-class): The content category of the website — F1 macro 0.790

This is the production-grade v5 checkpoint, the best-performing model in a 5-version iterative fine-tuning project.

---

Model Details

Model Description

finetune_cls_vs_content_12class_v5 is a dual-task text classifier fine-tuned on Vietnamese web content segments. It extends the 10-class backbone ttqdunggg/finetune_cls_vs_content_ronbackbone_100k (PhoBERT + multi-task head) to cover 12 content categories including two newly added classes: Phim ảnh (Film/Entertainment) and Thể thao (Sports).

The model is the culmination of 5 progressive fine-tuning iterations (v1–v5), each addressing specific data quality issues identified from misclassification analysis. v5 achieved the best overall Macro F1 (0.790) across all versions, with particular improvements in MXH (+0.157 vs v4) driven by strict data cleaning strategies.

• Developed by: phucleDio
• Model type: Multi-task Sequence Classification (PhoBERT / RoBERTa-based)
• Languages: Vietnamese (vi)
• License: MIT
• Fine-tuned from: ttqdunggg/finetune_cls_vs_content_ronbackbone_100k
• Base backbone: vinai/phobert-base-v2 (RoBERTa architecture, ~135M parameters)

Model Sources

• Repository: phucleDio/finetune_cls_vs_content_12class_v5
• Predecessor (10-class backbone): ttqdunggg/finetune_cls_vs_content_ronbackbone_100k

---

Uses

Direct Use

This model is designed for automated classification of Vietnamese website content segments. Each input is a text chunk (up to 256 Vietnamese word-piece tokens after ViTokenizer segmentation) extracted from a webpage.

Primary use cases:

• Copyright violation detection in Vietnamese web crawling pipelines (Task 1)
• Content category labeling for downstream filtering, moderation, or analytics (Task 2)
• Identifying high-risk content domains (gambling, adult content, lending, etc.)

Downstream Use

The model can be plugged into:

• Web crawling and content indexing pipelines as a pre-filter
• Copyright enforcement systems that need to classify the nature of flagged content
• Content moderation dashboards
• Recommender system pre-processing to filter out certain content categories

Out-of-Scope Use

• Non-Vietnamese content: The model is trained exclusively on Vietnamese web text and will produce unreliable results on other languages.
• Very short inputs (< 20 tokens): The model may not have sufficient context to classify reliably.
• Individual social media posts or comments: The training data consists of full webpage segments, not micro-text.
• Image, audio, or video content: Text-only model; does not process multimedia.
• Fine-grained sub-category classification: E.g., distinguishing specific sports types or film genres.

---

The 12 Content Categories (Task 2)

---

Bias, Risks, and Limitations

Known Limitations

1. E-commerce ↔ Tổ chức confusion (structural): The most persistent misclassification (≈27–32% of E-commerce predictions) is E-commerce → Tổ chức. Many Vietnamese e-commerce websites include company registration info, tax IDs, and organizational-looking footer boilerplate that overlaps with organizational content. This is partly an annotation ambiguity issue baked into the original dataset.

2. MXH underrepresentation: With only 551 training samples, MXH (social media) is the smallest class. While v5 recovers to F1=0.600, it remains the lowest-performing class and is sensitive to data distribution shifts.

3. Báo chí ↔ Phim ảnh / Thể thao overlap: Vietnamese entertainment news sites (e.g., kenh14.vn, 2sao.vn) produce content that straddles Báo chí and Phim ảnh. Sports news on general newspapers overlaps with Thể thao.

4. 18+ recall sensitivity: With only 32 validation samples, the 18+ class F1 (0.889) has high variance and may not generalize reliably to unseen adult content domains.

5. Vietnamese-only: The model uses ViTokenizer word segmentation and PhoBERT's Vietnamese vocabulary. Non-Vietnamese text will be poorly tokenized.

Recommendations

• For production use, consider post-processing rules based on URL/domain patterns (e.g., .gov.vn → Tổ chức) to resolve E-commerce ↔ Tổ chức ambiguity.
• Apply a confidence threshold on Task 2 predictions; for low-confidence outputs (softmax max < 0.5), treat as Chưa xác định.
• Monitor MXH performance carefully in deployment; add more training data if precision drops below 0.50.

---

How to Get Started with the Model

import torch
from transformers import AutoTokenizer
from pyvi import ViTokenizer

# Load tokenizer from PhoBERT base
tokenizer = AutoTokenizer.from_pretrained("vinai/phobert-base-v2")

# Load model
from transformers import AutoModel
import torch.nn as nn

class PhoBERTMultiTask12(nn.Module):
    def __init__(self, base_model_name="vinai/phobert-base-v2",
                 num_labels_task1=2, num_labels_task2=12):
        super().__init__()
        self.roberta = AutoModel.from_pretrained(base_model_name)
        hidden = self.roberta.config.hidden_size
        self.classifier_task1 = nn.Sequential(
            nn.Linear(hidden, hidden), nn.Tanh(), nn.Dropout(0.1),
            nn.Linear(hidden, num_labels_task1)
        )
        self.classifier_task2 = nn.Sequential(
            nn.Linear(hidden, hidden), nn.Tanh(), nn.Dropout(0.1),
            nn.Linear(hidden, num_labels_task2)
        )

    def forward(self, input_ids, attention_mask, **kwargs):
        outputs = self.roberta(input_ids=input_ids, attention_mask=attention_mask)
        cls = outputs.last_hidden_state[:, 0, :]
        return self.classifier_task1(cls), self.classifier_task2(cls)

# Inference
LABEL_NAMES = {
    0: "Bao chi", 1: "18+", 2: "Co bac", 3: "Vay", 4: "Tien ao",
    5: "To chuc", 6: "E-commerce", 7: "MXH", 8: "Game",
    9: "Chua xac dinh", 10: "Phim anh", 11: "The thao"
}

def predict(text: str, model, tokenizer, device="cpu"):
    # Vietnamese word segmentation (required for PhoBERT)
    segmented = ViTokenizer.tokenize(text)
    
    inputs = tokenizer(
        segmented,
        return_tensors="pt",
        max_length=256,
        padding="max_length",
        truncation=True
    ).to(device)
    
    model.eval()
    with torch.no_grad():
        logits1, logits2 = model(**inputs)
    
    binary_pred = torch.argmax(logits1, dim=-1).item()
    category_pred = torch.argmax(logits2, dim=-1).item()
    
    return {
        "copyright_violation": bool(binary_pred),      # Task 1
        "content_category": LABEL_NAMES[category_pred], # Task 2
        "confidence_task1": torch.softmax(logits1, dim=-1).max().item(),
        "confidence_task2": torch.softmax(logits2, dim=-1).max().item(),
    }

# Example
model = PhoBERTMultiTask12()
# Load weights from HuggingFace Hub
# model.load_state_dict(torch.load(...))

text = "Mua ngay! Thêm vào giỏ hàng. Giá ưu đãi 199.000đ. Miễn phí giao hàng toàn quốc."
result = predict(text, model, tokenizer)
# {'copyright_violation': False, 'content_category': 'E-commerce', ...}

---

Training Details

Training Data

Dataset: CLS_segment_25K_12class_v5.csv — 24,186 segments from Vietnamese websites.

Train / Val split: 85% / 15%, stratified by class (type).

• Training set: 20,558 segments
• Validation set: 3,628 segments

Data provenance and key cleaning decisions across versions:

v5 Strict Shopping-Feature Rule (E-commerce filter): A segment is kept in E-commerce training only if it contains at least one of: giỏ hàng, thêm vào giỏ, xem giỏ hàng, bảng giá, mua ngay, đơn hàng. 229 segments without these signals were discarded.

Footer Boilerplate Stripping: 290 E-commerce segments had boilerplate text (Giấy chứng nhận ĐKKD, MST, Địa chỉ trụ sở, Powered by) stripped before training to prevent false Tổ chức associations.

Training Procedure

Architecture

PhoBERT Backbone (vinai/phobert-base-v2, ~135M params)
         │  [CLS] embedding
    ┌────┴────┐
    ▼         ▼
Task 1      Task 2
Linear→Tanh→Dropout→Linear(2)   Linear→Tanh→Dropout→Linear(12)
Binary (0=safe, 1=violation)     12-class content category

Weight transfer from the 10-class backbone:

• roberta.* and classifier_task1.*: copied 1:1
• classifier_task2.out_proj.weight: first 10 rows copied, 2 new rows randomly initialized
• classifier_task2.out_proj.bias: first 10 copied, 2 new = 0

Training Hyperparameters (v5 Final)

The training followed a 2-phase schedule:

Phase 1 — Warmup (3 epochs, backbone frozen):

Phase 2 — End-to-End Fine-tune (7 epochs, all parameters trainable):

Loss function:

Total Loss = 0.35 × CrossEntropy(logits1, labels_task1)
           + 0.65 × WeightedCrossEntropy(logits2, labels_task2, class_weights)
Label smoothing applied to both tasks.

Combined metric for early stopping:

f1_combined = harmonic_mean(F1_task1_macro, F1_task2_macro)

Speeds, Sizes, Times

---

Evaluation

Testing Data

Validation set: 3,628 segments (15% stratified split from CLS_segment_25K_12class_v5.csv), held out during training.

Metrics

• F1 macro: Unweighted mean of per-class F1. Primary metric for Task 2 due to class imbalance.
• Accuracy: Overall prediction accuracy.
• Per-class precision / recall / F1: Reported for each of the 12 content categories.

Results

Task 1 — Binary Classification (Copyright Violation Detection)

              precision    recall  f1-score   support

           0     0.9931    0.9964    0.9948      3053
           1     0.9805    0.9635    0.9719       575

    accuracy                         0.9912      3628
   macro avg     0.9868    0.9799    0.9833      3628

Task 2 — 12-class Content Classification

                precision    recall  f1-score   support

      Bao chi     0.8000    0.7736    0.7866       212
          18+     0.9032    0.8750    0.8889        32
       Co bac     0.8815    0.9520    0.9154       125
          Vay     0.9256    0.9803    0.9522       203
      Tien ao     0.7812    0.8523    0.8152        88
      To chuc     0.6127    0.8593    0.7154       718
   E-commerce     0.9463    0.6031    0.7367      1081
          MXH     0.5327    0.6867    0.6000        83
         Game     0.8056    0.8614    0.8325       101
Chua xac dinh     0.7857    0.7869    0.7863       643
     Phim anh     0.6118    0.7266    0.6643       128
     The thao     0.7490    0.8364    0.7903       214

     accuracy                         0.7652      3628
    macro avg     0.7779    0.8161    0.7903      3628

Cross-task Overlap Matrix (Báo chí / Chưa xác định / Phim ảnh / Thể thao)

                 Bao chi  Chua xd  Phim anh  The thao
Bao chi            0.90     0.04      0.03      0.03
Chua xac dinh      0.02     0.94      0.02      0.01
Phim anh           0.02     0.05      0.89      0.05
The thao           0.02     0.01      0.02      0.96

Version History (Task 2 Macro F1)

Summary

v5 achieves the best overall Macro F1 (0.790) across all 5 versions. Key highlights:

• Task 1 (copyright binary): F1 0.983, Accuracy 99.12% — consistently excellent
• Top-performing classes: Vay (0.952), Cờ bạc (0.915), 18+ (0.889), Game (0.833)
• Weakest class: MXH (0.600) due to limited training data (551 samples)
• Persistent challenge: E-commerce ↔ Tổ chức confusion (~28% of E-com predicted as Tổ chức)

---

Environmental Impact

Carbon emissions estimated using the Machine Learning Impact calculator.

• Hardware Type: NVIDIA A100 (Google Colab Pro+)
• Hours used: ~3 hours per training run × 5 versions ≈ 15 GPU-hours total
• Cloud Provider: Google Colab
• Compute Region: Unknown (Colab managed infrastructure)
• Carbon Emitted: ~0.5–1.5 kg CO₂eq estimated (15 A100-hours)

---

Technical Specifications

Model Architecture

• Backbone: PhoBERT (RoBERTa-based), 12 transformer layers, 768 hidden size, ~135M parameters
• Task 1 Head: Linear(768 → 768) → Tanh → Dropout(0.1) → Linear(768 → 2)
• Task 2 Head: Linear(768 → 768) → Tanh → Dropout(0.1) → Linear(768 → 12)
• Classifier input: [CLS] token representation
• Tokenization: ViTokenizer (Vietnamese word segmentation) → PhoBERT BPE tokenizer, max_length=256

Software Stack

---

Glossary

---

Citation

If you use this model in your work, please cite:

BibTeX:

@misc{phucleDio2026contentclassifier,
  author       = {phucleDio},
  title        = {Vietnamese Web Content Classifier (12-class, v5)},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/phucleDio/finetune_cls_vs_content_12class_v5}},
  note         = {Fine-tuned from ttqdunggg/finetune_cls_vs_content_ronbackbone_100k}
}

---

Model Card Authors

phucleDio

Model Card Contact

Please open an issue on the HuggingFace model page for questions or bug reports.