ERNIE-4.5-0.3B-Yue

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

ERNIE-4.5-0.3B-Yue

This model is a specialized, fine-tuned version of baidu/ERNIE-4.5-21B-A3B-Base-PT, focused on adapting the base model to the linguistic and cultural nuances of Hong Kong Cantonese.

Model description

The primary purpose of developing ERNIE-4.5-0.3B-Yue is to serve as an experimental case study to:

1. Evaluate the efficiency of Continual Pre-Training (CPT) in boosting the performance of a small, 0.3B parameter model for a low-resource, specialized dialect (Hong Kong Cantonese).
2. Measure Catastrophic Forgetting of general knowledge after CPT and Supervised Fine-Tuning (SFT) against the gained specialization.

The model is the result of a two-stage tuning process: Continual Pre-Training (CPT) followed by Supervised Fine-Tuning (SFT).

1. CPT Stage: The base model was first continually pre-trained on a large corpus of Hong Kong-specific text, which includes informal Cantonese expressions, local terminology, and common written forms unique to the region.
2. SFT Stage: The model was then supervised fine-tuned on the sft_v2 dataset, which is designed to teach the model to follow instructions and generate high-quality, conversational responses relevant to Hong Kong daily life and culture.

This combined approach makes the model highly proficient in generating natural, contextually accurate Cantonese text.

Intended uses & limitations

Intended Uses

• Experimental Study: Testing the efficiency of CPT on small models for dialect adaptation and observing the plasticity-stability trade-off (catastrophic forgetting).
• Conversational AI: Building chatbots or virtual assistants intended for users in Hong Kong, capable of understanding and generating natural Cantonese, mixed with written Chinese characters (Vernacular Chinese), and unique local vocabulary.
• Localized Content Generation: Generating summaries, translations (e.g., from English/Mandarin to local Cantonese expressions), or creative content with a strong Hong Kong flavor.
• Dialectal Research: Serving as a strong baseline model for further research into the linguistic specifics of the Yue dialect.

Limitations

• The specialization towards Hong Kong Cantonese means its performance on standard Mandarin Chinese or general English tasks may not be as robust as general-purpose models.
• The model size (0.3B parameters) may limit its ability to handle extremely complex reasoning or long-context tasks compared to larger models.
• Factual Recall Deficiencies: Due to its small parameter count and the observed catastrophic forgetting, the model may exhibit limitations in recalling and accurately generating specific factual knowledge outside its specialized domain.

Training and evaluation data

The model was fine-tuned on Hong Kong-specific Cantonese datasets, including:

• CPT Dataset: A large, diverse corpus of text focusing on local news, social media, forums, and colloquial dialogue to capture the breadth of Hong Kong Cantonese usage.
• SFT Dataset (sft_v2): A curated set of high-quality instruction-response pairs designed to improve the model's ability to follow complex instructions and engage in natural, helpful conversation, specifically tailored to local context.

Note: Specific details about the size and composition of the proprietary datasets are omitted, but they are focused on Hong Kong/Cantonese usage.

Evaluation Benchmark

The performance metrics presented below are derived from the HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMs.

• Repository: https://github.com/hon9kon9ize/hkeval2025

Training procedure

The model underwent two distinct phases: CPT and SFT, building upon the initial hon9kon9ize/ernie-4.5-0.3B-cpt base.

Training hyperparameters

The following hyperparameters were used during the Supervised Fine-Tuning (SFT) phase:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 1
• seed: 42
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9, 0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3.0

Training results

Evaluation and Catastrophic Forgetting Analysis

The following metrics compare the performance of the final fine-tuned model (ERNIE-4.5-0.3B-Yue) against the base model (ERNIE-4.5-0.3B) to measure the degree of domain specialization gain versus the catastrophic forgetting of general knowledge.

Analysis of Catastrophic Forgetting:

The results confirm the experimental hypothesis of a specialization trade-off, demonstrating the plasticity-stability dilemma common in continual learning.

1. Observed Forgetting: The model shows clear signs of catastrophic forgetting on general and broad knowledge metrics (MMLU and Professional scores dropped by 0.0132 and 0.0481, respectively). This reflects the overwriting of general-purpose weights in favor of the new, specialized domain knowledge.
2. Specialization Efficacy: This forgetting is justified by highly significant gains in the critical targeted domain tasks. The model achieved a dramatic ↑ 0.0619 improvement on the highly specific HK Law benchmark and a notable ↑ 0.0263 gain on the DSE (Hong Kong education) task.
3. Domain Anchoring: The overall Cantonese general knowledge (CantoMMLU) remained highly stable (negligible ↓ 0.0009 drop), indicating that the CPT/SFT process successfully anchored the small model to the target language/dialect without losing its foundational Cantonese understanding.

Framework versions

• Transformers 4.57.1
• Pytorch 2.6.0+cu124
• Datasets 4.0.0
• Tokenizers 0.22.1