deberta-v3-large-ITO

DeBERTa-v3-large-ITO Fine-tuned Model

This model is a fine-tuned version of the deberta-v3-large model on a dataset of ITIL and DOLCE-related texts, with a total of 5500 chunks (256 tokens each).

Dataset

The dataset consists of texts from ITIL and DOLCE ontology-related articles, designed to aid tasks in ontology class representation and assist in merging the DOLCE and ITSMO ontologies. Dataset has been augmented by GPT-4o-mini inference. Added units: paraphrase, summary, topic, ITIL/DOLCE relations.

Prompts for augmentation:

augmetations = {'class': 'ITIL or Ontology, one word {ITIL} or {Ontology}',
                'topic': 'Topic in 5 words',
                'nearest ontology': 'find nearest analogy, process, class or entity from DOLCE ontology domain. 1 words, short sentence to explain, max 20 words',
                'nearest ITIL': 'find nearest analogy, process, class or entity from enterprise IT service domain. 1 words, short sentence to explain, max 20 words',
                'summary': 'text esscence in 20 words, no intro, just meaning'}

• Total Samples before augmentation: 5779
• Total Samples after augmentation: 17337
• Chunk Size: 256 tokens

Config

training_args = TrainingArguments(
    output_dir="./deberta-finetuned-ITIL-DOLCE",
    overwrite_output_dir=True,
    eval_strategy="steps",
    per_device_train_batch_size=16,
    per_device_eval_batch_size=2,
    num_train_epochs=100,
    save_steps=5_000,
    eval_steps=500,
    logging_steps=500,
    fp16=True
)

Intended Use

This model is optimized for class representation tasks in the context of ontology merging, specifically for DOLCE and ITSMO ontologies.

Training Details

Trained for ontology-based semantic representation and classification tasks.

Trained on raw data
Evaluation results: {'eval_loss': 3.6031947135925293, 'eval_runtime': 26.6242, 'eval_samples_per_second': 21.71, 'eval_steps_per_second': 10.855, 'epoch': 100.0}
Perplexity: 36.7153422841205

Trained on augmented data
Evaluation results: {'eval_loss': 1.4712743759155273, 'eval_model_preparation_time': 0.0065, 'eval_runtime': 76.2792, 'eval_samples_per_second': 22.732, 'eval_steps_per_second': 2.845}
Perplexity: 4.3547812347309875

---

Fine-tuned by Zamza.