dictalm2.0-instruct-fine-tuned

Model Card for ronigold/dictalm2.0-instruct-fine-tuned

This is a fine-tuned version of the Dicta-IL dictalm2.0-instruct model, specifically tailored for generating question-answer pairs based on Hebrew Wikipedia excerpts. The model was fine-tuned to improve its ability in understanding and generating natural questions and their corresponding answers in Hebrew.

Model Details

Model Description

The model, ronigold/dictalm2.0-instruct-fine-tuned, is a fine-tuned version of the dictalm2.0-instruct model on a synthetically generated dataset. This dataset was created by the model itself using excerpts from the Hebrew Wikipedia, which then were used to generate questions and answers, thereby enriching the model's capacity in this specific task.

• Developed by: Roni Goldshmidt
• Model type: Transformer-based, fine-tuned Dicta-IL dictalm2.0-instruct
• Language(s) (NLP): Hebrew
• License: MIT
• Finetuned from: dicta-il/dictalm2.0-instruct

Uses

Direct Use

The model is ideal for educational and informational applications, where generating contextual question-answer pairs from textual content is needed, particularly in the Hebrew language.

Out-of-Scope Use

The model is not intended for generating answers where factual accuracy from unverified sources is critical, such as medical advice or legal information.

Bias, Risks, and Limitations

While the model is robust in generating context-relevant Q&A pairs, it may still inherit or amplify biases present in the training data, which primarily comes from Wikipedia. Users should critically evaluate the model output, especially in sensitive contexts.

Recommendations

It is recommended to use this model with an additional layer of human oversight when used in sensitive or critical applications to ensure the accuracy and appropriateness of the content generated.

How to Get Started with the Model

To get started, load the model using the Transformers library by Hugging Face:

from transformers import AutoModelForQuestionAnswering, AutoTokenizer

model_name = "ronigold/dictalm2.0-instruct-fine-tuned"
model = AutoModelForQuestionAnswering.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Training Details

Training Data

The training data consists of synthetic question-answer pairs generated from the Hebrew Wikipedia. This data was then used to fine-tune the model using specific loss functions and optimization strategies to improve its performance in generating similar pairs.

# Example of setting up training in PyTorch using the Transformers library
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=3,              # number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=10,
)

trainer = Trainer(
    model=model,                         # the instantiated 🤗 Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=eval_dataset            # evaluation dataset
)

trainer.train()

Training Procedure

Training Hyperparameters

• Training regime: Mixed precision training (fp16) to optimize GPU usage and speed up training while maintaining precision.

# Configuration for mixed precision training
from transformers import set_seed

set_seed(42)  # Set seed for reproducibility

# Adding mixed precision policy
from torch.cuda.amp import GradScaler, autocast

scaler = GradScaler()

# Training loop
for epoch in range(int(training_args.num_train_epochs)):
    model.train()
    for batch in train_dataloader:
        optim.zero_grad()
        with autocast():  # applies mixed precision
            outputs = model(**batch)
            loss = outputs.loss
        scaler.scale(loss).backward()
        scaler.step(optim)
        scaler.update()

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model was evaluated on a separate holdout set, also generated synthetically in a similar manner as the training set.

Factors

• Domains: The evaluation considered various domains within the Hebrew Wikipedia to ensure generalizability across different types of content.
• Difficulty: The questions varied in complexity to test the model's ability to handle both straightforward and more complex queries.

Metrics

The evaluation metrics used include F1 score and exact match (EM), measuring the accuracy of the answers generated by the model.

Results

The model achieved an F1 score of 88% and an exact match rate of 75%, indicating strong performance in generating accurate answers, especially in context to the synthesized questions.

Technical Specifications

Model Architecture and Objective

The model follows a transformer-based architecture with modifications to optimize for question generation and answering tasks.

Compute Infrastructure

Training was performed on cloud GPUs, specifically using NVIDIA Tesla V100s, which provided the necessary compute power for efficient training.

Environmental Impact

<!-- Optional section: Discuss any measures taken to mitigate environmental impact during training, such as using renewable energy sources or carbon offsets. -->

Citation

BibTeX:

@misc{ronigold_dictalm2.0_instruct_finetuned_2024,
  author = {Goldshmidt, Roni},
  title = {Hebrew QA Fine-tuned Model},
  year = {2024},
  publisher = {Hugging Face's Model Hub},
  journal = {Hugging Face's Model Hub}
}

More Information

For more detailed usage, including advanced configurations and tips, refer to the repository README or contact the model authors. This model is part of a broader initiative to enhance NLP capabilities in the Hebrew language, aiming to support developers and researchers interested in applying advanced AI techniques to Hebrew texts.

Model Card Authors

• Roni Goldshmidt: Main researcher and developer of the fine-tuned model.

Model Card Contact

For any questions or feedback about the model, contact via Hugging Face profile or directly at [email protected].