mBert-finetuned-yiddish-experiment-3

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

bert-base-multilingual-cased-finetuned-yiddish-experiment-3

This model is a fine-tuned version of bert-base-multilingual-cased on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.4254

Model description

More information needed

Intended uses & limitations

Intended for use with a chatbot to correct raw Yiddish machine transcriptions, which have been generated by Transkribus

Training and evaluation data

Training dataset = Gavin model fine tuning_lines.csv

Training procedure

The training process described in Experiment 3 focuses on fine-tuning the pre-trained mBERT (multilingual BERT) model for improving raw handwritten text recognition (HTR). The fine-tuning dataset consists of raw HTR outputs paired with their human-corrected ground truth, as indicated in the line.csv file.

Key Parameters and Rationale:

1. Model Selection: The use of bert-base-multilingual-cased leverages the multilingual capabilities of BERT to accommodate the linguistic diversity likely present in the handwritten text dataset. This choice aligns well with the need to handle potentially mixed-language inputs or varying character distributions.

2. Data Handling:

• The dataset is loaded and structured into columns for raw HTR text and its hand-corrected counterpart.

1. Tokenization is performed using the mBERT tokenizer, with a maximum sequence length of 64 tokens. This length balances capturing sufficient context while preventing memory overhead.

3. Training Configuration:

• Batch Size and Gradient Accumulation: A batch size of 4 with a gradient accumulation step of 1 is chosen, likely due to the memory limitations of the L4 GPU, ensuring stable training while processing smaller data chunks.

• Learning Rate and Weight Decay: A low learning rate of 5e-6 allows for gradual updates to the pre-trained weights, preserving the pre-trained linguistic knowledge while adapting to the new task. Weight decay is set to 0 to avoid penalizing model parameters unnecessarily for this specific task.

• Gradient Clipping: The maximum gradient norm of 1 prevents exploding gradients, which could destabilize training given the small batch size and high learning rate sensitivity.

• Warm-Up Steps: 300 warm-up steps allow the optimizer to start with smaller updates, reducing initial instability.

• Epochs and Logging: The model is trained for 10 epochs with evaluation loss logged every 100 steps, providing a balance between sufficient training time and monitoring. Compute Setup:

The process was executed on an L4 GPU, which is optimized for such NLP workloads, providing efficient computation and faster training iterations.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 300
• num_epochs: 10

Training results

Framework versions

• Transformers 4.47.0
• Pytorch 2.5.1+cu121
• Datasets 3.1.0
• Tokenizers 0.21.0