llama3.2-1B-persianQAV2.0

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

llama3.2-1B-persianQAV2.0

Model description

Persian-QA-LLaMA is a fine-tuned version of LLaMA (1B parameters) optimized for Persian language question answering tasks. The model uses Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning, making it a lightweight adaptation that maintains the base model's capabilities while adding Persian language understanding. This adaptation specifically targets question-answering capabilities, allowing the model to understand and respond to questions in Persian while maintaining computational efficiency. This model is a fine-tuned version of meta-llama/Llama-3.2-1B-Instruct on azizmatin/question_answering dataset.

Intended uses & limitations

Intended Uses

Persian language question answering Information extraction from Persian texts Educational support and tutoring systems Research and academic applications

Limitations

Performance may vary on domain-specific questions Not optimized for other NLP tasks beyond question answering May struggle with highly colloquial or dialectal Persian Limited by the training dataset's coverage and diversity Should not be used for generating factual content without verificatio

Training and evaluation data

The model was trained on a modified version of a Persian question-answering dataset structured similarly to SQuAD v2. The dataset includes:

• Question-answer pairs in Persian
• Contextual passages for answer extraction
• Various question types and difficulty levels
• Coverage across different topics and domains

The dataset was split into training, validation, and test sets to ensure robust evaluation of model performance.

Training procedure

Training Details

• Base Model: LLaMA 1B
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Training Framework: Hugging Face Transformers with PEFT
• Optimization: AdamW optimizer with learning rate scheduling
• Training was conducted with parameter-efficient fine-tuning techniques to maintain efficiency

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.15
• num_epochs: 3
• mixed_precision_training: Native AMP AMC

Training results

Performance Metrics

The model was evaluated on the validation set, with the following results:

Framework versions

• PEFT 0.13.2
• Transformers 4.46.1
• Pytorch 2.5.0+cu121
• Datasets 3.1.0
• Tokenizers 0.20.1