Mistral-Nemo-Kurdish

Continued pre-training on mistralai/Mistral-Nemo-Instruct-2407 using the Kurdish wiki dataset with unsloth. This model should be further fine-tuned since the pre-training was to improve Kurdish language understanding. It's a quantized model using bitsandbytes so that it uses less memory. See bitsandbytes documentation.

There isn't a standard or even a good Kurdish metric to evaluate the model (that I could find). Will make it my next project to create an evaluation so that there's a reproducible baseline for Kurdish.

Will look into a multi-GPU training setup so don't have to wait all day for results. Would like to train it with both Kurmanji and Sorani.

Use

Should be fine-tuned further for a specific task. See instruction fine-tuned model nazimali/Mistral-Nemo-Kurdish-Instruct.

Training

Transformers 4.44.2
1 NVIDIA A100 80GB PCIe
Duration 6h 31m 4s

{
  "total_flos": 4121524790259794000,
  "train/epoch": 1,
  "train/global_step": 1960,
  "train/grad_norm": 3.1958093643188477,
  "train/learning_rate": 0,
  "train/loss": 1.2108,
  "train_loss": 1.256846008738693,
  "train_runtime": 23227.1752,
  "train_samples_per_second": 2.7,
  "train_steps_per_second": 0.084
}

Pre-training data:

• nazimali/kurdish-wikipedia-articles
  • Dataset number of rows: 63,076
  • Filtered columns title, text
    • Must have at least 1 character
• Number of rows used for training: 62,720

Training prompt format:

training_prompt = """Gotara Wikipedia
### Sernav: {}

### Gotar:
{}"""