finetuning_original_model

XpertGPT Fine-Tuning Pipeline

This repository contains the fine-tuning pipeline and downstream evaluation results for the XpertGPT model (anonym5035/swi_glu_sw_64_16_8_4_xpert_gpt) trained for the BabyLM 2026 Strict-Small track.

The pipeline (finetune_eval.py) is designed to fine-tune the causal pre-trained model on 9 diverse sequence classification and multiple-choice tasks, bypassing autoregressive generation in favor of classification heads.

Core Hyperparameters (Global)

Across all tasks, the following core training configurations are enforced:

Results & Task Hyperparameters

Because the datasets range from massive (~105K examples) to tiny (~250 examples), hyperparameters are bucketed by dataset size to prevent overfitting.

(Note: The "Base Batch Size" is per-GPU. Under a 2x T4 DataParallel setup, the effective batch sizes become 32 and 16 respectively).

Outputs

The script uploads two artifacts back to this repository upon completion:

1. results/finetune_results.json: The highest validation accuracy achieved for each dataset.
2. results/training_history.json: Epoch-by-epoch tracking of train_loss, val_loss, and val_acc for advanced overfitting analysis.

Kaggle Execution

To execute this pipeline inside a Kaggle notebook (using the T4 x2 accelerator):

# 1. Install dependencies
!pip install transformers datasets huggingface_hub torch tqdm

# 2. Download the fine-tuning script
!wget -O finetune_eval.py https://huggingface.co/anonym5035/finetuning_original_model/raw/main/finetune_eval.py

# 3. Run the pipeline and push results to Hugging Face
!python finetune_eval.py --hf_token YOUR_HF_TOKEN_HERE