codeforces-gpt-neo-lora

library_name: peft license: mit base_model: EleutherAI/gpt-neo-125M tags:

• code-generation
• python
• lora
• peft
• transformers
• competitive-programming pipeline_tag: text-generation model-index:
• name: codeforces-gpt-neo-lora results: [] datasets:
• MatrixStudio/Codeforces-Python-Submissions language:
• en

---

Codeforces GPT-Neo Python Assistant

Model Description

This model is a specialized version of EleutherAI/gpt-neo-125M, fine-tuned to assist in solving competitive programming challenges. It has been trained to generate structured Python solutions based on complex problem descriptions. This project was developed as a key task for the Generative AI track within the Digital Egypt Pioneers Initiative (DEPI) scholarship.

Intended Uses & Limitations

• Intended Use: Assistance with Python coding logic and competitive programming patterns.
• Limitations: Due to its lightweight architecture (125M parameters), the model may struggle with highly complex algorithmic logic and is best used as a coding assistant rather than a primary solver.

Training and Evaluation Data

The model was fine-tuned on a specialized subset of the MatrixStudio/Codeforces-Python-Submissions dataset. The training data followed an Instruction-Problem-Solution format to ensure the model learns to follow specific programming prompts effectively.

Training Procedure

Technique: LoRA (Low-Rank Adaptation)

To ensure efficient training, the model utilized PEFT (LoRA). This allowed for effective domain adaptation while significantly reducing the number of trainable parameters.

Training Hyperparameters

• Learning Rate: 0.0002
• Total Train Batch Size: 16 (4 batch size per device with 4 accumulation steps)
• Optimizer: AdamW with Mixed Precision (Native AMP)
• Epochs: 3
• LoRA Rank (r): 8

Training Results

The model showed consistent improvement over 3 epochs, with the final Validation Loss reaching 2.0083. The steady decrease in both Training and Validation loss indicates successful convergence without overfitting.

Framework Versions

• PEFT: 0.18.1
• Transformers: 5.2.0
• Pytorch: 2.10.0+cu128
• Datasets: 4.5.0