CODE-1

CODE-1

This model is a fine-tuned version of gpt2 on huggingface-course/codeparrot-ds. It achieves the following results on the evaluation set:

• Loss: 1.0612

Model description

A fine-tuned version of gpt-2 designed to autocomplete Python code requests left in comments. Based on the HF LLM course (training LMs from scratch)

Intended uses & limitations

To be used for Python code autocomplete.

As an example of the model output:

 \# create some data
 x = np.random.randn(100)
 y = np.random.randn(100)
 
 \# create scatter plot with x, y

The current version of the model added below this comment:

 fig = plt.figure()
 ax = fig.add_subplot(111)

 scatter1 = Scatter(x, y)
 scatter2 = Scatter(x, y)
 ax.scatter(x, y, s=5)

 \# create hexbin plot with default gridsize of 10
 p = plt.Rectangle((0, 0), 1, 1, fc=hexbin_with_ecdf(5, 10, 20, 20))
 ax.0, 0.0, 0.0, zorder=1, fc=0.0, lw-0.1)
 ax=0.25)
 
 ax.set_axis('off')
 ax.set_axis_axisbelow(p.set_coloraxisbelow)
 ax.set_frameon(axisbelow)
 
 _frame_frameon(gca(), set_savefig(gca(), set_gca(), fig.set_gca))

This model is purely experimental in nature for learning purpose. Beyond a few lines, code may generate that is unecessary.

Training and evaluation data

Trained on the train/test split provided by huggingface-course/codeparrot-ds.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 1000
• num_epochs: 1
• mixed_precision_training: Native AMP

Training results

Carbon Emissions

For transparency, experiments were conducted using private infrastructure, which has a carbon efficiency of 0.035 kgCO₂eq/kWh based on Ontario's 2022 energy profile. A cumulative of 22 hours of computation was performed on hardware of type RTX 3080 (TDP of 320 W).

Total emissions are estimated to be 0.25 kgCO₂eq.

Estimations were conducted using the MachineLearning Impact calculator
Lacoste, Alexandre and Luccioni, Alexandra and Schmidt, Victor and Dandres, Thomas, Quantifying the Carbon Emissions of Machine Learning, arXiv preprint arXiv:1910.09700, 2019

Framework versions

• Transformers 4.56.0
• Pytorch 2.5.1+cu121
• Datasets 4.0.0
• Tokenizers 0.22.0