gptesla-small

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: Shi Hao Ng, IB DP Student, Marlborough College Malaysia
• Model type: Transformer. Decoder only
• Language(s) (NLP): Python, HTML, etc.
• License: mit
• Finetuned from model [optional]: No

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/Ice-Citron/GPTesla

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

• You input half finished python code, and it will generate python code.

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

• Some level of fine tuning is likely needed or preferred. However I won't be working on this.

[More Information Needed]

Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

• This can easily be used for IDEs. Not ideal though as it's rarely correct in its answer. Likely mainly attributed to how it's a pretty small model after all.
• Even then, I'm already struggling to train it with my 4x Nvidia A100 PCIe 80GB, taking 15 hours!

How to Get Started with the Model

Use the code below to get started with the model.

• just follow the instructions on huggingface "use this model". Should work. If not try and contact me.

[More Information Needed]

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

[More Information Needed]

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

• Training regime: [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

• 111 million parameter. FP16, 444 Megabytes.
• Pretty fast and lightweight model when using T4 GPU.

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

• https://huggingface.co/datasets/shng2025/gptesla-valid

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

• https://huggingface.co/datasets/shng2025/gptesla-train

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

• Perhaps not accurate because I'm expecting 1 to 1 representation for code. As in reality there's many way of coding to reach the same logic. And a precise way of coding is not required.

Results

• 1.1 loss/train in the end. Model converged after 150,000 steps.
• weights and biases file: https://wandb.ai/marlborough-college-malaysia/gptesla-small/runs/m9sqzqo3?nw=nwusershng2025

Summary

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: 4x Nvidia A100 PCIe + 96x AMD CPU
• Hours used: 15 hours
• Cloud Provider: Azure
• Compute Region: Unclear
• Carbon Emitted: [More Information Needed]

Model Architecture and Objective

• Based on codeparrot. And using GPT2's architecture but it's weights are random initialised.

Compute Infrastructure

• NVMe Link
• 4x Nvidia A100 PCIe
• 96x AMD CPU from Azure
• 900 GB RAM

Hardware

• NVMe Link
• 4x Nvidia A100 PCIe
• 96x AMD CPU from Azure
• 900 GB RAM

Software

• Python 3.10.14
• Latest version of Pytorch, transformer, wandb libraries, etc. installed. Refer to github repo for versions
• Accelerate

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

• codeparrot used