MadMarx37_-_deepseek-coder-1.3b-python-peft-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

deepseek-coder-1.3b-python-peft - GGUF

• Model creator: https://huggingface.co/MadMarx37/
• Original model: https://huggingface.co/MadMarx37/deepseek-coder-1.3b-python-peft/

Original model description:

library_name: transformers tags:

• code license: mit datasets:
• ArtifactAI/arxiv_python_research_code language:
• en pipeline_tag: text-generation

---

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

A parameter-efficient finetune (using LoRA) of DeepSeek Coder 1.3B finetuned on Python code.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

A finetune of DeepSeek Coder 1.3B finetuned on 1000 examples of Python code from the ArtifactAI/arxiv_python_research_code dataset.

• Model type: Text Generation
• Language(s) (NLP): English, Python
• Finetuned from model: deepseek-ai/deepseek-coder-1.3b-base

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/kevin-v96/python-codecomplete-lm

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

To generate Python code

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import pipeline

model_name = "MadMarx37/deepseek-coder-1.3b-python-peft"

def generate_output(input):
    # Run text generation pipeline with our next model
    pipe = pipeline(task="text-generation", model=model_name, tokenizer=model_name, max_length=max_length)
    result = pipe(input)
    print(result[0]['generated_text'])

Training Details

Training Hyperparameters

• Training regime: fp16 mixed-precision with original model loaded in 4bits with bitsandbytes <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
• learning_rate = 2e-3
• lr_scheduler_type = 'cosine_with_restarts'
• max_grad_norm = 0.001
• weight_decay = 0.001
• num_train_epochs = 15
• eval_strategy = "steps"
• eval_steps = 25

Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

1.3B parameters. Training time of ~2 hours on an RTX3080.

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

https://huggingface.co/datasets/ArtifactAI/arxiv_python_research_code

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

Standard training and eval loss from the HF SFTTrainer.

Results

Training Loss: 0.074100 Validation Loss: 0.022271

Summary

The training had some unstability in the gradient norms, but the overall trend in both training and validation loss were downward, and validation loss has almost plateaud, which is ideally where we want our model. The code generation on the same prompts that we tested the original model on also seem better with the finetuned model. A good way to make the model better, if we wanted to increase the finetuning data, would be to also increase the epochs.

The training run metrics can be seen here: https://wandb.ai/kevinv3796/python-autocomplete-deepseek/reports/Supervised-Finetuning-run-for-DeepSeek-Coder-1-3B-on-Python-Code--Vmlldzo3NzQ4NjY0?accessToken=bo0rlzp0yj9vxf1xe3fybfv6rbgl97w5kkab478t8f5unbwltdczy63ba9o9kwjp