arnavj007_-_gemma-js-instruct-finetune-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

gemma-js-instruct-finetune - GGUF

• Model creator: https://huggingface.co/arnavj007/
• Original model: https://huggingface.co/arnavj007/gemma-js-instruct-finetune/

Original model description:

library_name: transformers tags: [qlora, peft, fine-tuning, javascript, causal-lm]

Model Card for gemma-js-instruct-finetune

Model Details

Model Description

This is the model card for gemma-js-instruct-finetune, a fine-tuned version of the gemma-2b-it model. This fine-tuned model was trained to improve the performance of generating long-form, structured responses to JavaScript-related instructional tasks. The fine-tuning process used the QLoRA (Quantized Low-Rank Adaptation) method, enabling efficient parameter tuning on limited hardware resources.

• Developed by: Arnav Jain and collaborators
• Shared by: Arnav Jain
• Model type: Decoder-only causal language model
• Language(s) (NLP): English
• License: Apache 2.0
• Finetuned from model: gemma-2b-it

Model Sources

• Repository: gemma-js-instruct-finetune
• Dataset: Evol-Instruct-JS-Code-500-v1
• Demo: Weights & Biases Run

Uses

Direct Use

The model can be directly used for generating solutions to JavaScript programming tasks, creating instructional code snippets, and answering technical questions related to JavaScript programming.

Downstream Use

This model can be further fine-tuned for specific programming domains, other languages, or instructional content generation tasks.

Out-of-Scope Use

This model is not suitable for:

• Non-technical, general-purpose text generation
• Applications requiring real-time interaction with external systems
• Generating solutions for non-JavaScript programming tasks without additional fine-tuning

Bias, Risks, and Limitations

Recommendations

• Users should validate generated code for correctness and security.
• Be cautious of potential biases or inaccuracies in the dataset that could propagate into model outputs.
• Avoid using the model for sensitive or critical applications without thorough testing.

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("arnavj007/gemma-js-instruct-finetune")
model = AutoModelForCausalLM.from_pretrained("arnavj007/gemma-js-instruct-finetune")

def get_completion(query: str):
    prompt = f"<start_of_turn>user {query}<end_of_turn>\n<start_of_turn>model"
    inputs = tokenizer(prompt, return_tensors="pt")
    outputs = model.generate(**inputs, max_new_tokens=1000)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

response = get_completion("Create a function in JavaScript to calculate the factorial of a number.")
print(response)

Training Details

Training Data

The training dataset consisted of 500 JavaScript instructions paired with relevant outputs. These instructions focused on tasks like code snippets, algorithm implementations, and error-handling scenarios.

Dataset: Evol-Instruct-JS-Code-500-v1

Training Procedure

Preprocessing

• Instructions and outputs were formatted using a standardized prompt-response template.
• Data was tokenized using the Hugging Face tokenizer for gemma-2b-it.

Training Hyperparameters

• Training regime: QLoRA (Quantized Low-Rank Adaptation)
• Batch size: 1 per device
• Gradient accumulation steps: 4
• Learning rate: 2e-4
• Training steps: 100
• Optimizer: Paged AdamW (8-bit)

Speeds, Sizes, Times

• Training runtime: ~1435 seconds
• Trainable parameters: 3% of the model (~78M)

Evaluation

Testing Data, Factors & Metrics

Testing Data

The test dataset consisted of 100 JavaScript instructions held out from the training set.

Metrics

• Quality of generated code snippets
• Ability to handle complex prompts with multiple sub-tasks

Results

The fine-tuned model demonstrated significant improvement in handling long prompts and generating structured code. It provided complete solutions for tasks like API creation with advanced features (e.g., caching, error handling).

Summary

Fine-tuning with QLoRA enabled robust performance improvements, making the model capable of generating detailed instructional responses.

Environmental Impact

• Hardware Type: NVIDIA Tesla T4 GPU (free-tier Colab)
• Hours used: ~0.4 hours
• Carbon Emitted: Minimal (estimated using ML Impact Calculator)

Technical Specifications

Model Architecture and Objective

The model uses a decoder-only architecture optimized for causal language modeling tasks.

Compute Infrastructure

• Hardware: NVIDIA Tesla T4
• Software:
  • Transformers: 4.38.2
  • PEFT: 0.8.2
  • Accelerate: 0.27.1
  • BitsAndBytes: 0.42.0

Citation

BibTeX:

@misc{Jain2024gemmajs,
  author = {Arnav Jain and collaborators},
  title = {gemma-js-instruct-finetune},
  year = {2024},
  howpublished = {\url{https://huggingface.co/arnavj007/gemma-js-instruct-finetune}}
}

More Information

For questions or feedback, contact Arnav Jain.