Coder_llama

Llama-3-3B CodeSearchNet Fine-tuned

This repository hosts a Llama 3 (3B) model fine-tuned on the CodeSearchNet dataset, which contains code in six programming languages.

📝 Model Details

• Base Model: Llama 3 (3B)
• Fine-tuning Dataset: CodeSearchNet
• Languages Covered: Python, Java, JavaScript, PHP, Ruby, Go
• Training Method: Supervised fine-tuning (SFT) with a contrastive loss objective for code search tasks
• Tokenization: Llama 3 tokenizer with additional tokens for code-specific keywords
• Frameworks Used: Hugging Face transformers, PyTorch, PEFT (for LoRA-based tuning)

📚 Dataset

The model is trained on the CodeSearchNet dataset, which contains:

• Function-level code snippets
• Paired natural language descriptions
• Multiple programming languages for multi-language search support

Dataset Sources

• CodeSearchNet Dataset
• Contains ~2M code snippets from open-source repositories

🚀 Training Setup

• Hardware: NVIDIA A100 GPUs
• Batch Size: 16
• Learning Rate: 2e-5 with cosine annealing
• Max Sequence Length: 512
• Fine-tuning Duration: 3 epochs

🔍 Intended Use

• Code Search: Retrieve relevant code snippets given a natural language query
• Code Completion: Provide context-aware code suggestions
• Code-to-Text Generation: Explain code functionality in natural language
• Multi-language Code Retrieval: Search across different programming languages