qwen2-7b-code

Qwen2-7B-Instruct Fine-tuned for Code Improvement

This repository contains a fine-tuned version of Qwen2-7B-Instruct specifically optimized for analyzing and fixing buggy code. The model was fine-tuned using the Parameter-Efficient Fine-Tuning (PEFT) approach with LoRA on the Python subset of the CommitPackFT dataset.

Model Details

• Base Model: Qwen/Qwen2-7B-Instruct
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Task: Code improvement and bug fixing
• Dataset: bigcode/commitpackft (Python subset)
• Training Format: Instruction-Response pairs

LoRA Configuration

The model was fine-tuned using the following LoRA hyperparameters:

lora_config = LoraConfig(
    r=16,              # Rank
    lora_alpha=32,     
    target_modules=["q_proj", "v_proj"],  # LoRA on attention layers
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

Training Details

• Training Data: 5% of the Python subset of CommitPackFT
• Batch Size: 2 per device with gradient accumulation steps of 8
• Learning Rate: 2e-4
• Epochs: 3
• Precision: Mixed precision (fp16)
• Hardware: 4-bit quantization for memory efficiency

Usage

Loading the Model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model and tokenizer
model_name = "Qwen/Qwen2-7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token

# Load model with adapter weights
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    load_in_4bit=True,
    trust_remote_code=True
)

# Load LoRA adapter
adapter_path = "PATH_TO_ADAPTER"  # Update with your model path
model = PeftModel.from_pretrained(model, adapter_path)

Example Inference

def improve_code(code, max_new_tokens=200):
    # Format prompt in the same way as training
    prompt = f"### Instruction:\nFix the following buggy code:\n{code}\n\n### Response:\n"
    
    # Tokenize
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    # Generate
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.7,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
            repetition_penalty=1.1
        )
    
    # Decode only the generated part
    generated_text = tokenizer.decode(
        outputs[0][inputs.input_ids.shape[1]:], 
        skip_special_tokens=True
    )
    
    return generated_text

# Example usage
buggy_code = """
def calculate_average(numbers):
    return sum(numbers) / len(numbers)
"""

improved_code = improve_code(buggy_code)
print(improved_code)

Limitations

• The model was fine-tuned on a small subset (5%) of the Python data in CommitPackFT
• Performance might be limited on non-Python programming languages
• Code fixes are based on patterns seen in the training data and may not address all bugs correctly
• As with all code generation models, human review is essential for any generated code

Fine-tuning Process

The model was fine-tuned using the Hugging Face Transformers library with the following process:

1. Load the Qwen2-7B-Instruct model with 4-bit quantization for memory efficiency
2. Apply LoRA for parameter-efficient fine-tuning, targeting only query and value projection matrices
3. Format training data as instruction-response pairs
4. Train for 3 epochs with a batch size of 16 (2 per device × 8 gradient accumulation steps)
5. Save the LoRA adapter weights for efficient deployment

Citation

If you use this model in your research, please cite:

@misc{qwen2-7b-code-improvement,
  author = {Tharun Kumar},
  title = {Qwen2-7B-Instruct Fine-tuned for Code Improvement},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/Tharun007/qwen2-7b-code}}
}

License

This model adapter is subject to the license of the original Qwen2-7B-Instruct model. Please refer to the Qwen2-7B-Instruct model card for license details.

Contact

GitHub