alizee-coder-devstral-1-small

Alizee-Coder-Devstral-1-Small

A fine-tuned version of mistralai/Devstral-Small-2505 trained for code generation with explicit reasoning.

Model Description

This model is a LoRA adapter fine-tuned on the nvidia/OpenCodeReasoning dataset, which contains programming problems with detailed reasoning chains. The fine-tuning modifies the model to:

• Provide step-by-step reasoning before generating code
• Follow the Instruct format for problem-solving
• Generate verbose explanations alongside code solutions

Important Note on Benchmarks

This model was fine-tuned for reasoning-first code generation, which produces a different output format than standard code completion benchmarks expect. The benchmarks below measure raw code completion accuracy, where the base model (designed for direct code completion) outperforms this reasoning-focused variant.

For reasoning-based coding tasks (explaining solutions, teaching, complex algorithmic problems), this model may be more suitable. For direct code completion, the base Devstral-Small-2505 is recommended.

Evaluation Results

Benchmark Comparison: Base vs Fine-tuned

Detailed Results

HumanEval (164 problems)

MBPP (257 problems)

BigCodeBench (100 problems subset)

Why the Base Model Scores Higher

The base Devstral-Small-2505 is specifically designed for code completion tasks. This fine-tuned version was trained on OpenCodeReasoning which:

1. Encourages verbose reasoning before code generation
2. Uses an Instruct format that differs from completion-style benchmarks
3. May cause the model to generate explanatory text mixed with code

For pure code completion benchmarks, the base model's direct completion style is more aligned with the evaluation methodology.

Usage

Loading the Model

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# Load base model
base_model = "mistralai/Devstral-Small-2505"
adapter_model = "stmasson/alizee-coder-devstral-1-small"

tokenizer = AutoTokenizer.from_pretrained(adapter_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, adapter_model)
model.eval()

Inference (Instruct Format)

The model was trained with the following prompt format:

prompt = """<s>[INST] Solve this programming problem with detailed reasoning:

Write a function that checks if a number is prime.

[/INST]"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024, temperature=0.1)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

When to Use This Model

Use this model when:

• You need step-by-step explanations of the solution
• Teaching or educational contexts
• Complex algorithmic problems requiring detailed reasoning
• You want to understand the thought process behind the code

Use the base model when:

• You need direct code completion
• Speed is important
• Working with IDE integrations expecting code-only output
• Running code completion benchmarks

Training Details

Dataset

• Dataset: nvidia/OpenCodeReasoning (split_0)
• Training samples: 10,000
• Format: Instruct format with reasoning chains

Training Hyperparameters

Training Results

Limitations

• Lower scores on standard code completion benchmarks compared to base model
• Generates verbose reasoning which may not be desired for all use cases
• Best suited for educational/explanatory contexts rather than production code completion
• Optimized for Python code generation

Framework Versions

• PEFT: 0.18.0
• Transformers: 4.57.3
• PyTorch: 2.9.1+cu128
• Datasets: 4.4.2
• Tokenizers: 0.22.1

Citation

If you use this model, please cite:

@misc{alizee-coder-devstral,
  author = {stmasson},
  title = {Alizee-Coder-Devstral-1-Small: Code Generation with Reasoning},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/stmasson/alizee-coder-devstral-1-small}
}