NextStep-Coder-MoE

NextStep-Coder-MoE

A high-performance tiny coding LLM with Interleaved Thinking capability for advanced reasoning and agentic workflows.

Model Description

NextStep-Coder-MoE is a LoRA fine-tuned model based on Qwen2.5-Coder-1.5B, optimized for:

• Chain-of-Thought reasoning with <think>...</think> tags
• Multi-step coding tasks
• Agentic workflows (plan → act → reflect)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load model
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-1.5B")
model = PeftModel.from_pretrained(base_model, "OsamaBinLikhon/NextStep-Coder-MoE")
tokenizer = AutoTokenizer.from_pretrained("OsamaBinLikhon/NextStep-Coder-MoE")

# Generate
prompt = "Write a Python function to check if a number is prime.\n<think>"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0]))

Training Details

Interleaved Thinking Format

The model uses <think> tags to show reasoning:

User: Write a binary search function.

Model: <think>
I need to implement binary search on a sorted array.
Key steps: find middle, compare, narrow search space.
Edge case: empty array returns -1.
</think>

def binary_search(arr, target):
    left, right = 0, len(arr) - 1
    while left <= right:
        mid = (left + right) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            left = mid + 1
        else:
            right = mid - 1
    return -1

Intended Use

• Code generation with step-by-step reasoning
• Debugging and code review
• Algorithm design and explanation
• Educational coding assistance

Limitations

• Best for Python, may vary for other languages
• Requires <think> tag retention in conversation history
• 2048 token context limit

Author

OsamaBinLikhon

License

Apache 2.0

Framework versions

• PEFT 0.18.0