DevRouter-1.5B-GGUF

DevRouter-1.5B

A tiny, fast router that turns a raw developer prompt into a single structured JSON decision.

DevRouter-1.5B reads a raw coding prompt and returns one JSON object that (1) rewrites the prompt into a cleaner version, (2) classifies its intent, complexity, and a suggested route (which model tier to send it to), and (3) flags missing context the developer should have included. It is meant to sit in front of your expensive models and make a cheap, deterministic triage call in ~1–3 seconds on a single consumer GPU.

It is a fine-tune of Qwen2.5-Coder-1.5B-Instruct (Apache 2.0), distilled on a dataset of developer prompts labelled by a stronger teacher model.

Output schema

Every response is a single JSON object with exactly these five fields:

Example

Input (user message):

My Flask app 500s on POST /upload with RequestEntityTooLarge, how do I fix it?

Output:

{
  "rewrite": "I'm encountering a 500 Internal Server Error on my Flask app when handling POST /upload. The error is 'RequestEntityTooLarge'. How can I resolve this issue?",
  "intent": "debug",
  "complexity": "low",
  "route": "small_local",
  "missing": ["Flask version", "Python version"]
}

Quick start

The router system prompt is baked into the model's chat template, so you do not need to supply a system prompt — just send the raw developer prompt as the user message.

Ollama

# from the -GGUF repo (Modelfile + DevRouter-1.5B-Q8_0.gguf)
ollama create devrouter -f Modelfile
ollama run devrouter "refactor this giant function into smaller ones"

llama.cpp

llama-server -m DevRouter-1.5B-Q8_0.gguf -c 8192 -ngl 99 --parallel 1
# then POST to /v1/chat/completions with just a user message, temperature 0

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("aipster/DevRouter-1.5B")
model = AutoModelForCausalLM.from_pretrained("aipster/DevRouter-1.5B")
msgs = [{"role": "user", "content": "write a FastAPI POST /items endpoint with a Pydantic model"}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inputs, max_new_tokens=1408, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Use greedy decoding (temperature=0) for stable, parseable JSON.

Evaluation

Evaluated on a held-out, intent-stratified validation split (val, in-distribution) and an out-of-distribution split (holdout, no GitHub-issue sources). Metrics: rate of valid JSON (strict schema parse) and accuracy of intent / route / complexity against the teacher labels.

Per-intent accuracy (Q8_0, val):

Performance

Single RTX 3090, Q8_0 GGUF via llama.cpp (single stream):

• Generation: ~280 tokens/s (~3.6 ms/token, constant across output lengths)
• Prompt eval (prefill): ~10,000–13,000 tokens/s
• Latency per routing call: ~1–3 s (up to ~5 s for the longest outputs)

Throughput scales further with batching/concurrency.

Limitations

• No PII detection. An earlier schema included a PII flag; it was removed from v1.1 because the training data had too few PII-positive examples (~2.4%) to learn it reliably. Do not use this model for privacy/safety filtering. (Planned for a future data-focused release.)
• Weaker on rare intents. review and documentation are under-represented and noisier in the training data, and accuracy on them is lower.
• Source bias / OOD gap. Training prompts skew heavily toward GitHub-issue-style text, so intent accuracy drops ~10 points on out-of-distribution prompts. Treat route/complexity as advisory.
• Quantization sensitivity. This is a small model doing strict structured output. Q6_K and below break the JSON (validity collapses); ship Q8_0 or F16 only. Always validate a quant before relying on it.

Training

• Base: Qwen2.5-Coder-1.5B-Instruct (Apache 2.0)
• Method: QLoRA (4-bit), rank 16, 2 epochs, effective batch 16, train_on_responses_only
• Data: ~2.6k developer prompts, each labelled by a stronger teacher model into the 5-field schema, filtered by a judge model and capped per intent. Distillation sources permit training and open release of derived weights.

Intended use

Pre-routing / triage of developer prompts in an LLM application: rewriting, intent/complexity classification, and model-tier selection. Not intended for safety filtering, PII detection, or as a general assistant.

License

Apache 2.0, inherited from the base model. You are free to use, modify, and redistribute, including commercially.