qwen25coder-7b-p2

qwen25coder-7b-p2

Fine-tune of Qwen/Qwen2.5-Coder-7B (base): filtered OpenCodeInstruct SFT + scaffold self-distillation.

IMPORTANT — this model does not reliably stop on its own

It writes correct code first, then keeps generating (trained without a reliable end-of-turn token). How you stop it depends on how you run it.

Served behind an endpoint (TGI / vLLM / Inference Endpoints)

There is no StoppingCriteria hook over HTTP — you must pass stop sequences on every request, and cap max_tokens:

from openai import OpenAI

client = OpenAI(base_url="https://<your-endpoint>.endpoints.huggingface.cloud/v1/", api_key="hf_...")

resp = client.chat.completions.create(
    model="tgi",                       # vLLM: use the served model name
    messages=[{"role": "user", "content": "Write a Python function that ..."}],
    max_tokens=1024,                   # hard ceiling — it will use all of it otherwise
    temperature=0.2,
    stop=["\n```\n", "\n```", "<|im_end|>", "<|endoftext|>"],
)

eos_token_id is [151645, 151643] (<|im_end|>, <|endoftext|>) so the server halts on either if the model emits one — but do not rely on that alone, hence the stop list above.

Local transformers

Stop at the end of the first code block:

from transformers import StoppingCriteria, StoppingCriteriaList
class StopAfterCodeBlock(StoppingCriteria):
    def __init__(self, tok, n): self.tok, self.n = tok, n
    def __call__(self, ids, s, **k):
        t = self.tok.decode(ids[0][self.n:], skip_special_tokens=True)
        i = t.find("```"); nl = t.find("\n", i) if i>=0 else -1
        return i>=0 and nl>=0 and "```" in t[nl+1:]
# model.generate(**enc, max_new_tokens=1024,
#   stopping_criteria=StoppingCriteriaList([StopAfterCodeBlock(tok, enc.input_ids.shape[1])]))

Serving notes

• Prompt format: ChatML (<|im_start|>role\n...<|im_end|>). The chat template ships both inline in tokenizer_config.json (for TGI / vLLM / the HF inference toolkit) and as chat_template.jinja (for transformers 5.x).
• Precision: bf16, 15.2 GB of weights. Needs a >16 GB GPU (T4 is out). KV cache is ~57 KB/token (28 layers × 4 KV heads × 128 dim × 2 × 2 bytes), i.e. ~1.9 GB for a full 32k sequence — so L4 / A10G (24 GB) serves 32k at low concurrency, and L40S (48 GB) gives room for real batching.
• Context: 32768 tokens, RoPE theta 1e6.
• The config carries both the transformers 4.x keys (torch_dtype, top-level rope_theta) and the 5.x keys (dtype, rope_parameters), so it loads correctly on either. Do not drop the 4.x keys — every current serving stack reads those, and without rope_theta they silently fall back to 10000.0 (wrong RoPE base → degraded output).