Qwen-Fixed-Chat-Templates

Fixed jinja chat templates for Qwen 3.5 & 3.6 (v20) & Qwen 3.8 Flash (WIP)

<details open> <summary><b>Update History & Changelog (ADD q3.8 support)</b></summary>

WIP Qwen 3.8 Flash support added

</details> <details> <summary><b>Update History & Changelog (v20)</b></summary>

2026-06-05 Update (v20): The Architect Patch. A monumental structural overhaul targeting deep agentic loops and C++ inference engine compatibility. (1) Minja AST Flattening: Dramatically optimized Jinja nesting depths to resolve severe parsing bottlenecks that were dropping inference throughput by 80% on llama.cpp. (2) Minja Replace Bug Fix (Hotfix): Bypassed a severe C++ parsing bug in llama.cpp where using the replace filter at index 0 of a user prompt silently dropped the entire text payload. Inline thinking toggles now use split and join for robust stripping. (3) Auto-disable Thinking: Introduced auto_disable_thinking_with_tools kwarg (default false) that allows users to instantly shut off reasoning blocks during tool use. (4) Deep Agent Fallbacks: Resolved exceptions triggered by mid-conversation system prompts or loops lacking human user messages. (5) Payload Truncation: Implemented max_tool_arg_chars and max_tool_response_chars configurations to definitively stop context-window explosions from massive data returns. (Huge thanks to barubary / spiritbuun for their contributions to these C++ architecture optimizations!)

</details> <details> <summary><b>Update History & Changelog (v19)</b></summary>

2026-05-18 Update (v19): The Agentic Loop Cure. (1) Abolished "Empty Think" Poisoning: Rewrote the AST history rendering to completely remove the injection of empty <think>\n</think> blocks. This cures a severe in-context learning bias where the model assumed tools could only be called if it didn't think first, which was causing 80%+ of premature <|im_end|> turn aborts. (2) System Prompt Logic Trap Removed: Softened the absolute tool mandate in the <IMPORTANT> block and restored Universal Synthesis instructions. The model is now explicitly permitted to transition from </think> to a conversational answer without panicking. (3) True 100% KV Cache & Amnesia Fix: preserve_thinking now defaults to true. Past thoughts are retained chronologically, permanently curing "amnesia stalls" during multi-step tool loops while mathematically guaranteeing 100% KV Cache prefix matching out-of-the-box.

</details> <details> <summary><b>Update History & Changelog (v11-v18)</b></summary>

2026-05-16 Update (v18): Stability & Precision Patch. (1) Bulletproof False-Positive Detection: Shifted agentic error detection from broad substring matching to strict structural formats (e.g., "error":, Exception:, Traceback), completely curing false-positive retry loops when successful JSON returns simply contain the word "error" or "fail". (2) Legacy Engine Compatibility: Replaced loop.previtem with explicit array indexing, fixing AST crashes on older llama.cpp and minijinja builds that do not track loop state items. (3) True Whitespace Normalization: Fixed a bug where reasoning bypasses and hallucinated tag recovery stacked hidden multi-newlines (\n\n\n), strictly fulfilling the 100% KV Cache hit rate claim for all edge cases. (4) Code Cleanup: Removed dead conditional branches during XML tool parsing.

2026-05-15 Update (v17): Major architecture overhaul resolving edge cases in agentic tooling and KV Cache. (1) Unified Template: Consolidated Qwen 3.5 and Qwen 3.6 into a single chat_template.jinja file that handles all variants seamlessly. (2) Fixed "Mutually Exclusive" Stopping Bug: Changed the history-pruning logic from wiping the entire turn to safely array-slicing out just the raw tool tags (content.split('<tool_call>')[0]). This preserves the conversational text in the history, which cures the bug where the model would artificially abort its turn (output <|im_end|>) when it wanted to talk and use a tool simultaneously. (3) 100% KV Cache Hit Rate Restoration: Fully normalized internal whitespace logic (\n\n -> \n) around think blocks and tool calls to exactly match the model's native autoregressive generation spacing. This perfectly synchronizes the template's rendered history with the cached generated tokens, completely eliminating the severe cache invalidation and full-prompt re-processing issues present in v16.

2026-05-14 Update (v16): Four-part fix addressing community-reported regressions. (1) Native XML tool format: reverted from JSON back to the native <function=name> / <parameter=x> format the model was trained on, restoring full compatibility with vLLM's qwen3_coder parser and all inference engines that implement the Qwen tool protocol. (2) --reasoning off respected in error paths: when thinking is disabled (enable_thinking=false / --reasoning off), the error escalation directives are now injected as plain text without opening any <think> block, preventing degenerate prompts in no-reasoning sessions. (3) Smarter false-positive detection: short shell command results (starting with $ ) and search results with timing footers (Took X.Xs) are now correctly excluded from error detection, preventing tool-retry loops when commands succeed but their output happens to contain the word error. (4) consecutive_failures counter no longer resets on assistant messages, allowing Tier 2 escalation to actually fire across multi-turn tool retry chains.

2026-05-13 Update (v15): Three-part fix for agentic tool-loop failures. (1) Two-tier error escalation: replaced the brittle backwards-lookahead error detection with a fully forward-tracking last_tool_failed + consecutive_failures counter. On the first error the generation prompt is pre-seeded with a correction directive inside <think>; on the 2nd+ consecutive error the think block is bypassed and an out-of-band directive forces an immediate corrected action. (2) Length-gated detection: error signals are only read from short tool responses (< 500 chars), preventing false positives when reading code files containing error, exception, etc. in legitimate content. (3) Static system prompt: tool instructions are now fully unconditional, permanently eliminating the KV cache invalidation vector introduced in v14.

2026-05-12 Update (v14): Cured tool amnesia loops and post-tool overthinking friction! Implemented Smart Loop Preservation to dynamically scan subsequent tool returns for error markers and conditionally preserve historical reasoning context during active tool failures. Broadened the system instruction scope to define <think> as a dual-purpose planning or synthesis space, completely eliminating indecisiveness post-tool retrieval.

2026-05-11 Update (v13): Radical simplification and compatibility overhaul! Reverted tool schemas and assistant output formatting to standard JSON to natively fix downstream MCP parser crashes and C++ implicit enum coercion bugs. Removed the ns_scan history loop to permanently fix KV cache invalidation mid-conversation. Replaced global string replacement for hallucinated tags with a C++ safe, localized array-slicing method to prevent data-corruption on user code blocks.

2026-05-10 Update (v12): Fixed agent stalls, parameter data-loss, and hallucination bugs! Restored dynamic tool instructions and the <IMPORTANT> formatting reminder block to stop grammar parser crashes.

2026-05-10 Update (v11): Fixed agent looping and overthinking! Re-implemented preserve_thinking kwarg to properly strip reasoning blocks from history by default, and restored the reasoning bypass (<think>\n\n</think>\n\n).

</details>

This is a drop-in Jinja template that fixes rendering errors, KV cache invalidation, token waste, and fatal agentic stalling in the official Qwen chat templates.

It is tested to work across LM Studio, llama.cpp, vLLM, MLX, oMLX, and any engine that supports HuggingFace Jinja templates.

---

Fork variant: chat_template_v21.jinja (v24.0 "loopguard")

This repository ships two templates:

Rule of thumb: for long agentic sessions (coding harnesses, MCP toolchains, autonomous multi-step loops), use the fork. For plain chat or light tool use, either works.

<details open> <summary><b>What the fork adds over the original</b></summary>

1. Loop-guard: escalating anti-stall / anti-loop system

The original detects consecutive tool errors with a single narrow heuristic and injects one of two warnings. The fork replaces this with a layered, threshold-driven system (all thresholds overridable via chat_template_kwargs):

• Consecutive-failure escalation — 1 error: gentle nudge with an "expected failure" escape hatch (a failing test mid-fix is normal, not a mistake). ≥ fail_warn_after (2): warning whose wording distinguishes identical repeated errors ("change the approach itself") from differing errors ("you may be burning down causes — continue"). ≥ fail_halt_after (5): hard halt.
• Tool suspension on hard halt — a pre-scan pass walks the history before the system prompt is emitted. If the failure threshold is reached, the entire <tools> block is replaced with a suspension notice. A model that never sees tool definitions cannot produce valid tool-call syntax — this is the only reliable way to break a hard loop; injected warnings alone are not.
• Repeat-call detection — fingerprints each assistant turn by tool name + arguments over a two-turn window (catches A-B-A-B ping-pong alternation, not just straight repeats). Warns at repeat_warn_after (2), halts at repeat_halt_after (4). Mutating tools are exempt (re-running a build or test between fixes is a normal cycle).
• Identical-result detection — nudges after repeat_nudge_after (6) identical tool results, catching cosmetically-varied calls that keep returning the same answer (something signature matching can never see).
• Ping-pong nudge for mutating tools — after pingpong_nudge_after (4) identical call signatures even when results differ (timestamps, screenshots). This is the only branch that catches mutating tools stuck in a loop, and deliberately has no mutating-tool exemption.
• Primed generation prompts — on warn states the <think> block is pre-seeded with a diagnostic instruction and left open (think_on_tool_failure, default true); on halt states the think block is closed and a prose sentence-starter is emitted, making a tool call syntactically unnatural. The original instead silenced thinking after 2 failures — removing the model's only self-diagnosis tool.
• Full state reset on genuine user turns — a real user message resets all loop-guard state (the original only reset the failure counter, and only partially).

2. Tool classification (mutating_tools / read_tools)

Two configurable name lists drive smarter heuristics:

• Read tools (read, grep, fetch, …) are exempt from error-text heuristics — a file reader quoting error: from the file it read is not a failed call.
• Mutating tools (write, edit, bash, …) are exempt from identical-result repeat detection — idempotent writes legitimately return the same confirmation.
• Results are attributed to the tool that produced them via tool_call_id → name mapping, so classification works even with parallel calls.

3. Redesigned failure heuristics (v23.0)

The original used one check: response < 500 chars with an error keyword in the first 80 chars. The fork uses four tiers, kept in sync between the pre-scan and the main loop:

• Hard failures: unambiguous error prefixes (Traceback, error:, fatal:…) at the very start (non-read tools only).
• Harness errors: tool-harness phrases (invalid input for tool, unknown tool, json parsing failed…) in the head.
• Generic errors: a broad pattern list, but head-160 only instead of body-wide, never for read tools.
• Permission errors: access denied / requires elevation anywhere in a short body — these get dedicated guidance telling the model to stop retrying and hand the user the exact command to run as Administrator, plus a matching primed generation prompt.
• Shell-transcript ($ ) and timing-footer (took ) escapes prevent false positives on successful commands whose output mentions errors.

4. Security hardening

• User messages can no longer toggle thinking. The original honoured <|think_off|> / <|think_on|> from user content — any user (or injected content) could silently disable CoT. The fork only honours the toggles from system/developer messages and strips the literal tokens from user content without applying them (v23.0).
• Tool output is sanitized (sanitize_tool_tokens, default true, v24.0): literal think-toggle strings inside tool results (e.g. a fetched web page or a read template file containing <|think_off|>) are stripped before emission so they cannot be tokenized as real control tokens.

5. Harness & ecosystem integration

• Structural skill triggering (upstream v14/v15): when a tool named exactly skill exists, the first genuine user message gets a skill-selection checkpoint and later ones get a compact continuation checkpoint naming already-loaded skills (tracked from <skill_content name="…"> results). Renders nothing without a skill tool.
• Vanished-tool warning (warn_vanished_tools, default true): if a tool called earlier in the session is missing from the current tool list (MCP server died, toolset changed), a system note tells the model explicitly — otherwise it keeps calling the ghost tool from its own transcript.
• Current-date injection into the system prompt via strftime_now (guarded for engines that lack it).
• OpenAI-style wrapper unwrapping: {"type": "function", "function": {…}} tool definitions are unwrapped so the model sees the clean schema.
• Updated tool instructions: agentic guidance (keep working until the task is done, when to end the turn, where to source argument values), plus a fixed no-thinking variant — the original always instructed the model to use <think> blocks even when thinking was disabled.

6. Fidelity & compatibility fixes

• Untrimmed tool-output emission (upstream v7): leading/trailing whitespace in diffs, Makefile tabs, and exact compiler output are preserved; heuristics still run on the trimmed copy.
• Middle-out truncation for max_tool_response_chars: keeps head and tail halves — exit codes, summaries, and final errors live at the tail, which head-only truncation lost.
• <arguments_json> wrapper for string-form tool arguments so replayed turns are unambiguous, with the same truncation cap as mapping-form parameters.
• Prefix-cache fix for parallel tool calls: \n instead of \n\n between consecutive <tool_call> blocks (same root cause as the PR #42 trailing-newline bug).
• llama.cpp <__media__> marker stripping for --mmproj vision runs (system, assistant, and reasoning content).
• Unclosed-reasoning recovery: content embedding an inline <tool_call> with no closing think tag has everything before the call recovered as reasoning instead of replayed as visible text.

New template kwargs (fork only)

</details>

---

Why you need this

The official Qwen templates contain restrictions and Python-specific Jinja logic that break usage on many inference engines and agent frameworks.

Here are the critical issues this template fixes:

---

Quick install

Choose your environment and update the template:

LM Studio

1. Open your Qwen model in the right-side panel.
2. Scroll down to Prompt Template.
3. Replace the template with the contents of chat_template.jinja.
4. Click Save.

llama.cpp / koboldcpp

--jinja --chat-template-file chat_template.jinja

vLLM

Replace the "chat_template" string in your tokenizer_config.json with the raw file contents. Use the qwen3_coder tool parser:

--tool-call-parser qwen3_coder

oMLX

Overwrite chat_template.jinja in your local model directory. Load with --jinja. Remove any chat_template_kwargs overrides because the template handles everything internally.

---

Which file do I use?

Both Qwen 3.5 and Qwen 3.6 variants (including 35B, 32B, 27B, and 14B parameters) have been consolidated. You only need the single chat_template.jinja file at the root of the repository.

One-line versions (chat_template_oneline.txt) are pre-minified for engines that require a single-line template string.

---

The thinking toggle

You can control the model reasoning behavior. Insert <|think_on|> or <|think_off|> anywhere in your system or user prompt.

The template natively intercepts the tag, removes it from the final context so the model never sees it, and flips the reasoning mode instantly.

Fast answer, no reasoning:

System: You are a coding assistant. <|think_off|>
User: What's 2+2?

Deep reasoning:

System: You are a coding assistant. <|think_on|>
User: Implement a red-black tree in Rust.

(The tag syntax uses Qwen's control-token delimiters to guarantee it will never collide with legitimate text or file paths, unlike earlier community templates that used /think)

---

Token Saving: Stripping past thoughts

By default in v19, this template preserves all past <think> blocks in the chat history. This is intentional: it prevents the model from suffering "amnesia stalls" during complex, multi-step agentic loops, and it mathematically guarantees a 100% Prefix KV Cache hit rate on local inference engines.

However, if you are running constrained hardware and need to save context tokens, you can explicitly disable this feature in your engine's template kwargs to automatically strip past thoughts:

{
  "preserve_thinking": false
}

(Note: Setting this to false will naturally reduce your KV Cache hit rate during multi-turn chats, as the prompt string will dynamically mutate).

---

<details> <summary>Technical Details of the Critical Fixes</summary>

1. The "Empty Think" Poisoning & Logic Trap Cure (v19)

Previous versions attempted to save tokens by replacing past thoughts with empty <think>\n</think> blocks, combined with an absolute system prompt demanding a tool be called immediately after </think>. This created a toxic in-context learning pattern: the model associated empty thoughts with tools, and full thoughts with forbidden conversational text, causing an 80%+ premature <|im_end|> stalling rate. v19 abolishes empty think injection and rewrites the <IMPORTANT> directives to explicitly authorize conversational synthesis after a thought block.

2. KV Cache Safety & Autoregressive Normalization (v18/v19)

Llama.cpp and vLLM utilize prefix KV caching to speed up generation. Because v19 now preserves historical thoughts chronologically by default, the rendered history perfectly synchronizes with the cached generated tokens. Combined with strict single \n normalization at autoregressive boundaries, this achieves a 100% KV Cache hit rate in multi-turn loops.

3. Native XML Tool Call Format (v16)

The model was trained with the XML-based tool call format used by Qwen3-Coder:

<tool_call>
<function=tool_name>
<parameter=param_name>
value
</parameter>
</function>
</tool_call>

v16 restored this format natively, making it compatible with all parsers while bypassing the |items crash by using C++ safe key iteration (for args_name in tool_call.arguments).

4. Two-Tier Agentic Error Escalation (v15)

When a tool call fails validation repeatedly, the model can enter a degenerate reasoning spiral. This template leverages a two-tier escalation system driven by a forward-tracked consecutive_failures counter:

• Tier 1 (1st error): Generation prompt prefix changes to seed reasoning at a different token position, breaking the cached attractor state.
• Tier 2 (2nd+ consecutive errors): Think block bypassed entirely. An urgent out-of-band directive forces an immediate corrected action wrapped safely within the user tool_response block.

5. Smart False-Positive Detection (v18)

Instead of broad substring matching that triggers false retry-loops on successful database returns containing words like "error", v18 utilizes strict structural guards looking for Exception:, "error":, Traceback, and command not found, combined with length gates and shell-echo exclusions ($ ).

6. minijinja Compatibility Constraints (v18/v20)

Python-only Jinja2 features crash or misbehave on minijinja/minja (the C++ runtime used by llama.cpp, LM Studio, and MLX). All instances have been refactored for universal support:

• content | replace('<|think_on|>', '') -> content.split('<|think_on|>') | join('') (Fixes a severe bug where minja silently drops the entire text payload if the replaced string is found at index 0).
• \| items -> for key in mapping
• loop.previtem -> messages[loop.index0 - 1]
• map('string') -> join('|')
• \| first -> '$ ' in content

</details> <details> <summary>Comparison Matrix: Official vs Fixed vs Community</summary> </details>

---

Running the test suite

python3 scripts/test_v20.py

Tests cover: auto_disable_thinking_with_tools, payload truncation logic, parallel tool spacing, mid-conversation system rendering, deep agent loop fallback, XML tool format, <|think_off|> / <|think_on|> inline overrides, and all legacy v19 regression tests.

---

Authorship

License

Apache-2.0, inherited from Qwen.