FriendliAI-broken-model-fixed

Changes Q1-a

1. Added 'chat_template to 'tokenizer-config.json'

• Problem: The original 'tokenizer_config.json' was missing the 'chat_template' field.
• Impact: Without a chat template, werving frameworks cannot convert(vLLM, TGI, SGLang).
• Fix: Added the standard Qwen3 chat template from the official 'Qwen/Qwen3-8B' repository.

2. Corrected 'base model' in 'README.md'

• Problem: 'base_model' was set to 'meta-llama/Meta-Llama-3.18B'.
• Evidence:
  • 'config.json' specifies '"model_type": "qwen3" and '"architectures": ["Qwen3ForCausalLM"]'
  • 'vocab_size' is 151936(Qwen3), not 128256(Llama-3.1)
  • 'num_hidden_layers' is 36(Qwen3-8B), not 32(Llama-3.1)
  • 'intermediate_size' is 12288(Qwen3-8B), not 14336(Llama-3.1-8B)
  • Tokenizer uses Qwen-specific special tokens('<|im_start|>', <|im_end|>, etc)
  • Total weight size(16.4GB) matches official Qwen3-8B
• Fix: Changed 'base_model' to 'Qwen/Qwen3-8B'

---

Changes Q1-b

Why 'reasoning_effort' has no effect

The 'reasoning_effort' parameter is designed to control how much "thinking" a model does before responding. However, this Qwen3-8B model only supports a binary toggle ('enable_thinking=True/False') via its chat template, not a continuous effort scale.

The 'reasoning_effort' parameter is an API-level concept that requires explicit support in both the serving engine and the model. Qwen3-8B was not trained to modulate reasoning depth based on an effort parameter — it either thinks (generates <think>...</think> blocks) or it doesn't.

Requirements to make it functional

1. Serving Engine Support: The inference API must parse the 'reasoning_effort' parameter and translate it into actionable behavior. Currently, most engines pass it through without interpretation for Qwen3 models.

2. Mapping to existing mechanisms: The engine could map 'reasoning_effort' to Qwen3's existing controls:

   • 'low' → 'enable_thinking=False'
   • 'medium' → 'enable_thinking=True' + limit thinking tokens
   • 'high' → 'enable_thinking=True' + generous thinking budget

3. Thinking budget control: The serving engine needs to implement a token budget for the <think> block

   • stopping generation of thinking tokens after a threshold and forcing the model to transition to the actual response. This requires modifying the generation loop to detect </think> or enforce a token limit within the thinking block.

4. Model level training: For 'reasoning_effort' to be truly meaningful (not just truncating thoughts), the model would need to be post-trained with varying levels of chain-of-thought depth:

   • Fine-tune on examples with short, medium, and long reasoning traces, each labeled with the corresponding effort level
   • Train the model to condition its reasoning depth on an effort token or system prompt instruction
   • This is how models like Claude natively support reasoning effort — it's baked into the training, not bolted on at serving time