Orca-4B-Instruct

Dolphin-4B-Instruct-0409

Best price/performance 4B Instruct model

Optimized for maximum capability at minimal cost. Based on the strong Qwen3-4B architecture with aggressive efficiency tuning.

Why this version is better price/performance

→ Same 4B intelligence, significantly cheaper to run, better focused answers.

Technical specs

• Architecture: Qwen3ForCausalLM
• Parameters: 4.0B
• Layers: 36
• Hidden size: 2560
• Context: 16 384 tokens (optimized)
• Vocab size: 151 936
• Recommended dtype: bfloat16 or float16
• Recommended quant: Q4_K_M / AWQ / GPTQ for best speed/quality balance

Recommended generation settings (already in generation_config.json)

{
  "temperature": 0.5,
  "top_p": 0.85,
  "top_k": 30,
  "repetition_penalty": 1.05
}

These settings produce more precise, less repetitive and higher-quality answers while staying efficient.

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "./Dolphin-4B-Instruct-0409"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "user", "content": "Explain the difference between CPU and GPU in simple terms."}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][len(inputs.input_ids[0]):], skip_special_tokens=True))

Deployment tips for maximum cheapness

• vLLM / SGLang: --max-model-len 16384
• llama.cpp: use Q4_K_M or Q5_K_M GGUF
• Ollama / LM Studio: load the GGUF version if available
• Keep context ≤ 8k–12k in production for best speed/VRAM ratio

License

Apache 2.0

---

This package contains the fully tuned config + tokenizer.
Place the original Qwen3-4B safetensors weights next to the index file (or convert to GGUF) and you have a ready-to-run high price/performance 4B model.