typhoon2-qwen2.5-7b-instruct-GGUF

Typhoon2-Qwen2.5-7B-Instruct — GGUF

GGUF quantizations of scb10x/typhoon2-qwen2.5-7b-instruct, a Thai 🇹🇭 / English instruction-tuned LLM (7B, based on Qwen2.5-7B) by SCB 10X.

Quantized to GGUF with llama.cpp for fast local inference on CPU / modest hardware (Ollama, LM Studio, llama.cpp, Jan, koboldcpp).

Modification notice (Apache-2.0): these files are a GGUF quantization of the original model — no weights were retrained or fine-tuned. Original model © SCB 10X, released under Apache-2.0. This repository preserves that license.

Files

K-quants are used (best for CPU inference). For a 7B on ~16 GB RAM, Q4_K_M is the sweet spot.

Usage

Ollama

ollama run hf.co/CaldarianX/typhoon2-qwen2.5-7b-instruct-GGUF:Q4_K_M

llama.cpp

# chat
./llama-cli -m typhoon2-qwen2.5-7b-instruct-Q4_K_M.gguf -cnv -c 8192
# OpenAI-compatible server (use with Aider, Open WebUI, scripts, etc.)
./llama-server -m typhoon2-qwen2.5-7b-instruct-Q4_K_M.gguf -c 8192

LM Studio

Search this repo, download the Q4_K_M file, load, chat.

Prompt format

Qwen2.5 ChatML:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
สวัสดีครับ<|im_end|>
<|im_start|>assistant

Credits & License

• Original model: scb10x/typhoon2-qwen2.5-7b-instruct — © SCB 10X
• Paper: Typhoon 2 (arXiv:2412.13702)
• License: Apache-2.0 (inherited from the base model)
• Quantization: llama.cpp · uploaded by CaldarianX