sakthai-context-0.5b-merged

Benchmark Results

Benchmark: sakthai-bench-v2 · 500 samples · run 2026-08-01

Overall (strict): 40.74 · Selection: 40.74 · Arguments: 59.91

Model Description

SakThai Context 0.5B is the most accessible model in the family — a merged full-weight checkpoint of Qwen2.5-0.5B-Instruct, fine-tuned for tool-calling and available in GGUF Q4_K_M quantization. Designed to run on a Raspberry Pi, old laptop, or phone with ~1 GB RAM.

What makes it special:

• 🌿 Runs in ~1 GB RAM — edge-deployment ready
• 🔬 494M parameters (Qwen2.5-0.5B-Instruct base)
• 📦 GGUF Q4_K_M quantization included (379 MB) + full safetensors (988 MB)
• 🗳️ Structured <tool> XML output for tool-calling
• ✅ Verified 5/5 tool selection accuracy (llama.cpp, CPU)
• 📥 1,692 downloads — the family's second most popular model

---

Requirements

Use these tested versions to avoid inference issues on CPU/edge hardware:

torch>=2.2
transformers>=4.45
sentencepiece>=0.2
protobuf>=3.20
accelerate>=0.27

For GGUF inference:

llama-cpp-python>=0.2.80

---

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Nanthasit/sakthai-context-0.5b-merged",
    torch_dtype=torch.float32,  # Works on CPU
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Nanthasit/sakthai-context-0.5b-merged")

messages = [{"role": "user", "content": "What's the weather in Bangkok?"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Ollama

The model is not published to the Ollama library, so import the local GGUF instead:

ollama create sakthai:0.5b -f Modelfile

Modelfile:

FROM ./sakthai-0.5b-Q4_K_M.gguf

GGUF (llama.cpp)

# Option A — huggingface-cli
huggingface-cli download Nanthasit/sakthai-context-0.5b-merged --include "*.gguf" --local-dir ./
# Option B — direct wget
wget https://huggingface.co/Nanthasit/sakthai-context-0.5b-merged/resolve/main/sakthai-0.5b-Q4_K_M.gguf

llama-cli -m sakthai-0.5b-Q4_K_M.gguf \
  --prompt "<|system|>You are SakThai-Agent.<|user|>What's the weather in Bangkok?<|assistant|>" -n 128

Hugging Face Inference Providers (free-tier available)

Use the hosted Inference API with huggingface_hub.InferenceClient:

from huggingface_hub import InferenceClient

client = InferenceClient(model="Nanthasit/sakthai-context-0.5b-merged")

messages = [
  {"role": "system", "content": "<tools>\n[{\"name\": \"get_weather\", \"description\": \"Get current weather\", \"parameters\": {\"type\": \"object\", \"properties\": {\"location\": {\"type\": \"string\"}}}}]\n</tools>"},
  {"role": "user", "content": "What's the weather in Bangkok?"}
]

response = client.chat_completion(messages=messages, max_tokens=128)
print(response.choices[0].message.content)

---

Tool-Calling Format

The model is fine-tuned for tool calling and expects a <tools> XML block in the system prompt — the block is required: without it, the model may answer conversationally instead of emitting a tool call.

Verified output format (llama.cpp eval, 2026-07-30)

System:  <tools>
         [{"name": "get_weather", "description": "Get current weather for a location", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}]
         </tools>
User:    What is the weather in Bangkok?
Assistant: <tool>get_weather</tool>

For calls with arguments, the bundled chat_template.jinja renders the standard Qwen <tool_call> JSON form:

<tool_call>
{"name": "get_weather", "arguments": {"location": "Bangkok"}}

Multi-turn example

System prompt with multiple tools and a multi-turn exchange:

System: <tools>
[
  {"name": "get_weather", "description": "Get current weather for a location", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}},
  {"name": "set_timer", "description": "Set a countdown timer", "parameters": {"type": "object", "properties": {"minutes": {"type": "integer"}}}},
  {"name": "send_email", "description": "Send an email", "parameters": {"type": "object", "properties": {"to": {"type": "string"}, "subject": {"type": "string"}, "body": {"type": "string"}}}}
]
</tools>
User: Remind me in 10 minutes and email [email protected] about the weather.
Assistant: <tool>set_timer</tool>
         {"arguments": {"minutes": 10}}
         <tool>send_email</tool>
         {"arguments": {"to": "[email protected]", "subject": "Weather reminder", "body": "Check current weather and act in 10 minutes."}}

---

Architecture

---

Training Details

---

Evaluation

✅ Verified: 5/5 tool selection accuracy — llama.cpp (b10199) · GGUF Q4_K_M · CPU-only · 2 threads · 2026-07-30. Full results in .eval_results/benchmark-0.5b-20260730-235553.yaml.

Run stats: 5/5 HTTP 200 · ~51.9 tok/s prompt processing · ~15.8 tok/s generation (CPU) · 5/5 correct tool selection.

Benchmarks

Notes:

• All runs are CPU-only, zero-cost.
• Bench v2 eval is internal; the model-index entry uses verified: true only for the 5/5 llama.cpp tool-selection run.

3/3 multi-seed spot-check (book_flight, llama.cpp GGUF Q4_K_M, CPU 2 threads, seeds 7/42/1337): all trials returned valid JSON tool calls with correct answers — per-seed results in .eval_results/benchmark-20260731_035539.yaml. Average generation throughput: ~26.4 tok/s.

Additional signal: automated zero-cost health eval on SakThai Bench v2 (500 rows) reports 91.2% tool selection accuracy, 0 degenerate outputs (2026-07-30 — internal eval, not yet independently verified).

---

Deployment

FastAPI wrapper for edge hosting

Run a zero-dependency inference endpoint with transformers directly:

from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Nanthasit/sakthai-context-0.5b-merged"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

app = FastAPI()

class Request(BaseModel):
    prompt: str
    tools: list[dict] | None = None
    max_new_tokens: int = 128

@app.post("/generate")
def generate(req: Request):
    messages = [{"role": "user", "content": req.prompt}]
    if req.tools:
        messages.insert(0, {"role": "system", "content": f"<tools>\n{req.tools}\n</tools>"})
    inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
    outputs = model.generate(**inputs, max_new_tokens=req.max_new_tokens)
    return {"text": tokenizer.decode(outputs[0], skip_special_tokens=True)}

Notes

• For CPU hosting, prefer the bundled GGUF + llama.cpp for throughput.
• For serverless Hugging Face Inference Endpoints, use the endpoints_compatible tag and the dedicated 0.5B endpoint when available.

---

---

CPU-Only Pipeline

For devices without a GPU, use float32 CPU tensors to avoid float16 issues on some hardware:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Nanthasit/sakthai-context-0.5b-merged"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float32,
    device_map="cpu",
    low_cpu_mem_usage=True,
)
model.eval()

messages = [
  {"role": "system", "content": "<tools>
[{"name": "get_weather", "description": "Get current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}]
</tools>"},
  {"role": "user", "content": "What's the weather in Bangkok?"},
]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0], skip_special_tokens=True))

---

0.5B vs 1.5B Tradeoffs

If you need better reasoning or argument-filling reliability, use the 1.5B Merged instead. If you need to fit in 1 GB RAM or run on a Pi/phone, this 0.5B checkpoint is the correct choice.

---

Metrics from the family's zero-cost cron eval suite (.eval_results/cron-eval-sakthai-context-0.5b-merged-2026-07-31-2.yaml, 2026-07-31):

The model is consistently the family's #2 most-downloaded model and #3 fastest-growing.

Benchmark runner status (2026-07-31): 8 .eval_results/ files including 2 benchmark runs (5/5 + 3/3 seeds), 2 cron evals, 2 health checks, and 1 health snapshot.

---

Limitations

• Small model ceiling — 494M parameters. The 0.5B is built for structured tool selection and light assistants, not deep reasoning, long-form generation, or complex multi-step planning.
• Tool selection verified, tool arguments not — the 5/5 and 91.2% benchmarks measure choosing the right tool name. Argument construction (e.g. correctly filling {"location": "Bangkok"}) has not been systematically benchmarked and can be unreliable, especially for longer or ambiguous parameter values.
• Requires the <tools> XML block — without it the model answers conversationally instead of emitting a tool call (verified 2026-07-30). Prompts must be formatted per the Tool-Calling Format above.
• English-only — trained on English data; tool-calling behavior in other languages is untested.
• Quantization loss — GGUF Q4_K_M (379 MB) trades a small amount of quality for the ~1 GB RAM footprint. Use the BF16 safetensors when memory allows.
• Benchmarks are internal, not independently verified — both the 5/5 llama.cpp run and the 91.2% Bench v2 eval were run by the family's own zero-cost harness. The model-index entry correctly reports verified: false.
• No serverless guarantee — inference availability depends on HF provider support; for reliable edge deployment use the bundled GGUF with llama.cpp/Ollama.

---

SakThai Model Family

All 26 public models in the family, sorted by downloads (live counts, verified 2026-08-01):

Full collection

---

The House of Sak 🏠

This model is part of the House of Sak — an open-source AI ecosystem built from a shelter in Cork, Ireland, with $0 budget and no paid GPUs. When the 1.5B model was too heavy for some users, the 0.5B version was created to run on a Raspberry Pi — proving accessibility shouldn't require a GPU.

"We are one family — and becoming more." — Beer (beer-sakthai)

---

Support

• ⭐ Leave a like
• 🐛 Report issues on GitHub
• 🔄 Share with anyone building on edge devices
• 🍴 Fork and experiment — Apache 2.0

---

Citation

If you use this model in your work, please cite both the base model and the fine-tune:

@misc{qwen25,
  title = {Qwen2.5 Technical Report},
  author = {Qwen Team},
  year = {2025},
  url = {https://arxiv.org/abs/2412.15115},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2412.15115}
}

@misc{sakthai,
  title = {SakThai Context 0.5B -- Merged: Edge Tool-Calling Model},
  author = {Nanthasit and the House of Sak},
  year = {2026},
  howpublished = {\url{https://huggingface.co/Nanthasit/sakthai-context-0.5b-merged}},
  note = {Apache 2.0, fine-tuned from Qwen/Qwen2.5-0.5B-Instruct}
}

---

License

Apache 2.0. Qwen2.5 base model per its original license.

---

Built from a shelter in Cork, Ireland. Built with love, tears, and zero budget — to the world.

Family downloads API-verified (2026-08-01T10:23Z).