openthai2.0-qwen3.8-27b-MLX-4bit

OpenThai2.0 - Opensource Thai Knowledge, Document, and Agentic AI (MLX 4-bit)

MLX 4-bit quantization (mlx-vlm) of iapp/openthai2.0-qwen3.8-27b for Apple silicon. Vision included. ~16 GB — runs on 24 GB+ unified memory.

v2.0.3 — rebuilt from the v2.0.3 weights (Thai knowledge-recall fix). The v2.0.0 launch build stays available at revision tag v2.0.0.

Run

pip install mlx-vlm
python -m mlx_vlm generate --model iapp/openthai2.0-qwen3.8-27b-MLX-4bit \
  --image document.jpg --prompt "อ่านข้อความในเอกสารนี้ทั้งหมด" --max-tokens 8192

⚠️ The model reasons before it answers — leave a large generation budget (8k+), or replies may come back empty.

Notes: the MTP draft head is not included (mlx-vlm has no drafter support for this architecture yet). Under TensorFold (below), drafting comes from z-lab's DFlash2 drafter instead. Sanity-verified on-device: Thai factual prompts answered correctly.

Run with TensorFold (Mac or NVIDIA)

TensorFold serves the MLX 4-bit checkpoint iapp/openthai2.0-qwen3.8-27b-MLX-4bit through an OpenAI-compatible API on Apple Silicon and on NVIDIA GPUs. It speeds up decoding with speculative drafts, and every drafted token is checked against the model, so replies are the same as TensorFold decoding this checkpoint one token at a time. The same checkpoint runs on both backends, with image input.

These instructions are written for TensorFold v0.6.5.

DGX Spark and other NVIDIA GPUs

Needs compute capability 8.9 or newer: RTX 40 / 50 series, Hopper, Blackwell, including the DGX Spark (GB10).

docker run -it --gpus all --ipc=host --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  nvcr.io/nvidia/pytorch:26.07-py3

# inside the container
python -m pip install "tensorfold[vision] @ git+https://github.com/ashhart/[email protected]"
tensorfold pull iapp/openthai2.0-qwen3.8-27b-MLX-4bit z-lab/Qwen3.8-27B-DFlash2
tensorfold serve iapp/openthai2.0-qwen3.8-27b-MLX-4bit \
  --name openthai2.0 --host 0.0.0.0 --port 8080 --vision

The first start compiles CUDA kernels for your GPU, which takes a few minutes once. Add --parallel 4 to decode up to four requests together. On CUDA the draft model is required: pull it as above, or serve with --no-drafts.

Apple Silicon (MLX)

48 GB of unified memory or more is recommended. On a 32 GB Mac the model does not fit TensorFold's default budget (70% of RAM); raise it with TENSORFOLD_MEMORY_LIMIT_GB, leaving little memory for other apps.

python -m pip install "tensorfold[vision] @ git+https://github.com/ashhart/[email protected]"
tensorfold pull iapp/openthai2.0-qwen3.8-27b-MLX-4bit z-lab/Qwen3.8-27B-DFlash2
tensorfold serve iapp/openthai2.0-qwen3.8-27b-MLX-4bit --name openthai2.0 --vision

Call it

Any OpenAI client works with base URL http://<host>:8080/v1 and model openthai2.0.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="none")

# Text
r = client.chat.completions.create(
    model="openthai2.0",
    messages=[{"role": "user", "content": "สรุปสาระสำคัญของ พ.ร.บ. คุ้มครองข้อมูลส่วนบุคคล ใน 3 ข้อ"}],
    max_tokens=4096,
)
print(r.choices[0].message.content)

# Thai document image (data URL; start the server with --vision-urls to accept public HTTPS links)
import base64
img = base64.b64encode(open("document.png", "rb").read()).decode()
r = client.chat.completions.create(
    model="openthai2.0",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "อ่านข้อความทั้งหมดในเอกสารนี้"},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img}"}},
    ]}],
    max_tokens=4096,
)
print(r.choices[0].message.content)

The model thinks before it answers by default; the reasoning arrives in reasoning_content and the answer in content, so leave enough max_tokens. Start the server with --no-thinking to answer directly.

Notes

• Draft model. z-lab/Qwen3.8-27B-DFlash2 (Apache-2.0) was trained on base Qwen3.8-27B, not on OpenThai 2.0. It accepts fewer drafted tokens on Thai replies, so the speedup is smaller than on the base model; the replies themselves are unaffected. Send "draft": false in a request to get the one-token-at-a-time reference.
• Same output, same engine. Drafted and serial replies match within TensorFold on this checkpoint. Output differs from the bf16 model served by vLLM or transformers, because the weights are 4-bit.
• Why MLX 4-bit on NVIDIA too. TensorFold also reads the NVFP4 checkpoint on one NVIDIA GPU, but in TensorFold's own measurements of base Qwen3.8-27B on a DGX Spark, the MLX 4-bit checkpoint decoded faster.

<!-- ### Speed (TensorFold v0.6.5, OpenThai 2.0 MLX 4-bit) | Machine | Thai chat | Thai document QA | Prefill 8k | Concurrent x4 (aggregate) | | --- | ---: | ---: | ---: | ---: | | DGX Spark (GB10) | — tok/s | — tok/s | — tok/s | — tok/s | | MacBook Pro M_ Max, __ GB | — tok/s | — tok/s | — tok/s | — | -->

Full benchmarks and model card: see the main bf16 repo.