Downloads · 30 days
114
11% of all-time downloads
SEBK4C/gemma-4-12b-it-qat-q4_0-llamafile
gemma-4-12b-it-qat-q4_0-llamafile is a text generation model from SEBK4C. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
One self-contained executable. Runs on NVIDIA GPUs, Apple Silicon (M1–M5) and plain CPUs — and tunes itself to whichever it finds. Serves chat completions (text + image + audio input), embeddings, and a full web UI fr…
Downloads · 30 days
114
11% of all-time downloads
All-time downloads
1.1K
Public
Repo size
135 GB
Likes
0
Public
Click a slice to open those files.
.llamafile9.1 GB · 100%
From the Hugging Face model README
One self-contained executable. Runs on NVIDIA GPUs, Apple Silicon (M1–M5) and plain CPUs — and tunes itself to whichever it finds. Serves chat completions (text + image + audio input), embeddings, and a full web UI from one model instance on one port.
Source, patches, changelog: SEBK4C/Llamafile-gemma-4-12B-it-qat-q4_0-gguf-Inferance-And-embeddings (release v0.7.0). Built on a performance fork of mozilla-ai/llamafile (v0.10.7).
One file: gemma4-server.llamafile (v0.7.0-universal). It detects its
platform at runtime and boots the right baked profile:
.args.xnu: ~20 tok/s on M1 Pro
vs ~6 under the CUDA-tuned flags) + a prewarmed system prompt (the
first message skips its prefill entirely)Also inside regardless of platform: baked Qwen3 embeddings (1024-dim
/v1/embeddings), /v1/ingest, MTP drafter, web UI with voice controls
(LLAMAFILE_TTS_PORT / LLAMAFILE_EMBED_PORT hook up external sidecars).
Previous artifacts remain available in this repo's commit history.
v0.7.0 release notes
One file now serves chat + retrieval-grade embeddings + document ingest:
/embed/v1/embeddings, /embed/health, /embed/tokenize. v0.6.1: the main /v1/embeddings endpoint serves these vectors by default — every OpenAI client gets retrieval-grade 1024-dim embeddings with zero config (best use: documents bare, queries prefixed Instruct: <task>\nQuery: ). The 12B's raw anisotropic output stays available via header X-Raw-Embeddings: 1; opt out entirely with LLAMAFILE_NO_EMBED=1.POST /v1/ingest: text in → one grammar-constrained enrichment call (title, summary, entities, task domain, chunking hints) + a deterministic fidelity gate (ungrounded entities dropped — composed dates and mutated numbers never reach your index) + token-budgeted chunks + 1024-dim doc & chunk vectors, returned as one ingest.v1 envelope for hybrid BM25+vector indexing.kill -9./v1/ingest 1.97 s on GPU. Measured pipeline quality (all reproducible from bench/): retrieval hit@1 0.929 / MRR 0.955 over a multimodal corpus; labeled-receipt key-field recall 100% with 0% gate false-drops; NFCorpus nDCG@10 0.3626. Full details: release notes · INGEST_GUIDE.This build ships with empirically-validated serving defaults (release notes):
/v1 API requests.Review the testing — these defaults were chosen from a benchmarked A/B study (Constitution+decline prompt wins or ties a bare prompt on accuracy, humanness, sophistication, and calibration, and declines persona/developer-mode/fiction jailbreaks 100% without over-refusing):
bench/RESEARCH_HISTORY.md, harness bench/serve_bench.py, spec bench/program.md in the GitHub repo.chmod +x gemma4-server.llamafile
./gemma4-server.llamafile
Then open http://127.0.0.1:8080/ in your browser. That's the whole install. The web UI supports markdown chat with visible reasoning, image and audio attachments (🎙 records straight from your microphone), conversation management, and llama.cpp's tooling.
On first start the file detects your hardware and applies tuned defaults:
| Detected | Applied automatically |
|---|---|
| NVIDIA GPU (CUDA baked in, driver-only) | 131072 ctx, f16 KV, MTP speculative n=4 — ~90–200 tok/s on an RTX 3080 Ti |
| Apple Silicon (Metal) | 8192 ctx, MTP n=2 — the config validated on an M4 |
| CPU only | 8192 ctx, speculation off |
Override any flag to take control of it (./gemma4-server.llamafile -c 32768),
or LLAMAFILE_NO_AUTOTUNE=1 to disable tuning entirely. Launched it twice by
accident? The second copy tells you, and opens the already-running UI.
| Endpoint | What |
|---|---|
POST /v1/chat/completions | OpenAI-style chat (+ SSE streaming, function calling); accepts text, image_url data URIs, input_audio (wav/mp3) |
POST /v1/messages + /v1/messages/count_tokens | Anthropic Messages API, native — tools, thinking blocks, SSE; Claude Code connects with env vars alone |
POST /v1/responses | OpenAI Responses API — reasoning + message output items, streaming |
POST /tts/v1/audio/speech | built-in Kokoro text-to-speech (returns WAV; /tts/health = liveness) |
POST /v1/embeddings | Retrieval-grade embeddings by default (v0.6.1) — transparently served by the baked Qwen3-Embedding-0.6B (1024-dim, Apache-2.0). Embed documents bare; prefix queries with Instruct: <task>\nQuery: (measured: −17% retrieval quality without it). Raw 12B pooled output (3840-dim, anisotropic, research-only): header X-Raw-Embeddings: 1 or LLAMAFILE_NO_EMBED=1 |
POST /v1/ingest | text → retrieval-ready JSON (v0.6.0): one grammar-constrained enrichment call (title/summary/entities/task-domain/chunk hints) + deterministic fidelity gate (ungrounded entities dropped) + token-budgeted chunks + doc & chunk vectors, in one ingest.v1 envelope |
/embed/v1/embeddings, /embed/tokenize, /embed/health | the embedding sidecar addressed directly (same vectors as /v1/embeddings default) |
GET /health, /props, POST /tokenize | usual llama-server extras |
curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Why is the sky blue?"}],"max_tokens":256}'
/tts; no sidecar, no extra
download--clear-all — wipe saved KV state and extraction caches, start fresh-c N — context (up to 262144 with -ctk q8_0 -ctv q8_0 on 12 GB GPUs)--port N, --host 0.0.0.0 — serve beyond localhostbin/llamafile with
external weights there: model + mmproj GGUFs from
google/gemma-4-12B-it-qat-q4_0-gguf,
and the MTP drafter GGUF from the
v0.5.0 release assets
(everything is also extractable from this llamafile: unzip gemma4-server.llamafile)curl -fsSLO https://raw.githubusercontent.com/SEBK4C/Llamafile-gemma-4-12B-it-qat-q4_0-gguf-Inferance-And-embeddings/cuda-3080ti-optim/bench/api_probe.py
python3 api_probe.py --base http://127.0.0.1:8080
21 end-to-end tests — every endpoint above (embeddings default + /v1/ingest included) plus vision, audio-in and TTS — 18 PASS / 0 FAIL on the v0.6.1 build;
each with wall-clock speed on your machine. Stdlib-only, no installs. All
published runs, charts and the protocol live in the
test-data repo.
⚠️ Gemma 4 12B is NOT a top coding model — it completes small, well-scoped agentic tasks at interactive speed, fully offline. Temper expectations accordingly and review what it produces.
| Harness | Surface | Verified |
|---|---|---|
| Claude Code | Anthropic /v1/messages (no adapter) | multi-turn file+run tasks, 9.6–12.5 s |
| OpenCode | OpenAI chat completions + tools | same tasks, 8–12 s |
| OpenClaw | openai-completions provider | chat + exec-tool turns, 7–11 s |
| Cline / Kilo Code | OpenAI-compatible (VS Code) | config verified at API level |
Setup guides with the exact verified configs: docs/integrations/.
mozilla-ai/llamafile · ggml-org/llama.cpp · ggml-org/llama-ui · hexgrad/Kokoro-82M · thewh1teagle/kokoro-onnx · google/gemma-4-12B-it-qat-q4_0-gguf