dogmatix

dogmatix v0.2

A QLoRA fine-tune of google/gemma-4-E2B-it (~2.3B effective) into a local, Python-specialised web-application coding agent. Trained on executed, test-verified trajectories distilled from a GLM-5.2 teacher running live inside the real harness. Runs on a single 16GB GPU.

v0.2 supersedes v0.1 (still available on the v0.1 branch). v0.2 is a sharper web-app specialisation: better at end-to-end app-building — the thing this project is for — at a measured cost to Django and general Python. The numbers below are honest, single-harness, and include the trade.

What it is for

Small, well-specified Python web work — FastAPI, Flask, SQLAlchemy/SQLModel — driven through an agent loop: read files, make the edit, run the tests, fix what broke. Fast, local, private; not a frontier engineering model.

Measured results

All figures are measured under one harness (the model sees the full 7-tool surface — read/write/edit/grep/bash/list/run_tests — with turn-based step counting and a blind-call guard). v0.1's original card reported a web-app number taken under an earlier, inflated harness; it is corrected here.

The headline is the first line. End-to-end app-building — build a working app from a natural-language spec, graded by a hidden acceptance suite the model never sees — is what this model is for, and it is the one benchmark with real headroom (base only 0.182). The tuned checkpoints improve monotonically, and v0.2 doubles the base.

Honest caveat: that benchmark is small (11–12 graded specs, one attempt each), so the gain is not yet statistically significant (base-vs-v0.2 McNemar p ≈ 0.63). A larger multi-attempt sweep is running to settle it. Treat "doubles base" as a strong signal, not a proven number. On the older web-app suite v0.2 is at parity with base (+0.04, not significant) — that suite is saturated (base 0.756), which is why the app-building benchmark was built.

Product-path parity — the proxies are faithful

Every number above is from the eval loop; the thing users run is the serving proxy. Running the 12 app-build specs through the actual proxies on v0.2:

Both proxy paths are at or above the eval reference — the serving and tool- name translation layer loses no capability. Tool-calling works end-to-end on both harnesses, including WebSearch pass-through (v0.2 reaches for and uses a WebSearch tool when offered, though it was not explicitly trained for search).

Where fine-tuning helps, and where it costs

• Helps at end-to-end app-building (the objective): 0.18 → 0.36. Notably, the base and v0.1 each have a task they cannot even build without running out of memory on a runaway trajectory; v0.2 completes every build in the suite.
• Costs general Python and Django. v0.2 is the worst of the three at Django (0.500 vs 0.567 base) and MBPP (0.700 vs 0.760 base) — a sharper specialisation than v0.1. Django is the thinnest part of the training corpus. If you need Django or a general Python coder as much as app-building, weigh this trade, or use the v0.1 branch (Django 0.550) or the base.

Honest limitations

• It cannot reliably build a whole application in one pass. It succeeds far more often when work is decomposed into small sub-tasks (how the bundled harness drives it). Even then ~1 in 3 of these small app specs succeeds end to end — useful with verification, not autonomous.
• It cannot be trusted to judge its own work — it has produced a correct app alongside broken tests for it. Use external verification.
• The app-building benchmark is small and underpowered (12 specs); the gain is directional, not yet significant.
• The web-app suite is saturated (base 0.756) — treat that headline as an upper bound.
• Python only. No other languages, no JS frontends, no deployment.
• Long horizons degrade. Design target is 5–10 tool-steps per loop; longer work needs decomposition.

Serving

Ships with an OpenAI-compatible and an Anthropic-compatible proxy (dogmatix/serving/). The translation layer is part of the model: it enforces a turn boundary the base chat template does not provide (without which the model emits long bursts of tool calls and never observes their results), guards against acting on unobserved tool calls, and maps client tool names (Claude Code / Pi) onto the trained tool vocabulary — passing through tools it has no trained equivalent for (Bash, WebSearch, …).

Training

QLoRA (4-bit NF4, r=32) on the language-model projections only, loss masked to assistant tokens, merged to bf16 on CPU. Single adapter, single run, curriculum expressed as data-sampling weights. Trajectories are executed and test-verified — never free-generated.

Reproducing the numbers

• App-building: python scripts/appbuild_bench.py (12 specs with hidden acceptance suites verified against reference implementations before any run).
• Product-path parity: python scripts/appbuild_proxy_bench.py.
• Web-app / Django / MBPP: python scripts/benchmark.py (Django via --suite benchmark/tasks-django).

Author

Prasad Vittaldev