opencoti-llamafile

opencoti-llamafile

Self-contained single-file inference engine from the opencoti project — a Mozilla-Ocho llamafile base carrying the opencoti patch series: advanced KV residency/quantization (PolyKV, KVarN — see docs/features/kvarn.md), the KV rolling window (a KV cache larger than VRAM), elastic multi-session serving (elastic slots, an enforced KV admission gate with fast 429 + Retry-After, PolyKV shared-prefix pools with a REST control plane), DCA long-context extension, MTP speculative decode, sparse attention, RYS layer duplication, CUDA and Vulkan GPU backends, and a runtime introspection/control API.

Start with USAGE.md — it explains exactly how this engine diverges from upstream llamafile, every added feature, its flags, defaults, limitations, and which features compose.

This repo hosts the packaged release artifacts (they exceed GitHub's 2 GiB release-asset cap) plus the full patch series under patches/ and the project documentation under docs/. Release notes and per-release SHA256SUMS live under releases/ and are mirrored on the corresponding GitHub release at mann1x/opencoti.

The repo root always holds the LATEST cut's single-file executables only. Superseded cuts are archived whole — binaries, manifests, their dso/ set (c7 and older), release notes and SHA256SUMS — under releases/archived/ in one self-contained directory per release tag (e.g. releases/archived/llamafile-v0.10.3+opencoti.c4/).

Supported / target model families

This engine is not model-agnostic in what it optimizes. Upstream llama.cpp GGUF support is unchanged (any GGUF that loads upstream loads here), but the opencoti feature set is built, tuned, and validated on two families:

Other architectures run with upstream behavior; opencoti features either fall back safely or are unvalidated on them — see the model-families section at the top of USAGE.md before relying on an opencoti feature elsewhere.

Artifacts

The main release is ONE file. Download the single executable for your platform; every GPU library it needs is inside.

Every embedded library is byte-identical to the separate file of the same release (below); the publisher refuses a release whose single files embed anything else.

The split form (for integrations that pin each part by sha256 — the pin files under pin/, format in PIN_FORMAT.md): the bare engine components/engine/<build>/opencoti-llamafile-<ver>-<tag>-bare.llamafile (~135 MB, the same bytes as …-win-x86_64.llamafile.exe there; CPU only by itself) and one directory per library under components/ (cuda, cuda12, sbsa, vulkan, macos (Metal + the signed loader), media (codec / speech sidecars)). pin/index.txt names the current file of each.

The APE runs natively on x86_64 and aarch64, Linux, Windows, macOS and BSD; the variants differ only in which GPU libraries are embedded. Windows refuses executables over 4 GiB — the libraries are built compressed (--compress-mode=size) so the .exe files stay well under it. With no usable GPU library the engine runs on the CPU.

Usage

chmod +x opencoti-llamafile-*.llamafile
# server mode
sh ./opencoti-llamafile-*.llamafile --server --port 8080 \
    -m <model>.gguf -ngl 99 --flash-attn on
# without --server it starts the interactive chat CLI

On Windows run the .exe directly (opencoti-llamafile-<ver>-<tag>-universal.llamafile.exe --server …); only the NVIDIA (or AMD / Intel) driver is needed — no CUDA Toolkit, no MSVC.

Where the GPU library goes (c10 r2+). A GPU library cannot be loaded from inside the file, so on the first GPU run it is extracted next to the executable: <exe folder>/.llamafile/v/opencoti-<ver>-<tag>/p/<key>/ (e.g. 0.7 GB for CUDA on Windows). Only when that folder is not writable (e.g. C:\Program Files) does it go to the home folder (~/.llamafile/… / %USERPROFILE%\.llamafile\…, unless HOME is set), and the engine prints one line saying so. On Windows that profile fallback needs c10 r3 or later: r2 put the library in the current directory there. The key is the library's own checksum, so a later run of the same release reuses it and two releases never overwrite each other. The boot log names the file it loaded (cuda: loaded … (bundled, beside the executable, … bytes)). Delete the .llamafile folder to reclaim the space. (c7 – c10 r1 always extracted to the home folder.) --version prints the opencoti version string; GET /props exposes an opencoti introspection section at runtime (see USAGE.md §introspection).

The bare engine + a separate library

Put the library file, under its published name, in the same folder as the engine — the engine looks there first and loads it as is (no rename, no copy into your profile):

opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe
ggml-cuda-win-x86_64.dll        # components/cuda/<build>/  (or ggml-vulkan-win-x86_64.dll for --gpu vulkan)

On Linux: ggml-cuda-x86_64.so / ggml-cuda-cu12-x86_64.so / ggml-vulkan-x86_64.so / ggml-cuda-sbsa-aarch64.so beside the APE. The boot line then reads (executable directory, … bytes). Check every file against the sha256 in its pin/<component>.txt.

Verification

Each single-file executable ships with a MANIFEST.json recording its sha256, the version string, the git commit, the full patch list and the sha256 of every embedded GPU library; releases/<full-tag>/SHA256SUMS lists every file of the release. An embedded library is the same file as the one in components/ — compare it with the sha256 in pin/<component>.txt without running anything:

unzip -p opencoti-llamafile-*-universal.llamafile.exe ggml-cuda.dll | sha256sum   # == pin/cuda.txt, win-x86_64 row

License

Apache-2.0 (llamafile) + MIT (llama.cpp and bundled projects). opencoti patches and packaging are MIT.

PolyKV multi-session stress benchmark (package_courier)

An agentic multi-session benchmark: LangGraph courier agents (tool-calling, 10-step delivery chains, 100 packages) run against ONE engine at --parallel 32, ctx 163840, q8_0 KV, with a tps-floor scheduler that admits new sessions only while every live session keeps ≥ floor tok/s (PolyKV /polykv/tps SSE telemetry drives admission). Every run below was measured on a single NVIDIA RTX 6000 Blackwell (96 GB) — one GPU, one engine process, no tensor/pipeline split; the whole model and all 32 slots' KV are resident, so the numbers are pure scheduler/serving behaviour rather than an offload artifact. Each worker episode is a compiled langgraph.StateGraph (capacity_gate → agent ⇄ tools, with the capacity-gate node reading the engine's /polykv/pools/{id}/capacity), so the harness doubles as the reference LangGraph integration for this engine. The full harness is published in this repo at bench/package_courier/ (work server + LangGraph agent app + launcher + HTML report generator — see its README and the usage guide at docs/features/package_courier_harness.md); reports under bench/reports/ (per-run HTML + raw stats).

The opencoti_langgraph Python package the harness builds on — OpencotiChatModel (LangChain BaseChatModel with PolyKV pool/session attach), PolykvClient/AsyncPolykvClient, pool tools, and the capacity-gate / compaction LangGraph nodes — is published at integrations/langgraph/ (pip install ./integrations/langgraph, needs langchain-core>=1.4.9

• langgraph>=1.2.9; see its README).

fan-out is the time-weighted median of concurrently running workers over the run — the number of agents actually in flight, not the scheduler's target and not the engine's attached-session count (which is cumulative and saturates at slots). Two token rates, and the distinction matters:

• gen tok/s — generated tokens ÷ wall. This is what the tps-floor scheduler gates on, so it is what sets fan-out. It is not a model-speed comparison: at 36:1–74:1 prompt-to-completion it largely reflects how terse a model is.
• work tok/s — (prompt tokens actually processed, i.e. net of prompt-cache hits) + generated, ÷ wall. This is the real work rate. Note the cache-hit column: each turn resends the whole conversation and the engine serves 25–76 % of it from KV cache, so counting raw prompt length would inflate this several-fold.

Reading these numbers

Work rate is set by active parameters. The A4B pair runs at 2912 work tok/s against 813 for the dense 27B at floor 15 — 3.6×, which is what a 4B-active MoE against a 27B-dense should look like. Weight precision costs on top of that: the same A4B at F16 does 1571, i.e. 1.85× behind Q4_K_M, a bandwidth-bound MoE reading 4× the bytes per active expert.

Terseness is a separate axis from speed, and gen tok/s conflates them. The dense 27B posts the higher generation rate at floor 15 (79.9 vs 64.5) while doing 3.6× less work per second — because it is a reasoning model that emits <think> blocks and produces 173 k tokens to the A4B's 55.6 k for the same 100 deliveries. Wall time follows the product of both axes: A4B finishes 2.5× sooner (864 s vs 2171 s) by being both faster and terser. If you take one thing from this table: do not rank engines by generation tok/s on an agentic workload — it rewards verbosity.

The cache-hit column is why rows are not freely comparable. Every turn resends the full conversation; the engine matches the common prefix in KV and only prefills the delta. A model that needs more turns per package re-sends a longer already-cached prefix, so its hit rate climbs and its measured work rate falls — not because the engine got slower, but because there was less left to do. That is why the 35B-A3B's 1609 cannot be read as "slower than the A4B's 2912": it ran at 75 % cache hit against 40 %. Compare rows freely only where the cache column is close (e.g. A4B Q4_K_M vs F16, 40 % vs 49 %).

None of this is a clean engine benchmark — it is a scheduler-and-serving measurement over a real agentic workload. For isolated decode/prefill throughput use docs/evaluations/mtp.md.

How fan-out is set. workers ≈ gen tok/s ÷ floor — the scheduler admits until the generation rate the engine can deliver is exactly divided into per-session shares of floor tok/s. That rate is not a constant: it rises as the floor rises (A4B-Q4: 40.4 → 72.0 tok/s from floor 1 to 25) at identical total work, so the engine does the same job faster with fewer concurrent sequences — consistent with per-token batch interference and KV pressure, though we have not isolated which dominates. Practically: a higher floor buys both a faster per-session experience and a shorter wall time. You only pay in parallelism.

The A4B + assistant-MTP pair finishes ~2.5× faster end-to-end than the dense 27B at floor 15 (864 s vs 2171 s) while generating 3.1× fewer tokens for the same 100 deliveries (55.6 k vs 173 k) — a terser agent on a faster engine.

All losses across all runs are agent-side (turn-budget wander on hard packages) — zero tool errors, zero LLM errors, engines clean throughout. The ab-pooled vs ab-naive pair isolates the SharedKVPool win at identical workload.

The A4B F16 ladder answers "what does weight precision cost under agentic load": same engine, same workload, F16 weights are ~1.55× slower end-to-end than Q4_K_M at every floor (1.9× at floor 1, where contention compounds it) — a bandwidth-bound MoE. At comfortable floors quality is saturated at Q4 (both 100/100), but at max fan-out precision shows: F16 floor 1 finished zero-loss at the full 64-worker cap grinding at ~1.1 tok/s median, while Q4_K_M at floor 1 loses 3-5 first-wave packages to turn-budget wander (95/100 original; a controlled re-run on the current StateGraph agent, same seed/cap, scored 97/100 with the identical loss signature — confirming the loss mode is model-precision-driven under cold-start contention, not a harness artifact).

The 35B-A3B ladder (worker cap raised to 64, floors extended to 20/25) is the cleanest full ladder: 100/100 at every floor including max fan-out, with the scheduler holding median per-turn tps on the floor (5→5.6, 15→17, 20→23, 25→28) as the fleet shrinks 64→6. It also locates the fan-out knee: wall time is flat (~890-1040 s) from floor 5 up, but floor 1 costs 1574 s — 64 workers oversubscribing 32 slots just queues (per-session tps 2.8, deliveries waiting 786 s mean). Sweet spot for this engine: floor 20-25 — same wall as floor 15 with the best per-package latency, at 6-7 concurrent workers each getting 23-28 tok/s. Note how little concurrency it takes to saturate: throughput is already at its maximum by floor 15, and everything above ~10 workers on this engine buys queueing, not work.

---

<img src="vodafone-logo.png" alt="Vodafone" width="96"/>

Computing resources kindly provided by Vodafone