AILOFlow

AILOFlow

A local runtime for open-weight LLMs, with dynamic weight streaming from storage.

Run models larger than your RAM by treating the weights as a dataset spread across every SSD in the machine — and measure honestly what that actually costs.

Source code: https://github.com/xxrickyxx/AiloFlow

---

The one rule

Every number shown is measured.

Where a value cannot be measured on the machine, the interface prints n/a instead of a plausible-looking placeholder. Estimates are labelled as estimates. A storage benchmark that was served by the OS page cache says so out loud, rather than reporting a SATA SSD at 9 GB/s.

This is not a demo that simulates a runtime. Nothing renders a number that did not come from a real measurement.

---

What it does

Manages its own inference engine. Picks between CUDA, HIP/ROCm, Vulkan, Metal and CPU based on the GPUs actually present, downloads the matching llama.cpp build from the official releases, verifies it and runs it as a child process. No external daemon required. Ollama is supported as an optional engine, and models already downloaded through Ollama run even with its daemon stopped — they are ordinary GGUF files.

Speaks two protocols on one port, so IDE assistants can target it directly:

Tested with Cline and Continue. Reasoning models keep their chain of thought in thinking / reasoning_content, separate from the answer, so it never contaminates generated code.

Cancelling a request stops the engine: a disconnected client whose generation keeps running would hold the GPU for minutes producing tokens nobody reads. For clients that stop reading without closing the socket, a new request supersedes the old one and POST /v1/generation/stop frees the GPU on demand.

Whatever base URL you configure works — .../v1, .../api or the bare host — because providers append /api/chat to whatever you typed and the duplicated prefix is normalised instead of returning an HTML 404 into the IDE.

For agent use, pick an Instruct model, not a Thinking one. A Thinking variant reasons before every answer and cannot be told not to: enable_thinking, reasoning_budget: 0 and /no_think are all ignored by a model trained to always reason. At 19 tok/s that is minutes of deliberation per step, and the reasoning eats the context the agent needs for its own instructions.

Reads the hardware instead of guessing it. CPU instruction sets come from IsProcessorFeaturePresent / /proc/cpuinfo / sysctl, so an i9-13900K is correctly reported as AVX2 rather than AVX-512. GPU VRAM comes from the driver registry key, because WMI's AdapterRAM is 32-bit and reports 4 GB for anything larger. GPU utilisation comes from performance counters that work on AMD, Intel and NVIDIA alike. Volumes are joined to physical disks through the partition table, so a drive is never mislabelled by array position.

Distributes a model across drives into a .sflow container, validates that the shards on disk match the manifest, and measures the streaming pipeline — bandwidth, per-layer latency, cache hit rate, and how many tokens per second storage alone could sustain.

Treats a split model as one model. Anything past 50 GB is published as an ordered set of files, and each part's header describes only the tensors it holds, with offsets into itself. Parsing, inspection and sharding follow every tensor to the part it actually lives in, and engine startup is given a budget derived from total size rather than a fixed timeout.

Estimates a model before you download it, from the machine's measured bandwidths.

---

Measured, on a real machine

i9-13900K · 32 GB DDR5-6000 · Radeon RX 6750 XT · 1 NVMe + 4 SATA SSD

The 80B sweep is genuinely cache-cold: the shard set is larger than RAM.

---

What it will not do

Decode is bandwidth-bound — one token requires reading every active weight — and the arithmetic is unforgiving:

A dense 300B streamed from disk would need 1345 GB/s to reach 10 tok/s — two orders of magnitude beyond any consumer array. Mixture-of-Experts models, which touch a fraction of their weights per token, are the ones this architecture can genuinely serve.

Generation itself runs on llama.cpp's compute kernels; rewriting those was never the goal. Chatting with a model therefore does not exercise the streaming pipeline — llama.cpp loads weights its own way. The useful question is not "AILOFlow versus llama.cpp on tokens/s" but can storage feed the rate the engine achieves?

---

Catalog

40 curated open-weight models — Kimi K2, DeepSeek R1/V3, Qwen3 235B/480B, Llama 4, GLM-4.6, MiniMax-M2, gpt-oss and more — filterable by size and capability, plus free search across Hugging Face. Every repository id is verified against the API in the test suite.

Downloads belong to the runtime rather than to a browser tab: they survive navigation, page reloads and server restarts, retry automatically on network failures with exponential backoff, and resume from the exact byte. Models published as split file sets are handled as a single model.

---

Install

git clone https://github.com/xxrickyxx/AiloFlow.git
cd AiloFlow
npm install
npm run server        # API on 127.0.0.1:11500
npm run gui:dev       # interface on localhost:3000

Then: Settings → Install engine, Catalog → download a model, Models → load, Chat.

Interface available in English and Italian.

---

Privacy

No prompt and no model leaves the machine. The API listens on 127.0.0.1 only, and there is no remote telemetry of any kind.

---

Licence

Dual-licensed:

• Non-commercial use — CC BY-NC-SA 4.0
• Commercial use — separate written agreement required

llama.cpp is a separate work under the MIT License and is not redistributed here; AILOFlow fetches it at runtime from the official releases. Models downloaded through the catalog carry their own licences, which the catalog displays.

Author: Riccardo Sparacino — LinkedIn