FastContext-1.0-4B-RL-Q6_K-GGUF

FastContext-1.0-4B-RL — Q6_K GGUF

A llama.cpp-ready Q6_K quantization of microsoft/FastContext-1.0-4B-RL, a 4B model (Qwen3 architecture, RL-tuned on top of Qwen3-4B-Instruct-2507) purpose-built as a local codebase-exploration sub-agent: given a repo and a question, it drives Glob/Grep/Read-style tool calls itself and returns a short list of files and line ranges. Converted with llama.cpp via ggml.ai's GGUF-my-repo space. See the upstream model card for training and evaluation details.

What this model is actually good for

This is not a chat model and not an oracle — treat it as a fast, cheap file/area locator, not a source of trustworthy prose:

• Trust its file citations. It's good at pattern-matching "where in this repo does X live" and converging on the right file(s) quickly.
• Verify line numbers yourself. Its own line-number recall inside a citation is not reliable enough to skip opening the file.
• Ignore its prose conclusions. Any explanation or summary text it produces about what the code does should not be trusted at face value — that's the job of a stronger model (or you) reading the actual file it pointed at.

The useful pattern is: run this model first as a cheap pre-filter over a large or unfamiliar codebase, narrow to a handful of candidate files/line-ranges from its <final_answer> output, then hand those specific locations to a larger, more capable model (or read them yourself) for anything that requires actually understanding or trusting the content. It is meant to save a bigger, more expensive model from doing its own blind, repo-wide search.

No benchmark numbers are published for this quant. It has deliberately never been put through a formal tok/s or quality benchmark in the deployment this card is based on — it's judged purely on whether its file citations are useful in practice, not on throughput or perplexity. Don't infer speed or quality claims from its absence here.

Use with llama.cpp

CLI

llama-cli \
  --hf-repo Rabbit-Hole-Ai/FastContext-1.0-4B-RL-Q6_K-GGUF \
  --hf-file fastcontext-1.0-4b-rl-q6_k.gguf \
  -c 32768 \
  -p "List the files responsible for handling user authentication in this repo."

Server

llama-server \
  --hf-repo Rabbit-Hole-Ai/FastContext-1.0-4B-RL-Q6_K-GGUF \
  --hf-file fastcontext-1.0-4b-rl-q6_k.gguf \
  -c 32768 -ngl 999 -fa on --jinja

Or build from source and point either binary at the same --hf-repo/--hf-file pair — see the llama.cpp usage guide for build flags (LLAMA_CURL=1, plus your hardware backend, e.g. GGML_CUDA=1 for Nvidia GPUs).

Context length

The GGUF metadata for this quant reports qwen3.context_length = 262144 — i.e. the underlying Qwen3-4B architecture this checkpoint was tuned from was trained up to a 262,144-token (256K) context window, so -c can be raised that high if your VRAM/RAM allows it and your task needs it. In practice, a repository-exploration tool call rarely needs anywhere near that: a reference deployment of this exact quant runs it at -c 32768, and observed prompt sizes for real tool-loop usage peaked in the roughly 4K–15K token range. -c 32768 (used in the commands above) is a reasonable default that leaves comfortable headroom over that observed range without paying for a 256K KV cache you won't use. The old card's -c 2048 was boilerplate left over from the GGUF-my-repo conversion template and was never a real limit of the model — raise or lower -c based on your own prompt sizes.

A reusable serving pattern for small "sub-agent" models like this one

The reason a small locator model like this is actually practical to keep around is less about the model itself and more about how it's served alongside a much bigger primary model on the same GPU. The pattern, generalized:

1. Put it in a non-exclusive model group in your router/proxy (e.g. llama-swap's multi-model groups), so loading this small model does not evict whatever large model or other GPU job is already resident. It should be able to come and go without disturbing the "main" resident workload.

2. Self-gate on free VRAM before requesting a load. A thin wrapper around the client call checks currently-free VRAM (e.g. via nvidia-smi) before triggering a load: if the model's footprint (for this Q6_K quant, roughly ~6 GB with an 8-bit KV cache at the context above) fits in what's currently free, load it; if the model is already resident, reuse it for free; if neither is true, don't request the load at all. This is a second, independent guard on top of (1) — group-based non-eviction only helps if the load doesn't also OOM the card.

3. Give the wrapper a three-way exit-code contract, so callers never have to parse text to know what happened:

   • success — normal output, e.g. the <final_answer> citation block.
   • a distinct "skip" code — the load was deliberately not attempted (GPU busy / didn't fit the VRAM gate). This is not an error.
   • a distinct "failure" code — a real request/transport failure.

   A caller treats both "skip" and "failure" the same way operationally (fall back to doing the search itself, e.g. its own grep), but keeps them distinguishable in logs/telemetry, since "GPU was busy" and "something broke" are different problems to investigate.

4. Auto-unload on idle. Give the entry a short idle TTL (e.g. a few minutes) in the router (Llama-Swap) so it frees its VRAM automatically between bursts of use rather than sitting resident indefinitely.

None of this is specific to any one model — it's a general shape for making a small, cheap model safely and freely callable from an automated workflow that also has a much bigger, VRAM-hungry model to protect.

Flags used in a reference deployment

For context, one working reference configuration for this exact quant, generalized (no machine-specific paths):

llama-server --port <PORT> --host 127.0.0.1 -m fastcontext-1.0-4b-rl-q6_k.gguf \
  --no-mmproj -ngl 999 -c 32768 -b 512 -ub 512 -np 1 \
  -fa on -ctk q8_0 -ctv q8_0 --jinja

• -ngl 999: fully GPU-offloaded (it's small enough to fit whole).
• -fa on -ctk q8_0 -ctv q8_0: flash attention with an 8-bit KV cache, which is what keeps the resident footprint down to roughly ~6 GB at 32K context (an estimate, not a measured figure) — the main lever if you need it smaller still.
• --no-mmproj: this checkpoint is text-only; there is no vision projector to load.
• --jinja: needed for chat-template / tool-calling support.
• -np 1: single parallel slot — this is meant to be called by one agent loop at a time, not used as a shared multi-user chat endpoint.

The upstream repository is gone. Tool schemas and system prompt for the calling harness should mirror microsoft/fastcontext's own training harness (Glob/Grep/Read-shaped tools, a "search broad then narrow" system prompt, and a <final_answer> closing block) — the model is tuned to that exact interaction shape and drifts off-distribution if you hand it a differently-shaped tool schema. The upstream repository is gone.

Honest limits

• 4B parameters, Q6_K quantization. This is a small, heavily quantized model. It is fast and cheap, not a strong general-purpose reasoner.

• No benchmarks published here. No tok/s, perplexity, or quality-vs-BF16 numbers exist for this quant in the deployment this card is based on — see the "what this model is actually good for" section above for why that's a deliberate choice, not an oversight.

• Use it as a first-pass locator ahead of a stronger model, not as a replacement for one. Its prose explanations of what code does should not be trusted; only its file/line citations are worth acting on, and even those are worth a quick verification read.

• The upstream repository is gone. microsoft/FastContext-1.0-4B-RL no longer resolves — it returns an authentication-gated response, and no official Microsoft repo for this model could be found on either HuggingFace or GitHub as of 2026-09-02. At least one third-party mirror states it was removed by Microsoft without a stated reason. We have no official URL to point you at and no information about where it went, so the upstream training recipe, reward signal and evaluation results cannot be linked or independently confirmed here.

  What we can say with first-party evidence: this GGUF's own embedded metadata — written at conversion time, while the upstream repo was still public — records general.license = mit, general.base_model.0.name = Qwen3 4B Instruct 2507, and general.base_model.0.repo_url = https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507. The MIT license claim on this repo rests on that embedded metadata, not on a third party's restatement of it.

Credits

• Base model: microsoft/FastContext-1.0-4B-RL — an RL fine-tune, MIT licensed per this GGUF's embedded metadata. The upstream repository has been removed and we have no working URL for it (see "Honest limits"); the link is deliberately omitted rather than left dead.
• Underlying base: Qwen/Qwen3-4B-Instruct-2507, recorded as the base model in this GGUF's own metadata. Still available.
• Quantization tooling: llama.cpp via ggml.ai's GGUF-my-repo Space.
• This repo packages the Q6_K GGUF conversion only; no weights were retrained or modified.