Javi0.2-4B-GGUF

Javi0.2-4B-GGUF (alpha)

Alpha. Javi0.2 is an experiment you can run, not a finished model. It fixed the big problems it set out to fix and picked up new ones along the way. Both are measured and listed below.

A 4B agent model that tries to look things up instead of guessing. Fine-tuned from Qwen3.5-4B to work inside coding and research agents (grok build, omp, opencode) with their own system prompts and tools.

What it was trained to do (how well it does it is in Results):

• Look up anything time-sensitive. "Latest", "current", "today", versions, prices, who holds a role: it searches or fetches before answering instead of answering from memory (1 slip in 64 on the eval).
• Say so when it can't verify something. If search is blocked or a page returns 403, it stops instead of looping and says what it couldn't verify, plus ways to get the answer. It stops too early, and the ways it suggests aren't always real (see Limitations).
• Reason from first principles. It checks the premise of the question first and corrects it when it's false.
• Stay upbeat without bending facts. Ask how it's doing and it's doing fantastic.

It's the second version of a learning project, trained on a single NVIDIA DGX Spark. Javi0.1 only worked with its own training prompt; Javi0.2 is trained on the exact request formats the real clients send.

The full story, including every way it broke along the way, is on X.

Files

Both were quantized from an F16 GGUF of the merged model. The LoRA was merged into the bf16 base, not into a 4-bit copy. Text only: the base model's vision tower is not included.

Usage

llama.cpp

llama-server -hf jvr0x/Javi0.2-4B-GGUF:Q8_0 --jinja -c 131072 \
  --chat-template-kwargs '{"enable_thinking": false}' \
  --temp 0.2 --top-p 0.8 --top-k 20 --min-p 0

Then point any OpenAI-compatible agent at http://localhost:8080/v1. The model needs tools to do its job: in a plain chat with no tools it can only answer from what it knows, and it will tell you when a question needs a lookup.

• Thinking off. It was trained and evaluated with enable_thinking: false.
• Low temperature (0.2) keeps tool calls well formed.

Results

Held-out eval: 136 questions it never trained on, each run inside a real client format (grok, omp, opencode, a bare OpenAI format, and a plain chat) with live tools: web search, page fetches, and a sandboxed shell over small code projects. Some questions force tool failures (search blocked, page 403) to test how it handles them. Every answer was graded by DeepSeek-V4.1-Flash for grounding, honesty, tool use, and tone.

Behavior (counted from the transcripts):

Answer quality (DeepSeek-V4.1-Flash as judge, strict; a pass needs grounded facts, honest gaps, good tool use and a good tone):

Read the two tables together. Javi0.2 does what it was built for: it finishes, it doesn't loop, and it almost never answers a time-sensitive question from memory. But its answers are worse. Its final answers run about 3x longer than the base model's, and when it can't verify something, its list of "ways to get it" is often padded with URLs and API endpoints it made up. The judge penalizes that hard, and it should.

Hardware

• Memory = weights plus an F16 KV cache. Only 8 of the 32 layers use full attention (the rest are linear attention with a fixed-size state), so the KV cache is small: 32 KiB per token, 8 layers × 4 KV heads × 256 dims × K and V × 2 bytes. -ctk q8_0 -ctv q8_0 halves it.
• Context: the base model supports 262k tokens. Javi0.2 trained on conversations up to 72k tokens, including multi-turn sessions and long tool outputs. Past that, it runs on the base model's long-context ability.
• Speed is llama-bench decode (256 tokens, 3 runs) on an NVIDIA DGX Spark (GB10, 128 GB unified memory), full GPU offload, flash attention on. Prefill runs at about 4,000 tok/s for both quants.

Example

In opencode, Q8_0, thinking off:

User: hey, how are you doing today?

Assistant: I'm doing great, thanks for asking! I'm ready to help with whatever you need.

Asked "What's the latest Qwen model?" with web search down, it doesn't guess. It says it couldn't verify, why, and how to find out. That part works. Its list of "ways to find out" is where it tends to invent endpoints (see Limitations).

How it was made

Client capture. A stand-in server recorded the exact requests grok build, omp, and opencode send: system prompts of 15-25k tokens and 10-27 tools each. Training rows use those prompts and tools verbatim, so the model learns to behave well inside the real clients rather than inside a prompt it will never see.

Episodes: teacher inside the harness. 1,383 seed questions (lookups, forced failures, first-principles reasoning, false premises, conflicting sources, unanswerable questions, small talk, coding tasks) were answered by a teacher, Qwen3.8-Flash-Next, running inside each client's exact format with real tools: web search through a local SearXNG, page fetches, and a Docker sandbox for shell and file work on six small projects. The working rules (first principles, look up anything time-sensitive, say when you can't verify and how to find out, try at most two routes after a failure, stay upbeat) were in the teacher's prompt only. Training rows keep the client's own prompt, so the student learns the behavior, not the instructions.

Filtering. DeepSeek-V4.1-Flash judged every episode for grounding, honesty, tool use, and tone. Code checks dropped anything unfinished, anything that answered a time-sensitive question without a tool, anything with more than about six tool calls, and anything that repeated an identical call. Clean recoveries from real web failures were kept; flailing was not. 692 rows made it, 131 of them carried over from Javi0.1.

Training: LoRA SFT with Unsloth, loss on decisions only.

Why loss on decisions only. A first version trained on every assistant turn, including each intermediate tool call. It learned to never answer time-sensitive questions from memory, but also to keep calling tools: it re-ran searches that had already worked and finished only 85 of 136 eval questions. Training only on the turns that decide something (answer directly or pick a tool, then give the final answer) fixed the loops without losing the lookups. It also concentrated all the learning on final answers, which is where this version's padded, invented "ways to get it" come from. The recovery step in the middle of a conversation (a call failed, try the next route) got no training at all, which is why it gives up after one failure.

Export: the adapter was merged into the 16-bit base, converted to an F16 GGUF, then quantized with llama-quantize.

Limitations (known issues in this alpha)

• Invents "ways to get it". When a lookup fails, it often suggests URLs, API endpoints or curl commands that don't exist, and repeats the same suggestion several times.
• Gives up after one failure. One failed fetch and it may stop and report, instead of trying the next route itself. It suggests curl commands to the user even when it has a shell tool to run them.
• Doesn't cross-check. It can take one secondary source as definitive (for example, a community fine-tune as "the latest Qwen model").
• Sometimes claims a check it didn't do. Rare, but seen in testing: it called a figure "verified" that no tool in the conversation returned.
• Careless tool calls. Unencoded URLs, missing format=json, JSON parsing with no error handling.
• Long answers. About 3x the base model's length, sometimes contradicting itself (a wrong headline number above the right calculation).
• It's a 4B model. It can still get facts wrong.
• It's built for agent clients with tools. Without tools it can't look anything up.
• The eval uses live web search, so scores move a few points between runs, and the grader is an LLM, not a person.
• Trained mostly on English.

What's next

Javi0.2.1 targets the issues above: learning the recovery step after a failed call, preferring and cross-checking authoritative sources, only calling something verified when a tool result shows it, and short failure answers with no invented endpoints.

License

Apache 2.0, inherited from Qwen3.5-4B.