llama-3.2-1b-mini-agent

<div align="center"> <img src="logo.png" alt="Llama Mini Agent" width="320"/>

Llama Mini Agent

A 1B tool-calling and shell-command fine-tune of Llama-3.2-1B-Instruct — shipped as a LoRA adapter and six GGUF quants.

checkpoint py-newds · evaluated 2026-09-09

</div>

---

Summary

llama-3.2-1b-mini-agent is a QLoRA fine-tune of Llama-3.2-1B-Instruct trained jointly on function calling, Bash one-liners, and Python. It is small enough to run on a laptop CPU and ships both as a 90 MB adapter and as merged GGUF builds from Q3_K_M to fp16.

On its own protocol it emits a well-formed tool call 99.2% of the time, picks the right function 98.0% of the time, and reproduces the entire call — every argument — in 80.4% of held-out cases.

The one thing to know before you use it: this model speaks a fixed protocol. It was trained on three specific system prompts, and for tool calls it answers with a JSON list of {"name", "arguments"} objects. Prompt it that way and you get the numbers above. Prompt it with a different tool-calling convention and it degrades sharply — see Out-of-format robustness. The three prompts are written out verbatim in How to prompt it; use them.

At a glance

Specifications

---

Evaluation

How this was measured

Every number below comes from a fresh run on 2026-09-08/09, scored with each benchmark's own official scorer. Where a task can be executed, it was executed — nothing here is string-matching against a reference answer except where explicitly labelled.

• Decoding — greedy (do_sample=False), batch size 8, 4096-token eval context. Both models were loaded with load_in_4bit=True, so quantization is held constant and the adapter is the only variable between the two columns.
• Tool calling, native protocol — 500 xLAM rows held out by construction. Training consumed shuffle(seed=3407)[:15000]; this set is [15000:15500] of the same shuffle, and zero id overlap with the training slice is asserted at runtime.
• Bash — InterCode-ALFA, 300 rows, executed in Docker and scored on filesystem/output equivalence. A different command with the same effect counts as correct.
• Python — EvalPlus 0.3.1, HumanEval and the hardened HumanEval+ suite, executed.
• Out-of-format probe — BFCL v3 official AST checker, pinned at commit 70b6a4a2144597b1f99d1f4d3185d35d7ee532a4, prompt mode.
• Confidence intervals — Wilson score, 95%.

Results

<img src="benchmarks.png" alt="Grouped bar chart comparing Llama-3.2-1B-Instruct against Llama Mini Agent. On tasks it was trained for: tool-call exact match 1.8% base vs 80.4% fine-tune; tool calls well-formed 8.4% vs 99.2%; Bash executed 16.3% vs 31.3%; HumanEval+ 28.7% vs 25.0%. On the off-format BFCL v3 probe: 15.4% base vs 2.8% fine-tune." width="100%"/>

The three tasks this model was trained to do, against the base model under an identical harness and identical prompts. Every figure in the chart is repeated in the tables below.

¹ Not a capability score — see Out-of-format robustness.

All three comparisons are paired — same problems, same prompts, same decoding — so they are tested with an exact McNemar test rather than by eyeballing the margins.

• Tool calling goes from unusable to reliable. The base model is not failing at tool selection so much as at format: 91.6% of its replies are prose wrapped around a differently-shaped JSON object, which a strict parser rejects. What the adapter buys is protocol reliability — 8.4% to 99.2% well-formed output.
• Bash roughly doubles, 16.3% to 31.3% (+15.0 pp). Of 300 paired rows, 60 are solved only by the adapter and 15 only by the base: exact McNemar p = 1.6 x 10^-7. This is the clearest real gain in the table.
• Python moves the other way, but not significantly. 28.7% to 25.0% is a 3.7 pp gap on 164 problems, with 15 solved only by the adapter and 21 only by the base — exact McNemar p = 0.41. The honest reading is "no measurable Python gain, and a possible small loss", not a demonstrated regression. Anyone needing Python specifically should benchmark it themselves rather than rely on a 164-problem result either way.

Tool calling, in detail

Scored on the protocol it was trained on: the training system prompt in, a JSON list of {"name", "arguments"} out.

95% CI: exact [76.7%, 83.6%], function-name [96.4%, 98.9%].

Nearly all remaining error is argument values, not tool selection — the model finds the right function 98% of the time and then occasionally fills a field wrong, frequently where the request underdetermines it:

request  daily low for 7203.T, and latest rates for JPY/EUR/GBP
produced [{"name": "stock_get_daily_low", "arguments": {"symbol": "7203.T", "region": "JP"}},
          {"name": "latest_exchange_rates", "arguments": {"resolution": "1d", "currencies": "JP,EUR,GBP"}}]
expected [{"name": "stock_get_daily_low", "arguments": {"symbol": "7203.T", "region": "S"}},
          {"name": "latest_exchange_rates", "arguments": {"resolution": "1d", "currencies": "JPY,EUR,GBP", "base": "USD"}}]

Parallel calls hold up. 276 of the 500 held-out rows require two or more calls in one reply, and accuracy barely moves:

A representative parallel case, reproduced exactly:

[{"name": "is_sum_of_cubes", "arguments": {"num": 153}},
 {"name": "longest_common_prefix", "arguments": {"strs": ["flower", "flow", "flight"]}}]

Bash, in detail

31.3% overall on executed InterCode-ALFA — but the average hides a steep gradient:

The model is a competent one-liner translator and a poor pipeline composer. It reliably produces lsof, hostname -I, vmstat, sed -n 10p file — the shape of thing its hand-written supplement and the frequency-capped NL2SH-ALFA slice actually taught it — and degrades sharply once a task needs find ... -exec, xargs, or multi-stage composition.

Measurement caveat. InterCode-ALFA's default tfidf output comparison is measured by the NL2SH-ALFA authors at 82% accuracy / 65% recall against human judgement, so it under-counts correct-but-different commands. Both models are scored under the same mode, so the comparison is fair, but the absolute number is a floor rather than a point estimate.

Out-of-format robustness

This section is a limitation probe, not a capability score. BFCL v3 prompt mode asks every model for [func_name1(params_name1=params_value1, ...)]. This model was trained for JSON, so BFCL is off-distribution for it by construction, and the result below should be read as "what happens when something else supplies the tool prompt" — not as a measure of whether it can call tools. Its actual tool-calling ability is the 80.4% above.

It matters because it is a real deployment hazard: drop this model into an agent framework that ships its own tool prompt and you will hit this, not the 80.4%. The cause is specific and diagnosable. Classifying all 1,000 non-live outputs:

expected  [light_travel_time(distance_in_light_years=4)]
produced  [func_name1(params=params['distance_in_light_years'], params=params['speed_of_light'])]

In 9.5% of rows it degenerates into unbounded nesting — [func_name1(func_name2(func_name1(func_name1(... — until the token budget runs out.

The two models fail in opposite ways, which is the interesting part:

The base model's problem is parsing — 72.4% of its answers are not valid calls at all, but when it does emit one it usually names the right function. The fine-tune's problem is naming — it emits syntactically valid calls at a far higher rate, then fills them with a function that does not exist, because it copied the placeholder out of the prompt. Narrow SFT bought a large gain inside the target format and paid for it in sensitivity to instructions about format. That trade is the honest one-line summary of this checkpoint.

---

How to prompt it

This matters more than usual here. The model was trained on three fixed system prompts and is sensitive to them. Use these verbatim.

Tool calling

You are a function-calling assistant. Available tools:
{tools_as_json}
Respond with a JSON list of calls.

replies [{"name": "...", "arguments": {...}}]

Bash

You are a Bash assistant. Translate the request into a single Bash command.

replies with a bare command — no fences, no prose

Python

You are a Python coding assistant. Solve the problem with correct, working code.

replies with a fenced python block

Do not reach for tokenizer.apply_chat_template(..., tools=[...]). That renders Llama 3.1's stock tool prompt, which asks for {"name": ..., "parameters": ...} — a different format from the one this model was trained on, and the direct cause of the results in Out-of-format robustness.

---

Development history

This checkpoint is one point in a 13-run ablation, not a single training job. All 13 runs are public, so the sequence can be inspected rather than taken on trust:

The recurring problem across the sweep was that joint tool + Bash + Python training depressed Python accuracy relative to the base model, and cleaning or re-weighting the Python slice (runs 6, 8, 9, 10) did not recover it. Run 11 is the informative one: removing Python from the mixture entirely did not lift Python out of the same band, which points at interference from the tool/Bash training pressure rather than at Python data quality. Run 12 — this checkpoint — takes the remaining untested angle, giving Python the same treatment tool and Bash already had: one clean, execution-verified source (bigcode/self-oss-instruct-sc2-exec-filter-50k) kept in full, filtered only for ast.parse validity. Run 13 isolates Python completely as a control.

Read the Python figure in the results table against that history — but note what this evaluation can and cannot support. The sweep's motivating observation was a Python regression under joint training; the 164-problem HumanEval+ comparison here shows the adapter 3.7 pp below the base, which is directionally consistent but not statistically significant (exact McNemar p = 0.41). So this run reproduces the direction, not the effect. Separating a real joint-training cost from noise at 1B needs a larger Python benchmark than HumanEval+ alone.

---

Training recipe

Ground truth from this adapter's own adapter_config.json and the training notebook.

Data mixture — each source capped at 15,000 rows, concatenated, shuffled, then filtered to <= 1024 tokens:

---

Known limitations

• Format lock-in. Outside its three training prompts, tool calling collapses — see Out-of-format robustness. If you need a model that follows arbitrary tool-calling instructions, this is not it.
• The bundled chat template cannot represent this model's output. chat_template.jinja renders a prior tool call as {"name": ..., "parameters": ...} (singular, unwrapped) and hard-fails on more than one:

  {%- if not message.tool_calls|length == 1 %}
      {{- raise_exception("This model only supports single tool-calls at once!") }}
  

  The model emits a JSON list and produces parallel calls in 55% of held-out cases. Feeding multi-turn tool history back through the template will therefore mangle the format or raise. The same template is embedded in the GGUF builds, so llama-cli --jinja inherits it. Normalize the shape yourself in the tool loop.
• Argument values. 17.6% of native-protocol replies pick the right function and get an argument wrong. Validate arguments before dispatching.
• Pipelines. Bash accuracy falls to 11% on long requests and 15.6% on find-rooted tasks.
• Python is weaker than the base model. Measured here at 25.0% vs 28.7% HumanEval+ — though at n = 164 that gap is not statistically significant (exact McNemar p = 0.41). This is the joint-training cost documented in Development history, not a data-quality defect — treat its Python output as a draft and run the tests.
• Over-eager tool calls. With tools in context, a closing pleasantry can still trigger a call instead of a plain reply. Gate the tool loop on the application side.
• 1B model. No multi-step reasoning, no long-horizon planning. Verify before executing anything it emits — especially shell commands.

---

Files

GGUF metadata: architecture llama, 1,235,814,432 params, context 131072, BOS <|begin_of_text|>, EOS <|eot_id|>.

The adapter here is byte-identical (sha256 7b8e6efd42f8...) to the one in Frost2o24/llama-3.2-1b-mini-agent-lora, which carries the adapter alone without the GGUF builds.

---

CPU performance

Measured with llama-bench on CPU only (-ngl 0, no GPU offload), 10 threads, 3 repetitions. Hardware: Intel Core i7-13620H (6 P-cores + 4 E-cores, 16 threads), laptop, Windows. pp = prompt processing, tg = token generation, both tokens/second.

Q4_K_M processes prompts 44% faster than the smaller Q3_K_M. That inversion is not a measurement error: prompt processing is compute-bound and Q4_K has better-optimised x86 SIMD kernels than Q3_K, which needs more unpacking work per weight. Token generation is memory-bandwidth-bound instead, so there the smaller file does win.

For this model that trade favours Q4_K_M, because tool-calling prompts are dominated by the tool schema. Across the 500 held-out tool-calling rows the median prompt is 327 tokens (mean 371, p90 667, max 1312) while the median reply is only 44 tokens (mean 51). So prompt processing, not generation, sets the latency:

Q4_K_M is both the quality recommendation and the fastest end-to-end for agent use, despite Q3_K_M generating 17% faster in isolation. Pick Q3_K_M only if memory is genuinely tight or your workload is long-generation rather than long-prompt.

Thread scaling (Q4_K_M): generation saturates early — 51.8 t/s at 4 threads, 57.2 at 8, and no further gain past that (55.5 at 16, where E-core contention costs a little). Prompt processing keeps scaling: 241 t/s at 4 threads to 321 at 16. If you are serving tool calls, give it all 16 threads; if you are generating long text, 8 is enough.

Reproduce:

llama-bench -m llama-3.2-1b-tool-py-bash-py-newds.Q4_K_M.gguf \
            -ngl 0 -t 10 -p 512,2048 -n 128 -r 3

---

Usage

llama.cpp

llama-cli -m llama-3.2-1b-tool-py-bash-py-newds.Q4_K_M.gguf -c 4096 -st \
  --system-prompt "You are a Bash assistant. Translate the request into a single Bash command." \
  -p "Count lines in every .py file under the current directory."

Pass the system prompt directly rather than using --jinja, given the template caveat above.

Transformers + PEFT

import json, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

REPO = "Frost2o24/llama-3.2-1b-mini-agent"
base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, REPO)
tok = AutoTokenizer.from_pretrained(REPO)

tools = [{"name": "get_current_weather",
          "description": "Current weather for a city.",
          "parameters": {"city": {"type": "str"}, "unit": {"type": "str"}}}]

msgs = [
    {"role": "system", "content":
     "You are a function-calling assistant. Available tools:\n"
     + json.dumps(tools)
     + "\nRespond with a JSON list of calls."},
    {"role": "user", "content": "What's the weather in Paris right now? Use celsius."},
]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              return_tensors="pt").to(model.device)
out = model.generate(enc, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0][enc.shape[1]:], skip_special_tokens=True))
# [{"name": "get_current_weather", "arguments": {"city": "Paris", "unit": "celsius"}}]

The system prompt is built by hand — apply_chat_template lays out the turns only, never with tools=.

Ollama

printf 'FROM ./llama-3.2-1b-tool-py-bash-py-newds.Q4_K_M.gguf\n' > Modelfile
ollama create llama-mini-agent -f Modelfile
ollama run llama-mini-agent

---

Reproducing the evaluation

# native-protocol tool calling on held-out xLAM (the primary tool-calling number)
python scripts/eval_native_tool.py --eval-target Frost2o24/llama-3.2-1b-mini-agent --n 500

# raw dumps for bash / HumanEval / the BFCL probe
python scripts/eval_model.py --eval-target Frost2o24/llama-3.2-1b-mini-agent --load-seq 4096

# official scorers
python scripts/score_icalfa.py        output/eval/bash_llama-3.2-1b-mini-agent.jsonl    # WSL + Docker
python scripts/score_humanevalplus.py output/eval/python_llama-3.2-1b-mini-agent.jsonl  # WSL
python scripts/score_bfcl_dump.py     output/eval/tool_llama-3.2-1b-mini-agent.jsonl

Swap --eval-target unsloth/Llama-3.2-1B-Instruct to reproduce the base column.

---

Intended use and limits

Good for: local and offline agents pinned to the prompts above, function-calling back-ends, shell one-liner translation, edge deployment, and as a worked example of how narrow SFT trades general instruction-following for in-format accuracy.

Not for: production automation without validation, arbitrary tool-calling protocols, multi-step reasoning, or anything safety-critical. Never execute generated shell commands unreviewed.

---

Citation

@misc{llama32_1b_mini_agent,
  title  = {Llama Mini Agent: a 1B joint tool-calling, Bash and Python fine-tune of Llama-3.2-1B-Instruct},
  author = {Frost2o24},
  year   = {2026},
  url    = {https://huggingface.co/Frost2o24/llama-3.2-1b-mini-agent},
  note   = {LoRA adapter and GGUF builds; evaluated on held-out xLAM, InterCode-ALFA and HumanEval+}
}

License

Llama 3.2 Community License, inherited from the base model. Derivatives must carry "Built with Llama". Training data licenses are those of the upstream sources (xLAM, self-oss-instruct-sc2, NL2SH-ALFA).

---

Supporting this work

The 13-run ablation above, and the evaluation sweep behind these numbers, were all produced on a single consumer laptop GPU. That is the main thing limiting how far the next version can go.

If your team has an NVIDIA DGX Spark or an AMD Ryzen AI Max+ ("Strix Halo") AI dev kit to spare, it would go directly into multi-turn tool data, the planned GRPO stage, and published eval numbers for the next generation of these models. Reach out via the model discussions tab. No obligation either way — the weights stay freely available regardless.