Agents-A1-IQ2_M-GGUF

Agents-A1 — IQ2_M GGUF

InternScience's Agents-A1 (a ~35B-A3B hybrid-attention MoE agent model) quantized to a single IQ2_M GGUF (~11.7 GB) that runs on a 16 GB machine. It solves real SWE-bench Verified tasks in an agentic Claude-Code-style loop. This is InternScience's model — ours is the quant + imatrix + eval + ready-to-run Modelfiles. (No other GGUF existed when this was published, so this is the first-mover GGUF; if unsloth/others ship one, ours adds an imatrix, a documented agentic eval, and configurable-context Modelfiles.)

📦 Which quant should I pick?

This repo ships the IQ2_M — the sweet spot for a ~35B MoE: smallest that still runs real agentic loops sharply, made with an importance matrix (imatrix).

<!-- Quant names: K-quants (Q*_K) = good general quants. I-quants (IQ*) = smaller at the same quality but need an imatrix (we ship `Agents-A1.imatrix`) and a bit more CPU. Higher number = more bits = better + bigger. -->

🧮 Will it fit? (RAM/VRAM cheat-sheet)

Total memory ≈ weights (~11.7 GB) + KV-cache (grows with context).

If you OOM, lower num_ctx (see CONTEXT.md).

🧠 Context size (native 262,144 / 256K)

Configurable — Modelfiles at 8K / 32K / 128K / native are included; full guide in CONTEXT.md. 32K is the recommended default for agentic/coding work.

🚀 How to run it

# Ollama (build a 32K-context model from the included Modelfile)
ollama create agents-a1-32k -f Modelfile.32k
ollama run agents-a1-32k

# llama.cpp (use a recent build — this is a qwen3_5_moe hybrid-attention MoE)
llama-server -m Agents-A1-IQ2_M.gguf -c 32768 --jinja --temp 1.0 --top-p 0.95

Recommended sampling: temperature 1.0, top_p 0.95. This is a reasoning model — it emits a thinking channel before its answer; keep temperature high (≈1.0). Low temperature (≤0.2) tends to cause repetition loops.

Agentic use: native tool-calling works out of the box with the model's bundled chat template — pass your tools via the standard OpenAI/tools parameter (or Anthropic-style via a bridge) and it returns proper tool_calls. No text-injection workaround needed.

⚠️ Needs a recent llama.cpp. Agents-A1 is a qwen3_5_moe hybrid (gated-delta-net linear-attention layers + MoE, every-4th-layer full attention). It requires a llama.cpp build new enough to include the qwen35moe graph. Older builds will fail to load it.

⚠️ Good to know

• Strengths: agentic coding / tool use — runs clean multi-turn Claude-Code-style loops, makes decisive single tool calls per turn, doesn't degenerate. Long native context (256K). Efficient: ~3B active params (MoE) so it's fast for its size.
• Limits: this is a 2-bit quant of a small-active-params MoE — it will not match a full-precision 70B. On the hardest, multi-file SWE tasks it often explores well but doesn't always land the exact fix. Use ≥ Q4 if you have the RAM and need max fidelity (not shipped here; quantize from the source if needed).
• Quirk: emits an internal reasoning/thinking channel; if your client shows it, you can strip everything before the final answer.

📊 Evaluation methodology (MANDATORY — never lose how it was measured)

• Score: 1/5 on a fail-fast swe-mix pre-screen subset (5 SWE-bench Verified django instances). Resolved: django__django-11099 (1/1 tests). The gate to publish is resolve ≥ 1 — this build passed. All 5 ran clean agentic loops (37–39 tool calls each, 0 empty/degenerate turns, no crash or loop).
• Exact instances (pre-screen): django__django-11133 (✗), django__django-11099 (✓), django__django-13590 (✗), django__django-13933 (✗), django__django-14608 (✗). The full 30-instance discriminating probe spec is in swe_mix.json — 6 continuity anchors + 24 discriminating (10 repos); the pre-screen runs the first 5.
• Harness: Claude Code + agent-bridge.js (Anthropic→OpenAI translation, context eviction) → llama.cpp llama-server, in terminal-bench / Docker, native tools calling, 80-message cap per task.
• Sampling: temperature 1.0, top_p 0.95. Tool format: native (tools param; not bypassed — the model's chat template emits proper tool_calls).
• Quant: IQ2_M, imatrix computed on a code+general corpus (273×512-tok chunks, final PPL ≈ 1.82 on the calib set), 11.66 GB, requantized from a near-lossless Q8_0 intermediate.
• Date: 2026-06-29. Caveat: small/hard local probe, relative ranking — NOT 1:1 with the official SWE-bench leaderboard.

🗒️ Changelog

• 2026-06-29 v1: initial IQ2_M release. SWE-mix pre-screen: 1/5 (resolved django__django-11099, 1/1 tests) — gate passed. Clean agentic loops on all 5 (native tool calls, 0 degenerate turns). Probe spec in swe_mix.json.

📚 Credit & license

Model, weights, and training: © InternScience — InternScience/Agents-A1. Quant (IQ2_M) + importance matrix + agentic eval + context Modelfiles: KikoCis. apache-2.0 (same as upstream). No weights were modified beyond quantization — this is a faithful re-quant, not a fine-tune.