astraforge-70b-TCR

astraforge-70b-TCR — Tool-Calling & Retrieval Agent (LoRA on Llama-3.3-70B)

Developed by 79Labs · Version 1.0.0 · Changelog

astraforge-70b-TCR (Tool-Calling / Retrieval) is a LoRA adapter for meta-llama/Llama-3.3-70B-Instruct specialised for reliable agentic tool use: retrieving the right tool from a large catalog (RAG), eliciting missing parameters, confirming before acting, emitting schema-valid calls, and staying grounded — while preserving the base model's reasoning. It is not a general capability upgrade; it is a focused, measurable improvement on the agentic behaviours that make a tool-using assistant trustworthy in production.

• Base: Llama-3.3-70B-Instruct (4-bit QLoRA)
• Adapter: LoRA (r=16), ~828 MB
• Training: continued SFT on 1,006,113 agentic examples; eval-gated early stop (best val loss ≈ 0.1198)
• Serving: load the adapter with PEFT / Unsloth on the base model (see Usage)

What the evidence supports (and what it doesn't). On an in-house agentic benchmark this model matches the base on reasoning (GSM8K) while substantially improving tool-calling correctness and learning a confirm-before-call discipline the base lacks. The full results table, run log, and machine-readable scores are included in benchmarks/ so every number here is verifiable. Sample size is stated (N=100); treat small-N differences as indicative, not leaderboard-grade.

---

What it does

Trained across 35 business domains (sales/CRM, finance, support, logistics, healthcare, HR, IT, …) on these agentic behaviours:

• Tool calling in OpenAI function-call JSON — the right tool, schema-valid arguments.
• Elicitation — when required parameters are missing, it asks instead of hallucinating them.
• Confirm-before-call — for consequential actions it asks the user to confirm, then calls.
• Multi-document RAG — pick the relevant document, ground the answer, cite it.
• ReAct / planning — reason → act → observe chains; plan before acting.
• Guardrails — never call an undeclared tool; refuse when out of scope / unsure.
• Analytics reasoning — arithmetic + stats (median / variance / %-change / forecast) over supplied data.

Intended use & scope

Use it for building tool-using / function-calling agents where reliability of the agentic protocol (right tool, ask-then-act, confirm, don't hallucinate tools) matters — especially private / on-prem deployments where a hosted frontier API isn't an option.

Do not expect frontier general intelligence. This is a 70B open model specialised on a narrow skill set. For open-ended coding or research, use a larger / code-specialised model.

---

Evaluation

Comparable open models under identical settings: 4K context (max_seq_len=4096, also AstraForge's trained/served window), greedy decoding, 2048-token budget so reasoning models finish, answer extracted after any </think> and from \boxed{} where present, N = 100. Each model is prompted in its own native tool format (its tokenizer chat template) so none is penalised for a foreign format.

Δ vs base: reasoning +0.00, tool-correct +0.27, confirmed-first +0.94. Evidence: benchmarks/nway_results.json, benchmarks/nway_run.log, methodology in benchmarks/BENCHMARK_METHODOLOGY.md.

Reading it honestly:

• Confirmed-first favours this model by design — that is the point, not a trick. The other models were never trained/prompted to confirm before acting, so their score is near zero. The claim is "this model learned a confirm-before-call protocol," not "other models are bad at agents." If your application doesn't want a confirmation step, weight this metric accordingly.
• Tool-correct is the fairer head-to-head: whether the model ultimately emits a valid call to the right tool. The +0.27 vs base reflects genuine specialisation.
• Reasoning is a control, not a headline — the goal was no regression; GSM8K parity shows the finetune didn't lobotomise general ability.
• N = 100 is indicative, not leaderboard-grade. Treat a few points as noise.

External benchmark: BFCL v4

BFCL v4 simple_python, Prompt mode, N=400: 51.00% (204/400).

One correction, because the earlier internal summary of this run was misleading and we would rather say so than quietly restate it: an aggregate of "0.43% overall" was computed at the time. That number averages eleven test categories that were never generated as zero. Only simple_python was ever run, and on it the model scored 51.00%. The aggregate was an artifact of the harness, not a measurement of the model.

Function-calling (FC) mode genuinely does score 0.00% — a tool-template defect on our side, still open. The remaining BFCL categories, τ-bench and API-Bank are still unrun.

For comparison on the identical 400 cases, our 8B sibling (79Labs/astraforge-8b-TCR) scores 37.50%, with 198 of its 250 misses being parse failures rather than wrong tool choices.

---

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "meta-llama/Llama-3.3-70B-Instruct"
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto", load_in_4bit=True)
model = PeftModel.from_pretrained(model, "79Labs/astraforge-70b-TCR")
tok = AutoTokenizer.from_pretrained("79Labs/astraforge-70b-TCR")

tools = [{
    "type": "function",
    "function": {
        "name": "book_flight",
        "description": "Book a flight for a traveler.",
        "parameters": {"type": "object",
            "properties": {"traveler_name": {"type": "string"}, "origin": {"type": "string"},
                           "destination": {"type": "string"}, "depart_date": {"type": "string"}},
            "required": ["traveler_name", "origin", "destination", "depart_date"]}}}]
msgs = [{"role": "user", "content": "Book Ada a flight from SFO to JFK on 2026-08-01."}]
ids = tok.apply_chat_template(msgs, tools=tools, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(input_ids=ids, max_new_tokens=256)[0][ids.shape[1]:], skip_special_tokens=True))

With Unsloth: FastLanguageModel.from_pretrained("79Labs/astraforge-70b-TCR", load_in_4bit=True).

---

Training

• Data: 1,006,113 deduplicated agentic examples, self-verified by construction (every positive passes an oracle; hard negatives fail it) across 35 domains + tool catalogs, continuous elicitation → confirmation → call, multi-doc RAG, ReAct, plans, guardrails, and real + synthetic reasoning. (Training data is not distributed with this repo.)
• Method: continued SFT (warm-started from a prior SFT champion), 4-bit QLoRA (r=16), max_seq_len=4096, LR 1e-5, effective batch 16, Unsloth gradient checkpointing.
• Governor: held-out validation with early stopping (load_best_model_at_end) — stopped at convergence, best checkpoint at val loss ≈ 0.1198.
• Hardware: single NVIDIA GB10 (Grace-Blackwell, 128 GB unified memory).

Limitations & risks

• Context: 4K trained / 128K max. The base supports 128K, so it runs at any context ≤128K, but the agentic behaviours were reinforced within ~4K — keep the working context (system prompt + retrieved tools + dialogue) within ~4K for best fidelity. Pairs naturally with tool-RAG.
• 70B open model — below frontier on general tasks; specialised, not general-purpose.
• Synthetic eval tools — real deployments must wire real executors and keep confirm-before-call and schema-validation guardrails in the harness, not rely on the model alone.
• Confirmation / elicitation phrasing is English-centric. Inherits base-model biases.

License

Governed by the Llama 3.3 Community License (inherited from the base model).

Citation

@misc{astraforge70b_tcr_2026,
  title  = {astraforge-70b-TCR: Tool-Calling and Retrieval Agent (LoRA on Llama-3.3-70B)},
  author = {79Labs},
  year   = {2026},
  note   = {Continued SFT on 1M agentic examples; eval-gated. Benchmark harness + raw evidence included.},
  url    = {https://huggingface.co/79Labs/astraforge-70b-TCR}
}