astraforge-8b-TCR

astraforge-8b-TCR v2: Tool-Calling, Retrieval and Calculator Reasoning (LoRA on Llama-3.1-8B)

Developed by 79Labs · Version 2.0.0 (2026-10-05)

This repository was replaced in place. v1.0.0, the earlier adapter with its own model card and benchmarks, stays available under the git tag v1.0.0: PeftModel.from_pretrained(base, "79Labs/astraforge-8b-TCR", revision="v1.0.0"). v2 uses a different tool-call format (see Prompt format), so update your parser when you upgrade.

astraforge-8b-TCR is a LoRA adapter for meta-llama/Llama-3.1-8B-Instruct. It does three things: choosing and calling tools from a large catalogue, including many calls in one turn; deciding when to retrieve (rag_search) and answering from what came back; and reasoning step by step with a calculator tool.

• Base: Llama-3.1-8B-Instruct (context 131,072)
• Adapter: LoRA r=16 on all attention and MLP projections, 160 MB
• Training: 33,250 rows, 1 epoch, max length 65,536, lr 1e-4. The data is synthetic: generated questions over a 101-tool console catalogue (strict-grounded, so no argument the question does not state), 20k RAG rows (5k call-only), and 5,250 reasoning rows in no-tool and calculator-tool forms. It contains no real user conversations.
• Checkpoint: step 2,079, the final one. It was chosen by benchmark, not by eval loss: the trainer's lowest eval loss was at step 100, and that checkpoint scored worse.

Results

Every model was prompted in its own native tool format. The Claude and DeepSeek rows were run through the same harness, parser and splits. Raw outputs are in benchmarks/.

Tool calling, eval v2 (strict-grounded, leak-free; n=60 per split)

Each cell is first-call accuracy / multi_recall. First-call accuracy means the first tool named is the right one. multi_recall is the share of the reference calls the model actually made. Turns in this catalogue average about 6 calls, so read multi_recall alongside first-call accuracy.

• heldout_real: real console tools never seen in training.
• alien_synth: synthetic tools from a different generator.
• On seen_real, Claude makes more of the reference calls (0.965 vs 0.901). That is the one tool cell where it leads.

Retrieval (RAG eval, n=600)

No external model was run on this set.

Reasoning (8 quantitative families, n=160)

This is the adapter's weak axis. Use the calculator path, and do not rely on unaided multi-step arithmetic. The 9B sibling family (Qwen3.5-based) reaches 0.950 / 1.000 on the same set.

Not re-measured for v2

v1's confirmed_first (asks before acting, 0.89), elicitation and GSM8K (0.75) figures came from a different harness and were not re-run on v2, so this card claims nothing about them. If those behaviours matter to you, test them, or pin v1.0.0.

Prompt format

v2 is trained on the template in chat_template.jinja, and you must pass it. Llama-3.1's built-in template rejects assistant turns that carry more than one tool call, and this model makes several per turn. Tool calls come out as XML:

<atem:function_calls>
<atem:invoke name="get_quote">
<atem:parameter name="symbol">NSE:RELIANCE</atem:parameter>
</atem:invoke>
<atem:invoke name="get_news">
<atem:parameter name="query">Reliance Industries</atem:parameter>
</atem:invoke>
</atem:function_calls>

Parse the invoke blocks and coerce each parameter to its schema type.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base = "meta-llama/Llama-3.1-8B-Instruct"
tok = AutoTokenizer.from_pretrained("79Labs/astraforge-8b-TCR")   # loads chat_template.jinja (the ATEM template)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "79Labs/astraforge-8b-TCR")

messages = [{"role": "user", "content": "Price and latest news for Reliance?"}]
tools = [...]   # OpenAI-style JSON schemas
ids = tok.apply_chat_template(messages, tools=tools, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=False))

Limitations

• Reasoning without a tool is weak (0.706). Route arithmetic through a calculator tool.
• The tool benchmarks use one product's catalogue, a financial-markets console. Expect weaker results on catalogues that look very different, and evaluate on your own tools.
• With 1,000+ tools the catalogue will not fit in one prompt: 1,000 full schemas come to about 145k tokens against a 131k context. Shortlist with a retriever first.
• English only. Not financial advice.

Versions