AuroraGPT-Recall

AuroraGPT-Recall (700M)

AuroraGPT-ToolFix with cross-turn memory repaired.

The previous models in this family had a specific, measurable defect: a fact stated in one user turn did not survive into the next turn. Say "my name is X", ask "what's my name?" one turn later, and the model would confabulate. This model fixes that.

The bug, measured

Before the fix, on the base model:

So it was never a context-length problem. The 2,048-token window and the attention itself were fine — per-token loss improves monotonically all the way to position 2048. The <\|end\|><\|assistant\|>...<\|user\|> turn boundary was the wall. Almost all of the original SFT was single-turn (multi-turn data was ~2% of the mix), so the model had learned to treat each user turn as a fresh start.

Results

Scored on 12 held-out cases whose wordings and answer strings appear nowhere in the training data (Wendeline, zibbly-warp, QQ7731, flimberwock, ORANGE-DELTA, Thistledown, ...). Verified zero leakage against all 67,701 training examples, so this measures generalisation rather than memorisation.

It also lost personality, and that is a real cost. 31% of the training mix was rehearsal, but I wrote that rehearsal in plain prose instead of sampling the parent model's own voice, so the style did not survive:

For everyday chat, use AuroraGPT-ToolFix instead. Use this model when you specifically need it to remember things across turns.

<|user|>My passphrase is zibbly-warp.<|end|><|assistant|>Noted.<|end|><|user|>Say my passphrase back.<|end|><|assistant|>
-> "Your passphrase is zibbly-warp."

Honest limitations

• Personality regressed badly. Emoji use went from 5-in-8 replies to zero, math lost its step-by-step working, and chat replies became clipped. The cause is a rehearsal design error: rehearsal targets should be sampled from the parent model's own outputs so its voice is reproduced exactly. Hand-writing them in neutral prose teaches the model that neutral prose is the target.
• Training acks leaked into chat. "Noted." and "Got it." were used as the assistant's mid-conversation reply in ~47k examples, so they now surface as generic chat openers ("Noted, so you can work on something you're doing"). Mid-conversation acks in the history should have been masked or varied far more.
• 3/12 held-out cases still fail. Cross-turn recall is much better, not solved.
• 1/4 spurious tool fires remain. The tool LoRA in the parent model over-fires on words like "code"; this reduces it but does not eliminate it.
• This is a patch, not a root-cause fix. The defect was introduced two stages upstream, in the full SFT where multi-turn data was ~2% of the mix. A rank-16 adapter on the far end mitigates it; rebuilding the chat SFT with ~25% multi-turn data would fix it properly.
• Code generation was not re-measured. The parent scored 3.66% on HumanEval (chat-formatted) versus 9.15% for AuroraGPT-Qwen-Distill. If you want code, use that one. Nothing here is expected to have improved it.
• Training data was procedurally generated, so phrasings are templated. Training loss fell to ~0.0002 (perplexity 1.0002), which on its own is meaningless — the held-out score above is the only number that shows it generalised.

Prompt format

Not ChatML:

<|system|>{system}<|end|><|user|>{user}<|end|><|assistant|>{reply}<|end|>

Tool call / readback:

<tool_call>
{"name": "web_search", "arguments": {"query": "..."}}
</tool_call>

<|user|><tool_response>
{"result": "..."}
</tool_response><|end|>

The GGUF ships with a tool-declaring chat template embedded, so on-device apps show the tool picker (a plain merge inherits a template without tools/tool_calls and the picker silently disappears).

Files

• model.safetensors — fp16, 707M params
• AuroraGPT-Recall.tools.Q8_0.gguf — 753 MB, tool template embedded, eos = <|end|>

Training

LoRA r=16, alpha=32 on all attention + MLP projections, merged into the base. 67,701 examples (~4.0M tokens), 2 epochs, lr 1e-4, bf16 on one A100, ~10 minutes. Data mix: 69% multi-turn recall (fact recall across turns, distractor turns, multi-fact, verbatim rare strings, instruction persistence, corrections, anaphora, tool-result recall) and 31% rehearsal (tool calls, math, identity, chat, plus negative examples that must not fire a tool).

Built by UltraLabs on free compute.