laya-kannaka-supersession

laya-kannaka-supersession

A Laya checkpoint (ModernBERT-large, 421M, non-autoregressive typed decisions) fine-tuned on the write-time supersession decision of a memory substrate:

noul — Does the later statement update, correct or replace a fact stated in the earlier statement?

State: {"earlier": "[date] role: text", "later": "[date] role: text"}. Answer: P(yes) in the noul field. Both statements in true chronological order, so date alone separates nothing.

Second trained reflex for Kannaka; experiments E-L3b → E-L3d in kannaka-bench (experiments/laya_reflex/). Sibling: laya-kannaka-evidence-gate.

Two revisions

The first revision was precise on pair evaluation and disastrous at the write path's base rate. The current one keeps the same recall with 13× fewer false stamps, expires no current fact, and lifts answers above the no-supersession control instead of two questions below it. What it does not do: catch every superseded fact — 19 of 51 training-set true pairs are not in a top-5 shortlist at all, and the reflex catches 13 of 17 held-out ones.

Later runs (2026-09-23, not pushed as revisions)

• E-L3e — the main model at write-path shortlist k=10 and k=20 (top-5 was used above): k=10 gives the same 14/17 answers with 48 stamps; k=20 expires one current fact and drops answers to 13/17. Widening the shortlist is not the bottleneck.
• E-L3f — retrained on the union of the e-l3b and main negatives: held-out pair AUROC 0.853, precision 1.00 / recall 0.65, 24 stamps, 11/17 true catches, answers 13/17. A loss; weights not kept.
• E-L3g — the main recipe cross-validated over all 78 LongMemEval knowledge-update questions (5 folds): 31/70 true supersessions caught, 0 current facts expired, 66 false stamps; answers 55/78 vs plain 54/78. Null; the line is parked in favour of a recall-side rule (kannaka-memory #1044).

Full write-ups in kannaka-bench RESULTS.md.

Use

import laya
agent = laya.Agent("flaukowski/laya-kannaka-supersession")
res = agent.predict(
    {"earlier": "[2023/05/23] user: I set a personal best of 27:12 in the charity 5K.",
     "later":   "[2023/05/30] user: I'm hoping to beat my personal best of 25:50 this time."},
    {"supersedes": {"type": "noul",
                    "instructions": "Does the later statement update, correct or replace a fact stated in the earlier statement?"}},
)
print(res["answers"]["supersedes"]["noul"])   # P(yes)

At the write path: recall the top-k earlier memories for the new one, ask this per pair, and stamp expires_at on the earlier memory at P(yes) >= 0.5. Gate the stamps on nothing else; a same-speaker gate removes a third of false stamps for one lost catch, a cosine floor loses recall.

Training (current revision)

build_e_l3d_dataset.py: chronological walk of each training question's haystack, MiniLM top-5 earlier neighbours per turn, label 1 only for the (old fact, new fact) pair, 40 negatives per positive (half hardest-by-cosine, half uniform), positives ×6. Upstream RLCD recipe on one GPU (train_e_l1b.py): 6 epochs, micro-batch 8, grad accumulation 4, fp16. RTX 5090, 364 s.

Limits

• 17 held-out positives; treat AUROC as ±0.06 and the answer delta (+1 question) as inside noise.
• English only, 1024-token context, statements cut at 1,500 characters.
• Trained on chat turns (LongMemEval); a store of terse notes is a different distribution.

Provenance

Base model Apache-2.0 (convaiinnovations). Training data derived from LongMemEval (Wu et al.), used under its license for research. Produced 2026-09-23 by Kannaka Labs; wins and losses of the surrounding experiments are in kannaka-bench RESULTS.md.