tensorcode-chatbot-hotpot-001

TensorCode evidence-conditioned chatbot (HotpotQA, oracle passages)

A TensorCode Chatbot: an owned sequence encoder, relational slot workspace and language decoder. It was initialized from google/flan-t5-small at revision 0fc9ddf78a1e988dac52e2dac162b0ede4fd74ab and fine-tuned on 256 HotpotQA questions, with the annotated (oracle) supporting passages supplied as evidence.

Revisions and compatibility

Pin the commit hash shown in this repository's history for reproducible loading. The PyPI 0.4.0a4 wheel's package files are identical to source 090ebc4, which was used for training and the checks below.

This revision is a retrain, not a conversion. The bounded update changes the forward computation, so the original weights cannot be relabeled for the new architecture. The original revision is unchanged and stays available at its pin.

Training

• Script: examples/train_chatbot.py at 090ebc4, with its defaults: 3 epochs, batch 4, AdamW (foundation lr 3e-5; workspace/projection/gate lr 1e-3), gradient clip 1.0, seed 7, max 512 input / 64 target tokens, 8 workspace slots, 2 steps, memory_mode: contextualized_evidence, untied LM head.
• The weights are those after the final epoch. No checkpoint or run was selected on held-out data.
• Hardware: one NVIDIA GB10 (CUDA 13.0, torch 2.14.0). Training plus evaluation took 32.6 s.
• Epoch training losses: 0.7308, 0.5275, 0.4226. The memory gate went from 0.0100 to 0.0462 (the residual scale is tanh(gate) times the native encoder RMS).
• CUDA training is not bitwise deterministic. Rerunning the same command gave weights that differ by at most 2e-4 per element, with identical held-out predictions and metrics equal to within 1e-6.

Data

• hotpotqa/hotpot_qa at revision 1908d6afbbead072334abe2965f91bd2709910ab, distractor configuration.
• 256 training questions from train-00000-of-00002.parquet and 64 held-out questions from validation-00000-of-00001.parquet. The source IDs are listed in evaluation.json.
• Frozen JSONL sha256: train 0baba8ad845b07a0b3a91fe8c872846791e5cd8829789bfdbcee17a333c6b932, held-out 2b4380bf6d38bc7dab6013c03e880ea2f88805f9872dd71b943563b82ccd3895. These are the same files as the original revision.
• Input format: Question: {q}\nEvidence:\n{title}\n{all sentences} for each annotated supporting title, joined by blank lines. Target: the official answer. Evidence is oracle-selected: no retrieval is learned or evaluated.

Evaluation (same 64 held-out questions, same scorer as the original)

The scorer uses greedy decoding (up to 64 new tokens) and SQuAD-style normalization before exact match (EM) and token F1. CE is mean teacher-forced cross-entropy.

Workspace contribution: none was reliably measured. Active and bypassed decoding differ on 2 of the 64 questions:

• Active is right and bypass is wrong on one (1838 vs 1917).
• Bypass has the higher F1 on another (Chief of Protocol of the United States vs Secretary of State, gold Chief of Protocol).

Teacher-forced cross-entropy is lower with the workspace bypassed. A 1-question EM difference on 64 questions is within noise. Most of the gain over the base model comes from fine-tuning the FLAN-T5 foundation. The zero-evidence collapse shows the model depends on the supplied evidence. It does not show the workspace is useful.

Seed sensitivity: seeds 8, 9 and 10 were run with the same recipe after this candidate was fixed. They were for characterization only, not selection. Every run used the same 64 held-out questions.

In all four seeds, teacher-forced CE was lower with the workspace bypassed.

Files

• evaluation.json: every held-out prediction (before training, after, workspace bypassed, evidence zeroed) with the scores above.
• training-config.json: exact command, source commit, script hash and environment.
• attempts.md: the attempt log, written before the candidate run; every run is listed there.
• replacement-record.json: all runs (candidate, determinism rerun, seeds 8-10), the original revision's metrics and the qualification checks.

Persistence checks

These were run on the local artifact before upload, in a fresh process, on CUDA and on CPU:

• All 208 stored tensors are bitwise equal after load and after save_pretrained → from_pretrained.
• The configuration is canonical-JSON equal.
• All 64 greedy generations (active, bypass and zero) equal the predictions recorded in evaluation.json.

Limitations

• This is a narrow demonstration of question answering given evidence. It does not show general conversation, retrieval, multi-hop search or autonomous cognition. Instruction behavior is inherited from FLAN-T5-small.
• The held-out set is small (64 questions): one answer is 1.56 EM points.
• The model does not verify its generated claims. It can answer confidently and wrongly even when the evidence is correct.
• Published weights contain no runtime conversation. HotpotQA is CC BY-SA 4.0; anyone who supplies training examples must license and review them.

from tensorcode.tools.chatbot import Chatbot
model = Chatbot.from_pretrained("jacob-valdez/tensorcode-chatbot-hotpot-001")  # bounded architecture
answer = model.generate_batch(["Question: ...\nEvidence:\n..."])[0]
# Original unbounded weights: revision="d74b40142c6e416cdc096f54e6d9d8c1de465568" with TensorCode 6607a8b