ThoxNeedle-Micro

ThoxNeedle-Micro

On-device function calling for THOX hardware, based on Cactus's public needle2, packaged in the .cact format the device runtime loads.

This model is THOX-trained. The weights shipped as needle2.cact differ from the upstream artifact, and the difference is measured below.

What this model is for

Local, on-device function calling for THOX hardware: given a set of THOX device tool schemas and a user request, emit the tool call that satisfies it — or emit an empty call list when no available tool can serve the request.

That second behaviour is a first-class objective, not an afterthought. On a device, a model that invents a plausible-looking call for an unservable request fires real actuators.

Training

Learning-rate sweep

Loss curve

Held-out loss by epoch: 0.1827 → 0.0450 → 0.0369

Evaluation

Scored on the 255 distinct examples held out from training (drawn from 255 rows — duplicates are scored once, not weighted by how often they repeat). Greedy decode, JAX float32. call_exact_match and refusal_accuracy are reported separately because averaging them would hide the failure mode that matters on a device.

Read these deltas with the sample size in mind. One example is 0.39 percentage points, so the only honest summary is that the fine-tune left exact-match unchanged — it neither improved nor meaningfully harmed the held-out score. The single-example movement in tool selection is not a trend this set can resolve.

Known limitation of this corpus

needle-tooluse.jsonl contains 1400 rows but only 1400 distinct (query, answer) pairs — 0.0% duplicates, with up to 1 copies of a single example.

Two consequences, both material to reading the numbers above:

1. A row-level train/test split of this file leaks. An earlier run of this pipeline split at row level and put a duplicate of 31 of its 32 held-out rows into training; those before/after numbers measured memorisation and have been discarded. The split is now group-aware and asserts zero leakage.
2. After holding out distinct examples for evaluation, only a few dozen distinct examples remain to train on. That is a hard ceiling on what any fine-tune of this corpus can demonstrate against a base model that already performs well on the task.

Deduplicating and expanding the corpus is the prerequisite for a meaningful improvement here — not more epochs, and not a different learning rate.

Did the fine-tune survive quantization?

The deployment scheme is mixed-precision at ~2.2 effective bits (embedding=4, mhc=4, default=2, group size 128). At that width a small LoRA delta can be quantized away entirely, which would be indistinguishable from training having done nothing. Both artifacts were dequantized and diffed:

• tensors changed: 135/404
• weight elements changed: 25,473,408 of 43,634,423 (58.38%)
• max relative L2 change in a tensor: 0.2212
• survived quantization: True

The exporter is bit-deterministic — two exports of identical parameters produce identical bytes — so the noise floor for this comparison is exactly zero and any non-zero delta is real.

Reproducing

Trainer, config, corpus, split, and eval are in Thox-ai/thoxneedle-trainer-src:

python -m thoxneedle.cli all --config configs/thoxneedle-micro.yaml

Lineage and licensing

Derived from Cactus-Compute/needle2, which declares Apache-2.0 for its weights; this repo carries that license forward.

Separately, and affecting the trainer rather than this artifact: the cactus-needle 2.0.0 PyPI wheel is internally inconsistent about its own license — METADATA declares Apache-2.0 while the bundled LICENSE file is MIT. THOX vendored parts of that wheel's code into its trainer and attributes against the MIT text there, since that is the document actually shipped with the code. That discrepancy does not govern the weights published here.

THOX did not use upstream's training code. Its finetune.py NaNs on the first optimiser step: the checkpoint is float16, init_lora inherits that dtype, zero-initialised B makes the step-1 gradient w.r.t. A exactly zero, and optax.adamw's default eps=1e-8 is below float16's smallest subnormal (5.96e-8) and rounds to zero — so Adam computes 0/(sqrt(0)+0) = NaN. THOX wrote its own trainer that keeps parameters and optimiser state in float32. Details: docs/UPSTREAM_NAN.md in the trainer repo.