techjam-scaffolding-tagger

Scaffolding tagger for conversational product search

A binary token classifier that separates product content from conversational scaffolding in a shopper's message. It is an optional preprocessing step in a conversational e-commerce search agent: it strips filler so that a deterministic, catalogue-grounded miner sees only the product-bearing text.

It does not retrieve products, rank them, decide what is valid evidence, or generate text. Every phrase it exposes is still checked against a frozen product catalogue before it may be used.

Labels

0 SCAFFOLD   conversational framing: greetings, politeness, hedging, meta-talk
1 CONTENT    product-bearing text: attributes, categories, requirements

What it is for, and what it is honestly worth

The host agent has a literal recognition gate for the message shapes its simulator emits. When a message does not match, wording has changed and the template parser is blind. This tagger removes the reworded framing so the miner can work on what remains.

Measured on held-out reworded wrappers, it retains 99.25% of the canonical constraint slots — it finds the product text reliably.

It is not, on its own, sufficient, and that is worth stating plainly. In the host agent the miner it feeds recovered 0.00% of short constraint values, because that miner keeps an n-gram only when the phrase is distinctive enough, and attribute values like cotton or zipper closure are far too common to survive. The tagger cleans the text and the miner then discards exactly the part that mattered. What closes that gap is an exact catalogue-dictionary lookup, not this model.

Beside that lookup, the tagger's end-to-end contribution is small: +0.0012 on a reworded condition and +0.0000 on every clean evaluation. It ships enabled because it costs nothing where the score is decided, runs locally with no network, and is mildly positive on compound perturbations — not because it carries the workload.

Intended use, and the gate around it

Call it only on messages a literal recognizer has already failed to match. In the host agent that is enforced by control flow: on clean traffic the gate matches 463 of 463 messages, so the model is never constructed and the score is unchanged by construction.

The host also refuses degenerate output — if the tagger would strip a message to fewer than two words, or below 15% of its length, the original text is used instead. A tagger that deletes everything is a failure mode, not a strong prior.

Training

• Base: distilbert-base-uncased
• Data: synthetic, catalogue-derived constraint messages with controlled scaffolding transformations, generated from participant-visible data only
• Domain-adaptive masked-language-model pretraining on the catalogue: tested and rejected, it added no value
• Training program and evaluation record are in the host repository's experiment registry

Limitations

• English only. Product-search phrasing specifically.
• Trained on synthetic scaffolding transformations. It generalises across the perturbation families we generated; that is a weaker claim than generalising to real users, and we do not make the stronger one.
• Useful only in front of an exact-match layer. On its own it improves almost nothing, as measured above.
• ~265 MB, requires PyTorch and Transformers. Every failure path — missing dependency, corrupt weights, inference error, degenerate output — returns the original message.

License

Apache 2.0, inherited from distilbert-base-uncased.