distilbert-trade-decision-classifier-v1

distilbert-trade-decision-classifier-v1

DistilBERT fine-tuned with LoRA r=32 for classifying user replies to trading-agent proposals into one of six decision intents. Pairs with a regex fast-path and a confirmation prompt for the bookends of a reply-routing pipeline.

How it works

Trading agents that DM proposals ("Approve / decline / hold / size N / trim N?") get free-form text replies back. This model converts the reply into one of six discrete intents so the agent can route it deterministically.

The model is invoked AFTER a fast-path regex tries the canonical phrases first ("approve", "decline", "size 10"). The regex handles routine replies; the model handles everything the regex doesn't match.

Reply text in
   ↓
Canonical-phrase regex      ← catches structured replies cheaply
   ↓ (no match)
THIS MODEL                  ← classifies into 6 intent labels
   ↓
Decision rule:
   • confidence ≥ 0.85 AND label ≠ UNCLEAR → commit
   • else                                  → confirmation prompt to the user

Labels (6)

UNCLEAR is a trained refusal label, not a fallback. The model is expected to emit it on multi-intent, ambiguous, or off-topic inputs. Treat it as the model saying "I don't know, ask the human."

Inputs

A single string with structural context tags prepended:

[dm|group][reply_to:N|no_reply_to][in_flight:K] <reply text>

• [dm] vs [group] — chat surface (DM vs group chat)
• [reply_to:N] vs [no_reply_to] — whether the user quote-replied to a specific proposal
• [in_flight:K] — number of proposals currently awaiting decision

Example inputs:

[dm][reply_to:200][in_flight:1] approve
[dm][no_reply_to][in_flight:1] dump half
[dm][reply_to:200][in_flight:2] trim at $49

The tags carry context the model can't infer from the text alone — "yes" with 1 proposal in flight is APPROVE; "yes" with 3 in flight and no quote-reply is structurally ambiguous and trained as UNCLEAR.

Usage

Python (transformers)

from transformers import pipeline

clf = pipeline(
    "text-classification",
    model="DoDataThings/distilbert-trade-decision-classifier-v1",
)
result = clf("[dm][reply_to:200][in_flight:1] dump half")
print(result)
# [{'label': 'COUNTER_SIZE', 'score': 0.991}]

Python (onnxruntime, CPU)

import onnxruntime as ort
import numpy as np
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("DoDataThings/distilbert-trade-decision-classifier-v1")
sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])

text = "[dm][no_reply_to][in_flight:1] hold off"
enc = tok(text, truncation=True, max_length=64, return_tensors="np")
logits = sess.run(
    None,
    {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]},
)[0][0]
probs = np.exp(logits) / np.exp(logits).sum()
labels = ["APPROVE", "DECLINE", "HOLD", "COUNTER_SIZE", "COUNTER_PRICE", "UNCLEAR"]
print(labels[int(probs.argmax())], float(probs.max()))
# HOLD 0.943

Deployment shape

The model is not safe to use standalone. Pair with:

• A confidence threshold (we recommend 0.85)
• Deterministic safety rails (position size, available cash, mode gate)
• A confirmation prompt for low-confidence cases

The model picks intent; the system decides whether to act. It does not have final authority over orders.

Design decisions

Narrow-waist split. The model classifies INTENT only, not proposal context. By design, upstream code disambiguates which proposal the reply targets (via quote-reply or single-default rule), and the model only sees the locked-in case. This makes the model independent of ticker / setup / portfolio specifics — its job is interpreting "what did the user mean," not "which one."

UNCLEAR as a trained refusal class. A 5-label classifier forced to pick one of {APPROVE, DECLINE, HOLD, COUNTER_SIZE} on ambiguous input is dangerous. The 6th label is the model's escape valve — it's trained on multi-intent, ambiguous, off-topic, and sarcastic inputs so it can refuse rather than guess. Combined with the 0.85 confidence threshold, this caps the blast radius of misclassification: an unsafe input either yields UNCLEAR (refusal) or a non-UNCLEAR label with low confidence (falls through to confirmation prompt).

Structural prefix as text, not special tokens. The [dm][reply_to:N][in_flight:K] tags are concatenated into the input string and tokenized as regular subword pieces. This works with off-the-shelf DistilBERT — no special-token registration, no tokenizer config drift between train and serve. The model learns the bracket conventions naturally via attention.

Six labels including COUNTER_PRICE. Earlier versions used five labels. The sixth (COUNTER_PRICE) was added because "trim at $49 instead of $48" is a fundamentally different action from "size 10" — different downstream extraction (price vs share count). Conflating them would force the consumer to disambiguate post-classification, defeating the purpose of the intent label.

Evaluation

Held-out eval set: 175 hand-curated adversarial examples, ~30 per class, zero-leakage verified against training.

Honest assessment. Zero high-confidence misclassifications on eval (no row labeled wrong at confidence ≥ 0.85). DECLINE and COUNTER_PRICE both hit perfect precision (1.000). UNCLEAR is the weakest class at F1 0.85, and the HOLD/UNCLEAR boundary on multi-intent inputs ("approve but only half") is genuinely fuzzy — these cases can be reasonably labeled either way. The 0.85 confidence threshold is calibrated so weak cases fall to confirmation rather than commit wrong.

Training

Limitations

1. Classifies INTENT only, not proposal context. The model never sees the actual proposal being responded to — upstream proposal-disambiguation must run before this model is invoked.
2. COUNTER_SIZE emits intent only; share count extraction is a separate downstream step (regex).
3. COUNTER_PRICE emits intent only; price extraction is a separate downstream step.
4. Trained on author-curated and synthetically-augmented data. Real-world reply variety may exceed training surface forms; expect ~5% of replies to fall to confirmation-prompt fallback.
5. UNCLEAR has the lowest F1 (0.85). The boundary with HOLD (active deferral vs no-position) is fuzzy on multi-intent inputs.
6. English-only. No localization in v1.

Dataset

Training and evaluation data: DoDataThings/trade-decision-classifier-v1-dataset

License

Apache 2.0.