int_wakeup-whisper_base-ctx20-ft4

int_wakeup-whisper_base-ctx20-ft4

Decides, turn by turn, whether a voice assistant should respond - without requiring a wake word. It reads the current utterance together with up to 20 preceding turns, so it can accept a follow-up like "and the one after that?" and reject a question that two people are asking each other.

Trained and evaluated on TCLResearchEurope/intelligent_wakeup v1.0.1.

Results

Test split: 243 conversations, 3,647 scored turns, 6.72 h of recording of which 5.28 h is speech not addressed to the assistant. Threshold 0.90, chosen on validation and applied unchanged.

False accepts per hour are over the 5.28 h of not-addressed speech. Over the whole 6.72 h recording the figure is 2.8/hour.

By turn type:

direct and contextual are all positives, so only recall is meaningful for them; non-assistance is all negatives.

Architecture

24.0M parameters.

1. Per-turn encoder - Whisper-base encoder (6 layers, 512-d) over the turn's audio, then learned attention pooling to one 512-d vector.
2. Context transformer - 4 causal layers, 256-d, over the last 20 turn vectors plus a speaker-role embedding per turn. Turn t attends to 0..t and never to the future.
3. Heads - a trigger logit, and an auxiliary turn-type head used only as a training signal.

Trained in two phases: Phase 1 with the encoder frozen and turn embeddings precomputed (3.4M trainable), then Phase 2 with the top 4 encoder layers unfrozen (16.0M trainable). Phase 2 is what the -ft4 suffix means, and it is where the accuracy comes from - it lifted direct recall from 0.867 to 0.937 by teaching the encoder to actually hear the wake word.

Two encoders at inference - please read

Phase 2 fine-tuned the encoder for the current turn while the context turns kept being read from a cache built with the frozen encoder. The model therefore expects those two to be different, and inference has to reproduce it:

Encoding everything with the fine-tuned encoder - the natural thing to write - raises no error and drops validation F1 from 0.896 to 0.718, with contextual recall collapsing from 0.759 to 0.290. inference.py does it correctly; start from that rather than reimplementing.

This asymmetry is a training artefact rather than a design choice, and it is the first thing to fix in a retrain.

Usage

pip install -r requirements.txt
python inference.py session.wav turns.json

turns.json is the turn segmentation - this model classifies turns, it does not find them, so bring your own VAD or diarisation:

[{"start": 0.0,  "end": 4.2,  "speaker": "Tariq"},
 {"start": 4.2,  "end": 9.8,  "speaker": "Sigma"},
 {"start": 9.8,  "end": 13.1, "speaker": "Priya"}]

Audio is 16 kHz mono. Use the literal name "Sigma" for the assistant's own turns - they form context but are never scored. Speaker ids are roles, not identities: the assistant is 1, humans are 2, 3, 4… by order of first appearance within the conversation. inference.py handles that mapping.

It is a causal model, so the same code shape works for streaming: encode each turn as it ends, keep the last 20 context vectors, classify.

Training

Needs the training repo, which carries the data pipeline. The recipe is one config:

python precompute_embeddings.py 'splits=[train,validation,test]' \
  dataset.embedding_cache_dir=.cache/embeddings-v2
python train.py +experiment=int_wakeup-whisper_base-ctx20-ft4 \
  dataset.embedding_cache_dir=.cache/embeddings-v2

training_config.yaml here is that experiment file; resolved_config.yaml is the exact config this checkpoint was produced with.

The settings that mattered, measured:

Phase 2 delivers most of its gain in the first epoch or two after the switch. One caveat: this run was still improving when it hit its 50-epoch limit (0.881 → 0.889 → 0.891 → 0.899 over its last four epochs), so it is not a converged number.

The embedding cache must be one your own run built. It stores the output of a randomly-initialised pooling layer, so a cache from a different run is a different vector space - the training repo pins the pooling head to the cache directory and refuses to mix them.

Limitations

• contextual recall is 0.759. Follow-ups with no wake word are the whole remaining recall problem, and nothing tried so far moved them.
• 3.6 false accepts/hour is not low enough for always-on use. A living room would see ~30 spurious activations a day.
• The corpus is synthetic - TTS voices over mixed room tone, one generation pipeline, English only. Splits are grouped so scenario variants never straddle them, but test is still unseen conversations, same distribution. Expect materially worse numbers on real recordings; treat these as a ceiling.
• Turn segmentation is assumed. Errors in your VAD land directly on this model's input.
• The wake word is the fixed string "Sigma", baked into the training data.

Citation

If you use this code or dataset, please cite:

@inproceedings{sowanski2026intelligentwakeup,
  title={Training Intelligent Voice Assistant Wakeup with Controllable Synthetic Conversations},
  author={Sowa{\'n}ski, Marcin and Leszczy{\'n}ski, Kacper and Krzywicki, Kacper and Wodnicki, Krzysztof},
  year={2026},
}