phoneme-wake-word

Wake Words Without Training — an open-vocabulary wake word engine for microcontrollers

A wake word here is not a model. It is ~50 bytes of configuration.

One small streaming phoneme recognizer is trained once on generic speech and never sees a wake word. Any phrase — typed, invented, whatever you like — becomes a detector in milliseconds via grapheme-to-phoneme lookup, optionally sharpened by five spoken examples. The whole runtime fits on an ESP32-S3 at 340 KB INT8 and 14 ms per 40 ms of audio, leaving wake words changeable without retraining, reflashing, or a cloud round-trip.

---

How it works

Runtime lane (wake-word-agnostic, always on): mic → 40-band log-mel + causal EMA normalization → streaming causal TCN → CTC phoneme posteriors every 20 ms → keyword-filler Viterbi decoder

Enrollment lane (per word, no training): text → G2P → phone sequence and/or 5 spoken examples → phoneme decode → pronunciation variants → automatic per-variant threshold calibration → config, or an explicit refusal if the phrase cannot be separated from ordinary speech.

The decoder is a filler-normalized Viterbi search with four gates, each closing a failure mode we hit in practice:

1. Phone-frame normalization — score is normalized by frames spent in emitting states only. Normalizing by total duration lets degenerate paths idle in blank states (free in silence) and fire rhythmically on nothing. Fixing this moved a clean operating point from 65% recall @ 12 FA/h to 95% @ 1.25 FA/h.
2. Strong-evidence gate — ≥50% of phone frames must have their phone within 1.5 nats of the frame's best class. Noisy audio yields flat posteriors that score fine on average while containing nothing; this cut false alarms on noise-degraded speech from ~100/h to ~3/h.
3. Duration bounds — 60–200 ms per phone. At a looser cap we observed 1.5–2.3 s alignments crawling across background television.
4. Acoustic energy veto (on device) — the matched span must exceed the rolling noise floor by 5 dB.

Detections use only within-frame logit differences, so the deployed engine never computes a softmax.

Models

Student architecture — causal TCN: stride-2 stem (k=5), 8 depthwise-separable causal blocks (k=5, dilations 1,2,4,8 ×2, 192 ch, BatchNorm+ReLU, residual), 1×1 head over 40 classes (39 stress-free ARPAbet phones + CTC blank). 20 ms output frames, ~2.4 s receptive field. Every op is BatchNorm-foldable conv / ReLU / residual — chosen so post-training INT8 survives intact (99.3% frame-argmax agreement with float).

Training — CTC over phonemized transcripts (CMUdict + neural G2P fallback) on LibriSpeech 960 h + 1.10 M Common Voice 17 English clips (~1,500 h, incl. 83k Indian-accented). On-GPU augmentation: synthetic room impulse responses (T60 0.1–0.6 s), additive noise (5–30 dB SNR), same-batch babble (10–25 dB), random gain, SpecAugment. The student was then distilled from the teacher with speech-weighted KD — frames are weighted by 1 − p(blank) because ~75% of CTC frames are blank and uniform KD otherwise spends its budget teaching silence (uniform KD degraded the student; the weighted version improved it).

Benchmarks

Ten phrases, Piper LibriTTS-R positives verified by Whisper (raw TTS is unreliable: "tornado" → "Pornado", "dakota" → "Decoder"), against 1.22 h LibriSpeech dev-clean + 0.82 h Common Voice dev negatives. Noisy conditions degrade positives and negatives identically so operating points stay condition-matched.

Safety property: calibration refuses phrases it cannot separate. "norman" (one phone from "normal") was rejected for 5/6 voices and "dakota" for 6/6 — independently flagged by the text-only phrase scorer (score_phrase.py) before any audio existed.

Deployment

engine_c/ is ~300 lines of dependency-free C with one INT8 ring buffer per layer, so each frame costs only its own ~350k MACs — no window recomputation. On ESP32 the binding constraint was memory latency, not math: moving weights out of memory-mapped flash (28.6 → 15.9 ms) and filling internal SRAM before PSRAM (→ 14.2 ms) mattered far more than loop optimization.

Files

phoneme_tcn_student.pt     deployable model (float, PyTorch state dict)
phoneme_tcn_teacher.pt     distillation teacher
model_int8.h               INT8 weights + layer table (C header, 340 KB)
act_scales.json            activation scales from PTQ calibration
frontend_data.h            mel filterbank + Hann window (exact training values)
engine_c/                  streaming INT8 engine + decoder + mel frontend (C)
phoneme_engine/            PyTorch model, decoder, enrollment, quantization, scorer
examples/                  a universal wake word config (~50 bytes of JSON)

Usage

Spot a typed phrase (Python):

import torch
from phoneme_engine.model import PhonemeTCN
from phoneme_engine.features import LogMel
from phoneme_engine.decoder import KeywordSpotter

model = PhonemeTCN().eval()
model.load_state_dict(torch.load("phoneme_tcn_student.pt",
                                 weights_only=True)["model"])
frontend = LogMel().eval()

spotter = KeywordSpotter("hey orbit")      # G2P -> HH EY AO R B AH T
with torch.no_grad():
    logp = torch.log_softmax(model(frontend(wav)).float(), 2)[0].numpy()
for frame, score in spotter.run(logp):
    print(f"detected at {frame * 0.02:.2f}s (score {score:.2f})")

Score a candidate phrase before committing to it:

python score_phrase.py "vucano"
# low reliability: /v/ is acoustically weak and often lost on small mics

A wake word config, in full:

{"phrase": "neptuno",
 "spotters": [{"phones": ["N","EH","P","T","UW","N","OW"],
               "source": "dictionary", "threshold": -2.5}]}

On a microcontroller: compile engine_c/ with model_int8.h and frontend_data.h, feed it 40-band mel frames, and pass the logits to pww_spotter_step(). Wake words are uint8_t arrays of phone ids plus a float threshold — swap them at runtime.

Limitations

• The 358k student is the binding constraint. Text-only speaker-independent recall is modest; 40% PER on real-world speech leaves little margin for accented, distant, or noisy input. Personal enrollment recovers much of it.
• Benchmarks are TTS-based (Whisper-verified, but synthetic). Human multi-speaker evaluation is future work.
• Onset phonetics dominate word quality. Nasals/stops/sibilants (N, M, K, T, S) work well; weak fricatives (V, F, TH, H) are often lost on small MEMS mics. The included scorer predicts this from text.
• False-alarm rates are measured against continuous speech — a worst case versus mostly-quiet rooms.
• Single-microphone. Far-field performance is a hardware question (beamforming arrays), not a decoder one.
• English only, though nothing in the architecture is language-bound beyond the phone inventory and G2P.

License

Apache-2.0. Trained on LibriSpeech (CC BY 4.0) and Common Voice 17 (CC0).

Citation

@misc{wakewordswithouttraining2026,
  title  = {Wake Words Without Training: Open-Vocabulary Wake Word
            Creation from Text and a Few Examples},
  author = {IOTEverythin},
  year   = {2026},
  url    = {https://huggingface.co/IOTEverythin/phoneme-wake-word}
}