conformer-landmark-detector-synthetic-pretrained

Conformer Acoustic Landmark Detector (synthetic pretraining)

Frame-level detector of acoustic landmarks (Stevens, 2002) in speech audio: abrupt acoustic events tied to articulatory gestures. Given a waveform, the model outputs per-frame probabilities (10 ms resolution) for 8 landmark types plus background, from which landmark times are extracted by per-class peak picking.

This is the synthetic-pretrained variant: first pretrained on ~265k synthetic English words with automatically-detected landmarks (all-words-in-english-with-pink-trombone and vtl-speech-landmarks), then fine-tuned on the real manually-annotated corpus (1,839 recordings). The base variant without pretraining is available at conformer-landmark-detector. The pretraining-only checkpoint (before fine-tuning) is included under pretrain_only/.

Landmark types

Macro F1 @ 20 ms tolerance (held-out test set): 0.784

Precision/recall/F1 are landmark-level at ±20 ms tolerance on a held-out test split (10%, speaker-stratified). Detection thresholds were tuned per class on a separate validation split (10%) — the test set was never used for model selection or threshold tuning.

Architecture

Conformer encoder (12 blocks, d_model 256, 4 heads, FF 1024, conv kernel 31) over 80-band log-mel spectrograms (16 kHz, 25 ms window, 10 ms hop), with a linear classification head over 9 classes. ~18.3M parameters. Trained with Gaussian soft labels (per-class σ: V 20 ms, G 15 ms, Fc/Fr 12 ms, Sc/Sr/Nc/Nr 10 ms), class-weighted cross-entropy and label smoothing 0.1.

Usage

With the training repository (MateoCamara/conformer-landmark-detector):

pip install -r requirements.txt
python scripts/predict.py --audio path/to/audio.wav --checkpoint mcamara/conformer-landmark-detector-synthetic-pretrained

Or programmatically:

from src.utils.hub import load_landmark_model

model, config, thresholds = load_landmark_model("mcamara/conformer-landmark-detector-synthetic-pretrained", device="cuda")
# model: LandmarkClassifier over (batch, frames, 80) log-mel features
# config: mel front-end + label map + sigmas; thresholds: per-class operating points

The output JSON of predict.py contains {type, time_ms, confidence} entries.

Files

• model.safetensors — model weights
• config.json — architecture, mel front-end, label map, soft-label sigmas
• thresholds.json — per-class detection thresholds tuned on validation
• pretrain_only/model.safetensors — checkpoint after synthetic pretraining, before fine-tuning

Citation

@inproceedings{camara2026conformer-landmarks,
  title={Conformer-based Acoustic Landmark Detection},
  author={C\'amara, Mateo},
  booktitle={Interspeech},
  year={2026}
}

References

• Stevens, K. N. (2002). Toward a model for lexical access based on acoustic landmarks and distinctive features. JASA, 111(4), 1872-1891.
• Gulati, A., et al. (2020). Conformer: Convolution-augmented Transformer for Speech Recognition. Interspeech 2020.