prosody-adaptation-encoder

CASPER Prosody Encoder

This repository contains the pretrained Phase-1 prosody encoder used in the prosody-or-adaptation project.

The encoder was trained on CASPER to learn frame-level prosodic representations from speech. Its 64-dimensional hidden representation is used as the frozen auxiliary prosodic input in the Phase-2 ASR experiments.

Model

The model takes 16-kHz speech as input and operates on 80-dimensional log-Mel features.

Architecture

• 80-dimensional log-Mel input
• Convolutional encoder with 128 channels
• Bidirectional GRU with hidden size 64
• 64-dimensional prosodic representation
• Prediction heads for five frame-level prosodic targets

The five training targets are:

1. log F0
2. voicing
3. delta log F0
4. log energy
5. spectral tilt

Only the learned hidden prosodic representation is used as the auxiliary representation in the downstream ASR experiments.

Checkpoint

The released checkpoint corresponds to the seed-1 CASPER Phase-1 training run.

• Training epochs: 50
• Batch size: 32
• Learning rate: 0.001
• Weight decay: 0.01
• Gradient clipping: 1.0
• Mixed precision: enabled
• Early-stopping patience: 10
• Selection metric: validation_total_masked_loss
• Target normalization: training-set statistics

The full training configuration is provided in training_config.yaml.

The checkpoint is the original model artifact from the Phase-1 experiment. For model definitions, preprocessing, and experiment code, see the GitHub repository:

https://github.com/Ki-Woong95/prosody-or-adaptation

Files

prosody_encoder_casper.pt   # pretrained Phase-1 checkpoint
training_config.yaml        # training configuration
SHA256SUMS                  # checkpoint integrity hash
README.md                   # model card

Intended Use

This checkpoint is intended primarily for:

• reproducing the Phase-2 experiments in the prosody-or-adaptation project;
• extracting learned frame-level prosodic representations;
• research on explicit prosodic conditioning for speech recognition;
• research on prosodic representation learning.

It is a research artifact rather than a general-purpose speech recognition model.

Training Data

The prosody encoder was trained on CASPER.

The training data are not redistributed in this repository. Users must obtain the underlying dataset separately and comply with its original license, access conditions, and terms of use.

Dataset paths in training_config.yaml have been replaced with portable placeholders; dataset and normalization hashes are retained for reproducibility.

Reproducibility

The accompanying configuration records the model architecture, training hyperparameters, random seed, dataset hashes, normalization hash, and checkpoint selection criterion used for the released model.

The implementation and full experimental pipeline are available at:

https://github.com/Ki-Woong95/prosody-or-adaptation

The released checkpoint is preserved as the original experimental artifact. It was produced with an earlier version of the Phase-1 implementation, so some parameter names differ from those used in the current source tree. See the GitHub repository README for checkpoint compatibility details.

License

The released model weights are provided under CC BY-NC-SA 4.0.

The CASPER training data are not redistributed and remain subject to their original license, access conditions, and terms of use.

Citation

If you use this model or the accompanying code in academic work, please cite the associated paper once the final bibliographic information is available.

Until then, please cite the project repository:

Moon, K.-W. & Brenner, D. Prosody or Adaptation?
https://github.com/Ki-Woong95/prosody-or-adaptation