HPA-RMVPE

Pitch Detection Algorithm Benchmark Report

Benchmark Methodology

Evaluation Setup

This benchmark evaluates pitch detection algorithms across multiple datasets with different characteristics, including synthetic and real audio from speech and music domains. Each algorithm is tested on noisy audio generated by mixing clean datasets with CHiME background noise at various signal-to-noise ratios (10-30 dB) and voice gain variations (-6 to +6 dB).

Performance Metric Definition

The Overall Performance Rankings show the Harmonic Mean (HM) score as percentages, computed from six complementary components:

HM = 6 / (1/RPA + 1/CA + 1/P + 1/R + 1/OA + 1/GEA)

Where:

• RPA (Raw Pitch Accuracy): Fraction of voiced frames within 50 cents of ground truth
• CA (Cents Accuracy): exp(-mean_cents_error/500), penalizing larger deviations exponentially
• P (Voicing Precision): TP/(TP+FP), fraction of predicted voiced frames that are truly voiced
• R (Voicing Recall): TP/(TP+FN), fraction of truly voiced frames detected
• OA (Octave Accuracy): exp(-10×octave_error_rate), robustness against octave errors
• GEA (Gross Error Accuracy): exp(-5×gross_error_rate), penalizing deviations >200 cents

Speed Benchmark Details

CPU timing measurements are performed on 1-second audio signals at 22.05 kHz sample rate with 256-sample hop length. The reported CPU Time (ms) represents the average processing time per 1-second audio segment across multiple runs. Relative Speed shows performance relative to CREPE as the baseline algorithm.

Optimal Threshold Analysis

The Optimal Threshold refers to the voicing confidence threshold that maximizes the Harmonic Mean score. Algorithms test multiple thresholds (0.0 to 1.0 in steps of 0.1) and select the one yielding the highest combined score. CV stands for Coefficient of Variation (std/mean), measuring consistency across datasets.

Dataset Descriptions

The benchmark evaluates algorithms across diverse datasets covering speech, music, synthetic, and real-world conditions:

Key Characteristics:

• Synthetic datasets provide perfect ground truth but may lack real-world complexity
• Real datasets capture natural acoustic variations but have imperfect ground truth annotations
• Speech datasets focus on vocal pitch tracking challenges
• Music datasets encompass instrumental and vocal music scenarios
• SpeechSynth addresses the gap of lacking synthetic speech data with accurate pitch labels

Overall Performance Rankings

No speed benchmark results found.

Detailed Performance Analysis

Voicing Detection Performance

Measures how well algorithms distinguish between voiced (pitched) and unvoiced (unpitched) audio segments.

Pitch Accuracy Metrics

Detailed pitch estimation accuracy across different error types and magnitudes.

Additional Metric Definitions:

• RCA (Raw Chroma Accuracy): Fraction with correct pitch class (note name), ignoring octave
• Cents Error: Mean absolute pitch deviation in cents (raw error, before exponential transform used in CA)
• RMSE: Root Mean Square Error in Hz

Pitch Contour Smoothness

Measures the temporal stability and continuity of pitch tracks.

Metric Definitions:

• Relative Smoothness: Coefficient of variation of consecutive pitch changes (std/mean of relative frame-to-frame changes)
• Continuity Breaks: Fraction of ground-truth voiced segments where predicted voicing has gaps
• Overall Smoothness Rank: Average rank across both smoothness metrics (1=best, lower is better)

Optimal Threshold Analysis

Voicing confidence thresholds that maximize overall performance scores.

Algorithm Consistency

Measures performance stability across different datasets using Coefficient of Variation (CV = std/mean).

Performance by Dataset Subsets

By Origin

• Synthetic: Bach10Synth, MDBStemSynth, SpeechSynth, NSynth
• Real: MIR1K, PTDB, PTDBNoisy, Vocadito

By Domain

• Speech: PTDB, PTDBNoisy, SpeechSynth
• Music: Bach10Synth, MDBStemSynth, NSynth, Vocadito, MIR1K

By Cross-Dimension

• Synthetic + Speech: SpeechSynth
• Synthetic + Music: Bach10Synth, MDBStemSynth, NSynth
• Real + Speech: PTDB, PTDBNoisy
• Real + Music: Vocadito, MIR1K