FireRedVAD-ESP32-P4

FireRedVAD Models for ESP32-P4

Converted FireRedVAD models optimized for ESP32-P4 microcontrollers. All models use the custom .frvd binary format with native RISC-V PIE (Position Independent Execution) acceleration.

Source Code: https://github.com/Strg-Alt-Entf-0x00/firered-vad-esp32-p4

Quick Start

cd examples/console_vad
pip install huggingface-hub
python download_models.py

idf.py build flash monitor

firevad> vad_model_list
firevad> vad_model_load stream-vad/int8-ch/firered-stream-vad-int8-ch.frvd
firevad> vad_infer_mic

Available Models

stream-vad/ — Real-time streaming (causal, 10ms latency)

Best for real-time voice activity detection. Model is fully causal — no future context. Runs in ~4.5ms per 10ms frame on ESP32-P4 @ 400MHz.

vad/ — Offline batch VAD (non-causal, 1-second chunks)

Uses bidirectional context. Higher accuracy than stream-vad, but adds latency. Not suitable for real-time streaming.

aed/ — Audio Event Detection (Speech / Music / Singing)

Multi-class audio classifier. Identifies speech, music, and singing simultaneously.

Quantization Explained

Why INT8-CH (Per-Channel) is Recommended

Standard per-tensor INT8 quantization assigns one global scale factor per weight matrix. DFSMN architectures have wide variance in weight distribution across output channels — a single scale factor cannot capture this range accurately, causing silent accuracy loss.

Per-Channel INT8 (int8-ch, Version 4 in the .frvd format) assigns one scale factor per output channel. This preserves near-FP32 accuracy at INT8 speed and memory cost.

Real-time budget for 10ms frames: 10ms. Anything above 10ms (>100% load) causes audio drops.

🎤 Audio Frontend & AGC

We achieved a highly robust audio frontend using the INMP441 I2S microphone combined with a custom Automatic Gain Control (AGC) and DC-Offset High-Pass filter:

• Dynamic Range Compression: Without AGC, voice volume drops steeply from -27 dB (at 10cm) to -47 dB (at 2m). With AGC enabled, the volume stays rock-solid between -30 dB and -37 dB!
• Distance Boost: The AGC automatically boosts weak signals at a 2-meter distance by +10 dB, while seamlessly throttling loud, up-close speech.
• SNR: Crystal clear 23 dB SNR.
• Golden Test: Bit-parity and VAD-quality mathematically verified against original PyTorch outputs.

Hardware Requirements

• MCU: ESP32-P4 (RISC-V dual-core, 400MHz)
• PSRAM: 32 MB
• Flash: 16–32 MB
• RAM at runtime: ~150 KB
• Microphone: INMP441 or equivalent I2S digital microphone @ 16kHz

Note: INT8 and INT8-CH models use ESP32-P4 PIE vector instructions (esp.vmulas.s8.xacc etc.) with mandatory 16-byte memory alignment, handled automatically by the runtime. FP32/INT16 models work on other ESP32 variants (S2, S3) but without PIE acceleration.

Known Limitations (Honest Assessment)

1. Noise sensitivity: Performance degrades in low-SNR environments (loud machinery, strong wind). False positive rate increases at SNR < 5dB.
2. Microphone dependency: Model was trained on clean 16kHz PCM. A high-quality I2S microphone with hardware PGA gain control is required for reliable results.
3. No built-in noise suppression: The ESP-IDF runtime does not include NS/AEC. Echo cancellation is available via the shared APLL (I2S0 + I2S1 synchronized clocking).
4. APLL sharing warning: When both TX and RX I2S ports are active, the ESP32-P4 APLL runs at 8,191,999 Hz instead of 8,192,000 Hz (1 Hz deviation). This is hardware-expected behavior, not a bug. Both ports share the same clock, which is ideal for AEC.

.frvd File Format

Custom binary format, version-tagged in header byte [4..7]:

Header (32 bytes):
  [0..3]   Magic: "FRVD"
  [4..7]   Version: 1=fp32, 2=int8, 3=int16, 4=int8-per-channel
  [8..11]  Model type: 0=VAD, 1=Stream-VAD, 2=AED
  [12..15] Total parameter count
  [16..23] DFSMN block count + DNN layer count
  [24..31] Reserved

Architecture Metadata (32 bytes):
  Input dim, hidden size, projection size, output dim,
  lookback order/stride, lookahead order/stride

CMVN block:
  dim (uint32) + means[dim] (float32) + istd[dim] (float32)

Layer data (sequential):
  Per tensor: CRC32 name hash + element count + [scale per channel for int8-ch] + data

Conversion Pipeline

Original FireRedVAD PyTorch checkpoints -> .frvd:

# Requirements
pip install torch kaldiio numpy

# Stream-VAD INT8-CH (recommended)
python tools/converter/export_weights.py \
    --model-dir tools/original_models/Stream-VAD \
    --output-dir examples/console_vad/converted_models/stream-vad/int8-ch \
    --model-type stream-vad \
    --quantize-int8-per-ch

# Stream-VAD INT8
python tools/converter/export_weights.py \
    --model-dir tools/original_models/Stream-VAD \
    --output-dir examples/console_vad/converted_models/stream-vad/int8 \
    --model-type stream-vad \
    --quantize-int8

# Verify conversion
python tools/converter/verify_conversion.py \
    --frvd examples/console_vad/converted_models/stream-vad/int8-ch/firered-stream-vad-int8-ch.frvd

License & Attribution

Original Models

• FireRedVAD by Xiaohongshu (FireRedTeam) — Apache 2.0
• Source: https://github.com/FireRedTeam/FireRedVAD
• HuggingFace: https://huggingface.co/FireRedTeam/FireRedVAD

ESP32-P4 Port

• FireRedVAD-ESP32-P4 by Strg-Alt-Entf-0x00 — Apache 2.0
• Repository: https://github.com/Strg-Alt-Entf-0x00/firered-vad-esp32-p4

Citation

@misc{fireredvad-esp32p4,
  title={FireRedVAD for ESP32-P4: Optimized Voice Activity Detection for Embedded Systems},
  author={Strg-Alt-Entf-0x00},
  year={2026},
  howpublished={\url{https://github.com/Strg-Alt-Entf-0x00/firered-vad-esp32-p4}},
}