shiur-review-whisper

Shiur Review with Whisper

This repository is a Hugging Face Space that uses OpenAI Whisper to review shiur audio with:

• timestamped transcription
• clickable transcript playback
• local topic outlines
• auditable terminology post-processing

The implementation is optimized for real shiur evaluation on GPU hardware. The expected baseline is a Hugging Face GPU Space running Whisper large-v3.

What is implemented

• Whisper transcription with word_timestamps=True
• VAD-based pre-chunking for long files using webrtcvad
• resumable batch checkpoints for long-form runs
• segment-level clickable review UI in Gradio
• local topic outline generation using multilingual embeddings with a lexical fallback
• JSON, VTT, and TSV exports
• glossary prompting plus a small rules-based post-processing layer for Torah terminology
• automatic glossary file loading from config/torah_glossary.yaml

Recommended Hugging Face runtime

• SDK: Docker
• Hardware: Nvidia L4 or A10G
• Default model: large-v3
• Fallback model when latency matters more than quality: turbo

Environment variables

• WHISPER_MODEL: Whisper model name, defaults to large-v3
• DEVICE: auto, cuda, mps, or cpu; defaults to auto
• MAX_AUDIO_MINUTES: upload ceiling in minutes, defaults to 180
• OUTPUT_ROOT: local artifact root, defaults to outputs/ inside this repo
• PERSIST_OUTPUTS: if true and /data exists, write artifacts to /data/shiur-review
• CHECKPOINT_ROOT: optional long-run checkpoint directory; defaults to <output-root>/checkpoints
• MODEL_DIR: optional model cache directory
• DEFAULT_GLOSSARY: optional newline-delimited glossary text applied to every run
• GLOSSARY_FILE: glossary file path, defaults to config/torah_glossary.yaml
• OUTLINE_EMBEDDING_MODEL: defaults to sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• WHISPER_LANGUAGE: optional language hint used for the first batch and then reused
• WHISPER_TASK: defaults to transcribe
• WHISPER_BEAM_SIZE: defaults to 5
• WHISPER_BEST_OF: defaults to 5
• WHISPER_CONDITION_ON_PREVIOUS_TEXT: defaults to true
• WHISPER_CARRY_INITIAL_PROMPT: defaults to false
• WHISPER_TEMPERATURE_START: defaults to 0.0
• WHISPER_TEMPERATURE_INCREMENT: defaults to 0.2
• WHISPER_MAX_TEMPERATURE: defaults to 1.0
• VAD_ENABLED: defaults to true
• VAD_AGGRESSIVENESS: defaults to 2
• VAD_FRAME_MS: defaults to 30
• VAD_PADDING_MS: defaults to 300
• VAD_MERGE_GAP_MS: defaults to 700
• VAD_MIN_SPEECH_MS: defaults to 400
• VAD_MAX_SEGMENT_SECONDS: defaults to 240
• BATCH_MAX_SPEECH_SECONDS: defaults to 900
• BATCH_MAX_SEGMENTS: defaults to 48
• RESUME_ENABLED: defaults to true

How long files work now

The app no longer sends a long shiur as one opaque transcription request.

1. It probes the source file with ffprobe.
2. It converts the audio to mono 16 kHz PCM for VAD analysis.
3. webrtcvad finds speech regions.
4. Nearby regions are merged and very short ones are dropped.
5. Very long regions are split into bounded segments.
6. Those speech regions are grouped into batches, defaulting to about 15 minutes of speech each.
7. Each batch is transcribed with absolute clip timestamps, so the final transcript still aligns to the original audio.
8. After every batch, a checkpoint JSON file is written.
9. If the run stops, the next run can resume from the checkpoint.

Best-practice evaluation workflow

1. Start with large-v3.
2. Keep task=transcribe for mixed-language shiur audio.
3. Add a language hint only when the dominant language is stable and obvious.
4. Expand config/torah_glossary.yaml with real names, seforim, masechtos, and recurring transliterations from your corpus.
5. Leave VAD enabled for long files.
6. Compare large-v3 and turbo on the same clips only after you have a stable glossary and checkpointed batch flow.

Local smoke test

1. Install ffmpeg.
2. Create a Python 3.11 environment.
3. Install dependencies from requirements.txt.
4. Run python app.py.

Local runs now save artifacts under outputs/ in this repo by default.

On Apple Silicon, DEVICE=auto will prefer mps, which is a better local evaluation path than plain CPU. For real mixed-language accuracy bakeoffs, a dedicated Nvidia GPU is still the target reference environment.

CPU execution is only intended for smoke tests. Use a GPU runtime before judging quality on mixed English, Hebrew, Yiddish, or Aramaic clips.

Recommended workflow right now

Given the current Hugging Face billing block on paid GPUs, the practical setup is:

1. use the private Hugging Face Space as a deployment and UI smoke-test target
2. run serious transcript evaluation locally on Apple Silicon with mps
3. only switch back to Hugging Face GPU once paid credits are available

If you want an on-demand Nvidia environment without enabling Hugging Face paid GPU, use a temporary external GPU instance instead. The repo now includes a provider-agnostic workflow in docs/external-gpu.md plus helper scripts in scripts/verify_gpu_host.sh and scripts/run_gpu_container.sh.

If you run the private Hugging Face GPU Space through an API client, use scripts/run_space_transcription.py to copy the returned artifacts into outputs/remote-runs/ in this repo instead of leaving them in Gradio's temp cache.

Deployment notes

The Docker image is designed for local Docker and external GPU runs and exposes port 7860. On Hugging Face, use /data for persistent server-side artifacts. Browser and Gradio client downloads still land on the client machine unless you copy them into the repo yourself.

Next evaluation tasks

• compare large-v3 and turbo on the same representative shiur clips
• expand the glossary with names, seforim, and recurring transliterations
• decide whether segment-level review is sufficient before adding word-level clickable rendering