whisper-preprocess

name: whisper-preprocess metadata: version: 1.0.0 description: "Audio preprocessing and transcription pipeline using ffmpeg + OpenAI Whisper. Use this skill whenever the user wants to transcribe audio or video files (lectures, meetings, podcasts, interviews), improve audio quality before transcription, remove silence from recordings, or mentions Whisper, speech-to-text, or transcricao. Also trigger when the user has MKV, MP4, WAV, M4A, or other media files they want converted to text. This skill handles the full pipeline: audio extraction, silence removal, voice enhancement, segmentation, and Whisper transcription. Works 100% offline."

Whisper Preprocess Pipeline

You are setting up and running an audio preprocessing + transcription pipeline. This pipeline solves four critical Whisper problems:

1. Hallucinations during silence — Whisper invents text like "Obrigado", "E ai", "Subscribe" during quiet segments
2. Hallucination cascade in long audio — after the first hallucination, Whisper repeats it for the rest of the file
3. Poor transcription from noisy audio — background noise reduces accuracy
4. Silent failures on Windows due to encoding — the whisper CLI crashes when printing accented/CJK characters in a cp1252 console, producing empty .txt files with no clear error

The pipeline runs 100% offline when the Whisper model is already cached locally — only needs ffmpeg, Python, and the downloaded model.

Critical lessons learned (from real-world testing)

These are hard-won insights from iterating on a 2h13min lecture recording and a multilingual (PT+JA) meeting. Follow them carefully:

1. NEVER use afftdn (FFT noise reduction)

The afftdn filter creates tonal artifacts ("musical noise") that sound like constant beeping/whistling. This completely destroys the audio for Whisper — every segment will transcribe as garbage. The same applies to lowpass=f=8000 which can interact badly with other filters.

Safe enhancement chain: highpass=f=100, acompressor, loudnorm — nothing else.

2. Remove silence BEFORE enhancement

The loudnorm filter normalizes everything to -16 LUFS, including silence. If you run silenceremove after loudnorm, it finds nothing to remove because the "silence" has been boosted. Always: extract → remove silence → enhance.

3. Silence threshold should be -30dB, not -40dB

Most recordings have background noise above -40dB. At -40dB threshold, silenceremove finds zero silence. Use -30dB as default. For noisier recordings, try -25dB.

4. Segment long audio (mandatory for >10 minutes)

Whisper hallucinates on long audio files. A 2h13min file produced only ~19 lines of real content before devolving into repeated "Obrigado" or "E ai" for the remaining 2 hours. Segmenting into 10-minute chunks solved this completely — each segment produced ~6-7K characters of accurate transcription.

5. Use --condition_on_previous_text False

Without this, once Whisper starts hallucinating, it conditions on its own hallucinated output and the error cascades. With False, each segment starts fresh. This doesn't prevent hallucinations entirely (segmentation does that), but limits the damage.

6. Compress output audio to OPUS (listening copy, decoupled — see lesson #9)

Enhanced WAV files are enormous (2.6 GB for 2h). The listenable *_enhanced.opus is now encoded at 48 kHz, 64 kbps, -application audio (wideband, fuller voice). The old -application voip is narrowband telephony tuning and was dropped. Bitrate/sample-rate are configurable via --listen-bitrate / --listen-sr.

9. NEVER feed the listening copy through silence-removal + dynamic AGC (anti-"picotamento")

Symptom: the voice in *_enhanced.opus is choppy / cuts between syllables ("picotando"), worst for a low-volume or impaired (dysarthric, slow) speaker.

Cause: the listenable file used to inherit the whole transcription chain — silenceremove (cuts quiet speech) + acompressor release=50ms (pumps every syllable gap) + single-pass loudnorm (dynamic AGC that "breathes" on pauses). Three gain-modulation/chopping sources stacked.

Fix (now built in): the listening copy is decoupled from the transcription path and generated by build_listenable() from the original file:

• wideband (48 kHz), no silence removal by default → continuous audio, the listener follows every pause and nothing is clipped;
• stable gain only — slow-release compressor (release=400ms) + fixed makeup + true-peak alimiter with lookahead. No dynamic AGC. Verified A/B on a real recording: a 2-pass loudnorm linear=true reverted to "Dynamic" on clipping sources, so it is NOT used here.
• limiter ceiling has headroom (--listen-limit 0.6) for Opus inter-sample overshoot → final true-peak stays below 0 dBFS (the old recipe actually clipped at +0.7 dBFS).

The transcription path is unchanged in spirit (16 kHz, silenceremove at -30dB, then enhance) — only made gentler: detection=rms, window=0.025, stop_silence=0.5, stop_duration=2.0, which also helps Whisper on slow speakers. Lessons #1 (no afftdn) and #3 (-30dB) still hold; denoise on the listening copy is opt-in and off by default (--denoise).

7. UTF-8 everywhere on Windows (handled automatically)

On Windows the default console encoding is cp1252. When whisper CLI prints transcription progress containing accented or CJK characters it crashes with UnicodeEncodeError, and the Python subprocess.run reader thread separately crashes with UnicodeDecodeError on bytes > 0x7F. Result: all segments come back empty with the misleading log line "Segmento vazio ou falhou". The pipeline now sets PYTHONIOENCODING=utf-8, reconfigures sys.stdout/stderr, and forces encoding="utf-8", errors="replace" on every subprocess.run — no user action needed. If you ever invoke whisper standalone on Windows, set $env:PYTHONIOENCODING="utf-8" first.

8. For multilingual recordings, run two passes and merge

A single --language choice loses everything spoken in another script. Run a primary pass in the dominant language, a secondary pass in the second language, and merge the SRTs choosing the secondary version where the script ratio is high. See "Multilingual recordings" below — the script does this for you with --secondary-language.

The Pipeline (6 stages)

Input (MKV/MP4/etc.)
  |
  v
[1] Extract audio → WAV 16kHz mono
  |
  v
[2] Remove silence (transcription path, on RAW 16kHz, before enhancement)
  |  silenceremove threshold=-30dB, duration=2.0s, padding=0.5s, detection=rms
  v
[3] Enhance audio (transcription path)
  |  highpass=100Hz → acompressor → loudnorm EBU R128
  |  NO afftdn, NO lowpass (they cause artifacts)
  v
[4] Segment into ~10min chunks → cached by input hash
  |
  v
[5] Transcribe each segment with Whisper
  |  temperature=0, condition_on_previous_text=False
  |  optional 2nd pass in --secondary-language + merge
  |
  |  (PARALLEL, decoupled) build_listenable() from the ORIGINAL file →
  |  48kHz, NO silence removal, stable gain (slow compressor + true-peak limiter,
  |  no dynamic AGC) → continuous, no "picotamento" — see lesson #9
  v
[6] Compress listening copy to OPUS (48k, application=audio) + cleanup
  |
  v
Output: *_transcricao_<lang>.txt/.srt, *_transcricao_merged.txt, *_enhanced.opus

Prerequisites

ffmpeg -version        # audio processing
whisper --help         # transcription (pip install -U openai-whisper)
python --version       # 3.9+

Check required ffmpeg filters:

ffmpeg -filters 2>/dev/null | grep -E "silenceremove|loudnorm|highpass|acompressor"

How to use

Copy scripts/whisper_preprocess.py to the user's working directory, then:

# Basic — full pipeline with auto-cleanup
python whisper_preprocess.py "lecture.mkv"

# With context for proper nouns and technical terms
python whisper_preprocess.py "lecture.mkv" --initial-prompt "Seminario de Teologia, professor Carlos"

# Test preprocessing only (listen before transcribing)
python whisper_preprocess.py "lecture.mkv" --only-preprocess

# Keep intermediate files for debugging / segment reuse
python whisper_preprocess.py "lecture.mkv" --keep-intermediate

# CPU only (no GPU)
python whisper_preprocess.py "lecture.mkv" --device cpu

# Better quality model (needs ~10GB VRAM)
python whisper_preprocess.py "lecture.mkv" --model large-v2

# Noisier recording — more aggressive silence detection
python whisper_preprocess.py "lecture.mkv" --silence-threshold="-25dB"

# Quiet / impaired (slow, low-volume) speaker — louder, fuller listening copy
python whisper_preprocess.py "call.m4a" --listen-makeup 7 --listen-comp-threshold="-24dB"

# Longer segments for very clean audio
python whisper_preprocess.py "lecture.mkv" --segment-minutes 15

# Multi-idioma (ex.: reuniao PT + JA com merge automatico)
python whisper_preprocess.py "meeting.mkv" \
  --language Portuguese --secondary-language Japanese \
  --initial-prompt "Reuniao tecnica sobre UPS, participantes Kusaba, Murata, Heitor" \
  --secondary-initial-prompt "Kusaba Murata UPS battery substation"

# Modo offline (modelo ja baixado, sem internet)
python whisper_preprocess.py "lecture.mkv" --offline

# Forcar reprocessamento ignorando cache de segmentos
python whisper_preprocess.py "lecture.mkv" --force-reprocess

Important: when passing negative thresholds, use = syntax: --silence-threshold="-30dB" (not --silence-threshold "-30dB", which argparse interprets as a flag).

Multilingual recordings

Use --secondary-language when the audio mixes two languages (e.g., a Portuguese meeting with Japanese speakers, or English narration with Spanish quotes). The script will:

1. Preprocess once
2. Transcribe all segments in --language (primary)
3. Transcribe the same segments in --secondary-language
4. Merge the two SRTs into a unified *_transcricao_merged.txt. The merge prefers the primary version, but for cues where the secondary script (kanji/hiragana, cyrillic, etc.) appears prominently in both passes (sign of foreign speech mistranscribed phonetically), it substitutes the secondary version prefixed with a tag like [JA].

Outputs in this mode:

• *_transcricao_pt.txt / .srt — primary pass
• *_transcricao_ja.txt / .srt — secondary pass
• *_transcricao_merged.txt — unified, with [JA] markers

The merge logic also lives in scripts/merge_passes.py and can be called standalone:

python merge_passes.py --primary meeting_pt.srt --secondary meeting_ja.srt \
  --output merged.txt --secondary-script japanese --marker-tag JA

Supported scripts: japanese, chinese, korean, cyrillic, arabic, greek, hebrew, thai. For anything else pass a regex via --secondary-script-regex.

Offline use

When --offline is passed (or env var WHISPER_OFFLINE=1 is set), the script:

• Sets HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1 so Whisper won't touch the network
• Pre-validates that the model is in local cache; fails fast with a clear path if it isn't

Whisper caches models in:

• ~/.cache/whisper/<model>.pt (classic models: tiny, base, small, medium, large-v1, large-v2, large-v3)
• ~/.cache/huggingface/hub/models--openai--whisper-large-v3-turbo/snapshots/ (turbo)

To pre-download a model while online:

python -c "import whisper; whisper.load_model('turbo')"
# or
python -c "import whisper; whisper.load_model('large-v2')"

You can also pass --model-dir DIR to use a portable copy of the models.

Segment cache (reuse across runs)

Intermediate processing is cached per-input. The script computes a short hash from the input's path + size + mtime and uses it to name the intermediate directory (e.g., meeting_intermediate_a1b2c3d4/). On subsequent runs with the same input (e.g., changing only --secondary-initial-prompt), the extract → silence → enhance → segment pipeline is skipped and the existing segments are reused.

To force a clean reprocess, pass --force-reprocess.

Combined with --keep-intermediate, this lets you iterate cheaply on prompts or experiment with alternate secondary languages without redoing the heavy ffmpeg work.

All parameters

Output files

Intermediate files are cached under <input>_intermediate_<hash>/ and removed at the end unless --keep-intermediate is used (or until --force-reprocess discards the cache).

Whisper model recommendations

Prefer large-v2 over large-v3 — v3 has more hallucination issues for non-English languages.

Troubleshooting