An AI dubbing platform that keeps every speaker's own voice, built solo and shipped.
An AI dubbing platform I built solo for a media studio: a GPU pipeline that separates, transcribes, diarizes, translates and re-voices a video unattended, and a browser console where volunteer linguists fix what the machine got wrong. Put into production in September 2026 for studio staff and volunteer linguists: 19 projects, 7,559 translated lines.
RoleProduct · ML pipeline · Backend · Frontend · Ops
19 projects · 189 min of video · 7,559 translated lines
1 Claude and Gemma evaluated in native-speaker blind tests.
The full storyRead the full storyCollapse
The pipeline
Demucs splits vocals from music; WhisperX transcribes with word timestamps; pyannote diarizes with a second model checking the speaker count and ECAPA voiceprints correcting labels. Sentences are re-cut at speaker turns, a clean reference clip is chosen per speaker, and Qwen3-TTS voices each line several times. Takes pass a naturalness gate (UTMOS ≥ 3.8), a voiceprint match, gender and duration checks; failed lines retry with new references. Lines are fitted to their slots, mixed with the original music, and muxed back into the 4K source.
Translation
Three passes, not one. A global read produces the program type, register, speaker profiles and a glossary; chunked translation works under a per-line character budget derived from each line's duration; a native-editor pass polishes; only lines still over budget get shortened. Six target languages with dialect rules. Production runs Qwen3.5-27B locally on llama.cpp with four parallel slots after native-speaker blind comparisons: roughly $0.50 of GPU time per language per video instead of $7–11 in API calls. A malformed model reply now splits and retries instead of failing the language.
The review console
One timeline lane per speaker, waveforms on every clip, drag, trim and snap with a live speed readout capped at what export allows. Each line shows the original and the translation, both editable; fixing a misheard word re-translates that line in every language. Re-voicing a line takes seconds and lands as a temporary take you accept or reject. A notes box asks "What should be better about this line?" and keeps the answer for training. Right-click adds speech the transcription missed. One green Done per language; downloads live on the project page. Every control has a plain-English tooltip.
What it took to make voices stay put
The first dub put the wrong voice on 19.7% of lines with 14 gender flips. Diarization looked guilty, but three independent speaker systems agreed; the real causes were reference clips contaminated by another speaker's interjection and TTS drift on short lines. Choosing references nearest each speaker's voiceprint centroid, verifying every synthesized take against its speaker, and retrying with alternates brought drift to 0.5% and gender flips to zero. A later cache bug that could borrow the previous project's voice was caught the same way: a regenerated line now scores 0.53 against its own speaker and under 0.19 against every other.
Engineering notes
Three weeks, 118 commits, 93 API tests plus Playwright acceptance scripts; a two-minute clip goes from upload to dubbed export in under four minutes. Six systemd services and three timers inside WSL2 on one RTX 6000 Ada workstation, including a resident TTS service and an interactive lane that turned single-line regeneration from minutes into seconds, and a self-check that probes the public URL every two minutes and repairs the tunnel in 14 seconds. Chunked resumable upload with the parts on disk as the source of truth. Media as single-file fMP4 HLS behind signed, path-normalized URLs. A full read-only audit closed connection-pool exhaustion, token leakage via query strings, and a path-traversal gap in media links. Known limits: the review console is desktop-only, simultaneous edits of one line are last-write-wins, and Chinese, Japanese and Korean targets wait on a different length model.
More frames
Add a line the machine missedOne Done per languageProjects, status at a glanceBatch upload, six languages
