The ML was the easy part

I spent a weekend building a pipeline that takes a song, isolates the vocal, swaps the singer for a different voice with a Retrieval-based Voice Conversion (RVC) model, remixes the new vocal over the original instrumental, and renders a branded promo video. Fully automated, six small Python scripts, one command in and a finished cover plus a YouTube-ready video out.

The voice conversion is the part everyone asks about. It was also the easy part. The actual work, the part worth writing down, was the systems engineering it took to make a fragile, half-abandoned ML stack run reliably across a Windows box and a Colab GPU. That fight is the reusable lesson, so that is what this is about.

What it does

download  ->  separate  ->  condition  ->  convert  ->  mix  ->  render_promo
(yt-dlp)      (Demucs)      (UVR + DSP)    (RVC/GPU)   (pydub)  (ffmpeg)
 mp3          vocals +      dry vocal      swapped     final    branded
              instrumental  (RVC-ready)    vocal       .mp3     promo .mp4

Each stage is an independent CLI with argparse, real error handling, and a consistent timestamped logger, so any step can run standalone or be chained into the next.

Stage         Script             Tech                        Output
1. Acquire    download.py        yt-dlp + ffmpeg             source audio
2. Separate   separate_audio.py  Demucs (htdemucs)          vocals.wav, instrumental.wav
3. Condition  conditioning.py    UVR de-reverb + torchaudio dry vocals (RVC-ready)
4. Convert    convert.py         rvc-python (RVC v2)        vocals_kodak.wav
5. Mix        mix.py             pydub                      final .mp3
6. Promo      render_promo.py    ffmpeg                     branded .mp4

Engineering decisions that mattered

Demucs over Spleeter

htdemucs, Hybrid Transformer Demucs, produces noticeably cleaner vocal isolation. Residual instrumental bleed is what makes a downstream voice conversion sound robotic, so spending quality here pays off everywhere later.

Instrumental as the sum of stems, not original minus vocals

Subtraction reintroduces phase and rounding artifacts. Summing the model's own drums, bass, and other stems is higher fidelity, and --two-stems vocals makes Demucs do exactly that.

A dedicated conditioning stage before RVC

RVC faithfully reproduces whatever you feed it, including reverb tails and odd levels. So between separation and conversion there is a stage that de-reverbs with a UVR MDX model, downmixes to mono, peak-normalizes, and resamples to the model's rate. No time-stretching anywhere, so the original rhythm and cadence stay sample-accurate.

A direct ffmpeg wrapper, not a higher-level library

ffmpeg's -stream_loop -1 gives gapless looping at the container level, its CLI flags are version-stable, and sourcing the binary from imageio-ffmpeg means no system ffmpeg install is needed. The promo visualizer is generated from the audio itself with showfreqs and showcqt, so there is no external asset and no third-party footage.

The dependency fight

This is the real content. The ML was a solved problem. Getting the dependencies to coexist was the job. Every one of these was diagnosed from a stack trace and fixed at the root.

Three Python versions, and the stack hates two of them

My local machine runs Python 3.14. The ML stack (torch, fairseq, demucs, rvc-python) has no wheels for 3.14 and will not build. Colab runs 3.12, which is better, except rvc-python hard-pins numpy<=1.25.3, and that has no 3.12 wheel and fails to compile from source. The resolution was to stop trying to make one interpreter happy. I run the lightweight scripts locally on 3.14, run the GPU stack on Colab, and for the RVC step specifically I use uv to build an isolated Python 3.10 venv on the fly and run just convert.py inside it. No kernel restart, no global downgrade. The right Python for each tool, side by side.

demucs.api does not exist in the released package

I wrote the separator against demucs.api.Separator. It imports fine from the GitHub source, but it is not shipped in the PyPI release (4.0.1), only on main. The fix was to drive Demucs through its stable CLI, python -m demucs --two-stems vocals, instead. The lesson: prefer the stable interface over the convenient one when the convenient one is not in the release you will actually pip install.

pip silently backtracked Demucs to a 3.x

A combined pip install was quietly resolving demucs down to a 3.x build, missing the features I needed, to satisfy another package, and then aborting the whole transaction on an unrelated build failure. Two fixes: split the installs so one bad wheel cannot roll back the rest, and pin (demucs>=4.0.0) so pip cannot backtrack past the version I need.

PyTorch 2.6 broke fairseq's model loading

RVC loads HuBERT through fairseq's torch.load. PyTorch 2.6 flipped the default to weights_only=True, which rejects fairseq's pickled Dictionary global with an UnpicklingError. The fix was a small, scoped monkeypatch that restores weights_only=False right before the trusted RVC assets load, applied in my own code rather than by downgrading torch for everything.

Google Drive's FUSE mount drops under load

Running the pipeline from a Drive-mounted folder died mid-install with Transport endpoint is not connected. Drive's FUSE layer cannot take heavy I/O, a big install plus a 250 MB model download, and when it drops it takes the working directory down with it. The fix was to treat Drive as storage, not workspace: copy the project to Colab's local /content disk, run there, copy the results back.

YouTube bot-blocks datacenter IPs

yt-dlp from Colab hit "Sign in to confirm you're not a bot." YouTube blocks cloud IPs. The fix was to acquire the source on a residential connection and ship it with the project. The notebook auto-detects the bundled file and skips the download.

ffmpeg drawtext versus Windows drive letters

The brand overlay failed because a Windows C:\... font path collides with ffmpeg's filtergraph syntax, where : is the option separator, and escaping it is miserable. The fix was to copy the font to a colon-free relative filename next to the output and run ffmpeg from that directory, which sidesteps the escaping entirely.

What generalizes

1. Prefer stable interfaces (CLIs) over convenient ones (latest-main APIs) when integrating fast-moving or abandoned libraries.
2. Isolate incompatible dependencies instead of fighting them. A throwaway uv venv for one script beat every attempt to make everything coexist in one interpreter.
3. Pin and split installs so resolution is deterministic and one failure does not cascade.
4. Match the runtime to the constraint. Local for light I/O work, Colab GPU for ML, a pinned venv for the one picky tool.
5. Read the stack trace to the root. Every fix here came from the actual error, not a guess. The numpy pin, the weights_only flip, and the FUSE drop were all one layer under the surface symptom.

The stack, for the curious: yt-dlp, Demucs, audio-separator (UVR), torchaudio, rvc-python, faiss, pydub, ffmpeg via imageio-ffmpeg, uv, Google Colab on a T4 GPU, and picklescan to scan any third-party .pth before loading it.

A note on responsible use

This was a personal, experimental remix, not built for distribution, and I drew two deliberate lines.

No realistic face deepfakes of real people. A photorealistic face swap of an identifiable person, especially for anything promotional, is a non-consensual likeness problem, so the video stage uses an abstract, audio-generated visualizer instead. There is no real-person likeness anywhere in the output.

Untrusted model weights get scanned, not trusted. Any third-party .pth is a pickle that can execute code the moment it loads, so the model here went through picklescan before it ever touched torch.load.

If you have a fast-moving or fragile stack that still has to run reliably, that gap between a model that works in a notebook and a pipeline that ships is the work I do. It starts at /services.