Fully Local AI Podcast

I've wanted my own podcast for a while, but I didn't want to rent a voice. So I'm building a fully local text-to-speech pipeline for it on the PC I put together for exactly this kind of work. This page is the build log. Each part gets added here as it happens, and I'll keep the earlier parts intact so the mistakes stay visible.

The goal

A roughly 30 minute, two-host English podcast about AI, produced entirely on hardware I own. No cloud TTS, no per-minute bill, no terms of service deciding what my show can say. My priorities, in order: it has to sound human, it has to be repeatable, and it has to be easy to tinker with. GPU efficiency is deliberately not on the list.

Part 1: research and the test plan

September 13, 2026. Before installing anything, I picked the candidate models, pinned their versions and weight hashes, and wrote the scoring sheet I'd judge them by. I planned it by running Claude Fable 5.1 and GPT-5.6 Sol on the same brief and feeding their work back into each other until they agreed. The full write-up, including the four reproducibility bugs the cross-review caught before I ran a single command, is on X: I Made Two AIs Argue Before I Installed a Single TTS Model.

The machine

RTX 5090 with 32 GB of VRAM, Ryzen 9 7950X, 64 GB DDR5, 2 TB NVMe, Windows 11 Pro. Native Windows, no WSL, which turned out to be a real constraint on which models were even candidates.

What I'm testing

• VibeVoice-Large (7B, MIT). The original upstream code is gone, so I'm using a community-preserved fork with the original ModelScope weights, pinned to a revision and verified with per-file SHA256 hashes.
• Dia2 (2B, Apache 2.0, with its Mimi codec under CC-BY-4.0, so the final show needs a credit line).
• VibeVoice-1.5B as a control. If blind listeners can't tell it from the 7B, the smaller model wins.
• A third candidate, Breeze TTS 2, was dropped the moment I read its license. Non-commercial.

Reference voices for the two hosts are synthetic, generated with Kokoro-82M as a neutral third model. I'm intentionally not cloning a real person.

How I'll score it

Transcript accuracy with openai-whisper and jiwer, speaker consistency with WavLM-SV, and automated checks for repetition, dropped lines, and abnormal silence. Weights: seconds until it sounds like AI (30%), naturalness (25%), dialogue flow (15%), speaker consistency (15%), artifacts (10%), word error rate (5%), plus hard failure gates. Blind A/B listening is the tiebreaker. Raw and mastered audio are scored separately, and mastering is gain-only loudness normalization so it can't hide anything.

How I planned it

Two models, one brief. Fable 5.1 was the better scout: it found models that fit the use case, surfaced the licensing issue, and pushed for verifying weight hashes before trusting a mirror. GPT-5.6 Sol was the better systems engineer: environment separation, smoke tests, the QC chain, the benchmark structure. Then Fable turned GPT's plan into staged runs with explicit stop points, and GPT became a picky code reviewer once it had something concrete to review. Nothing executes until I've signed off on the previous stage.

Where Part 1 ends

Nothing installed yet. Research approved, install plan reviewed through five revisions, moving to execution. Next: machine audit, install, smoke-test both models, a 5 to 7 minute benchmark, blind listening.

Part 2: the voices passed, the conversation failed

October 3, 2026. Both models installed cleanly and passed every automated check. Then I listened, and it sounded like two people reading a script at each other. The full write-up is on X: The Voices Passed. The Conversation Failed.

What happened

• The install was the easy part. Everything pinned and hash-checked, both models running natively on the 5090. The lab runs offline, which caught VibeVoice quietly fetching an unpinned tokenizer from another model's repository at load time. It's now pinned locally. Any hidden download is a stop, not a warning.
• Both models passed. Both failed. Every line sounded fine on its own. The conversation didn't: the second host never sounded like they'd heard the first. Two different architectures failed the same way, which told me the problem was my question, not the model.
• A perfect transcript with the wrong voices. On a 14-line stretch with real friction in it, Dia2 got every word right and put every line in the wrong voice, because it assigns voices by whoever speaks first in each chunk. The transcript check would have passed it. The speaker check caught it.

What changed in the plan

• A new thing to score. Does a reply sound like a reply? Does tension carry across a change of speaker? This is where it stopped being a TTS benchmark and became a conversation-quality problem.
• Licenses became a hard gate. Monetizing moved from "maybe" to "soon," so every model has to be commercially clean from day one. That ruled out ZONOS2 for now: the main repository is MIT licensed, but the Windows build ships with no license file. I've asked the maintainers.
• A director layer. A local language model (Qwen3-14B in llama.cpp) reads the script and writes notes for each line: emotion, energy, pace, pauses, emphasis, and which earlier line it's reacting to. Interpretation and performance are now separate jobs. Same script and seed, byte-identical notes. I logged its misses rather than tune the prompt to pass my own test.

Engines tried so far

• VibeVoice-Large and Dia2: functional pass, conversational fail.
• ZONOS2: the best fit for directed delivery, on hold until the Windows build's license is clear.
• VoxCPM2: excellent cloning, but it ignored its instructions. Across 32 renders, only "much slower" did anything reliable, and changing the random seed moved the audio more than any instruction did.
• CosyVoice 3: the next candidate, put through the same instruction test.

What I learned

Before testing whether a control works, measure how much the system moves when you touch nothing. And a quality check only covers what it actually measures. My ears caught the failure none of the metrics did.

Where Part 2 ends

Same day update: CosyVoice 3 passed the instruction test that VoxCPM2 failed. Pace, volume, and basic emotion instructions all moved the audio in the right direction, in plain English, well beyond its seed-to-seed noise. Whether "happy" sounds happy or just louder is a listening call I haven't made yet. Next: listen, then put directed voices on the frozen eight-minute script and find out whether they can actually react to each other.