I've wanted my own podcast for a while, but I didn't want to rent a voice. So I'm building a fully local text-to-speech pipeline for it on the PC I put together for exactly this kind of work. This page is the build log. Each part gets added here as it happens, and I'll keep the earlier parts intact so the mistakes stay visible.
The goal
A roughly 30 minute, two-host English podcast about AI, produced entirely on hardware I own. No cloud TTS, no per-minute bill, no terms of service deciding what my show can say. My priorities, in order: it has to sound human, it has to be repeatable, and it has to be easy to tinker with. GPU efficiency is deliberately not on the list.
The finalized stack
Updated October 30, 2026. After three parts of testing, this is what actually makes an episode. Everything runs on my own PC, and every model in it is licensed for commercial use.
Voice and audio
- Dia2 (2B, Apache 2.0) is the voice engine. Each host is conditioned on their own last few lines of the conversation, which is what makes a reply sound like a reply.
- Mimi (CC-BY-4.0) is Dia2's audio codec, so every episode carries a credit line.
- Kokoro-82M made the two synthetic host voices: Theo is am_liam and Nina is af_sarah. No real person is cloned.
- ffmpeg assembles and masters the episode: whole segments with natural edges, one two-pass loudness pass to -16 LUFS, then WAV and MP3 with an AI-voices disclosure in the file's tags.
Quality checks
- Whisper large-v3 and jiwer check that the audio says what the script says.
- WavLM-SV checks that each line is in the right host's voice, against a similarity floor I validated for these two voices.
- ffmpeg ebur128 measures loudness and true peak.
- Custom Python checks catch what those miss, like a stray sound after a chunk's last word. Every failure I hear becomes one of these, with a regression test.
Pipeline
- Claude Code runs five agents one at a time: curator, director, scriptwriter, generator, and quality checker. Each stops when it's done and waits for my go.
- A production adapter, a plain deterministic Python script rather than an AI, turns an approved script into renderer input.
- Python 3.12, PyTorch with CUDA 12.8, and uv, one pinned environment per model, with weights verified by hash. Native Windows, no WSL.
Hardware
- RTX 5090 (32 GB), Ryzen 9 7950X, 64 GB DDR5, Windows 11 Pro.
Tested and set aside
- VibeVoice-Large: the weights are research use only, so it stays a lab benchmark.
- CosyVoice 3: the best at taking direction, but lost to Dia2 on four of five scores.
- Qwen3-14B in llama.cpp: the local director from Part 2. Dia2 barely uses written direction, so it's not in the production path.
- VoxCPM2: ignored its instructions. ZONOS2: on hold until its Windows build has a clear license.
Part 1: research and the test plan
September 13, 2026. Before installing anything, I picked the candidate models, pinned their versions and weight hashes, and wrote the scoring sheet I'd judge them by. I planned it by running Claude Fable 5.1 and GPT-5.6 Sol on the same brief and feeding their work back into each other until they agreed. The full write-up, including the four reproducibility bugs the cross-review caught before I ran a single command, is on X: I Made Two AIs Argue Before I Installed a Single TTS Model.
The machine
RTX 5090 with 32 GB of VRAM, Ryzen 9 7950X, 64 GB DDR5, 2 TB NVMe, Windows 11 Pro. Native Windows, no WSL, which turned out to be a real constraint on which models were even candidates.
What I'm testing
- VibeVoice-Large (7B, MIT). The original upstream code is gone, so I'm using a community-preserved fork with the original ModelScope weights, pinned to a revision and verified with per-file SHA256 hashes.
- Dia2 (2B, Apache 2.0, with its Mimi codec under CC-BY-4.0, so the final show needs a credit line).
- VibeVoice-1.5B as a control. If blind listeners can't tell it from the 7B, the smaller model wins.
- A third candidate, Breeze TTS 2, was dropped the moment I read its license. Non-commercial.
Reference voices for the two hosts are synthetic, generated with Kokoro-82M as a neutral third model. I'm intentionally not cloning a real person.
How I'll score it
Transcript accuracy with openai-whisper and jiwer, speaker consistency with WavLM-SV, and automated checks for repetition, dropped lines, and abnormal silence. Weights: seconds until it sounds like AI (30%), naturalness (25%), dialogue flow (15%), speaker consistency (15%), artifacts (10%), word error rate (5%), plus hard failure gates. Blind A/B listening is the tiebreaker. Raw and mastered audio are scored separately, and mastering is gain-only loudness normalization so it can't hide anything.
How I planned it
Two models, one brief. Fable 5.1 was the better scout: it found models that fit the use case, surfaced the licensing issue, and pushed for verifying weight hashes before trusting a mirror. GPT-5.6 Sol was the better systems engineer: environment separation, smoke tests, the QC chain, the benchmark structure. Then Fable turned GPT's plan into staged runs with explicit stop points, and GPT became a picky code reviewer once it had something concrete to review. Nothing executes until I've signed off on the previous stage.
Where Part 1 ends
Nothing installed yet. Research approved, install plan reviewed through five revisions, moving to execution. Next: machine audit, install, smoke-test both models, a 5 to 7 minute benchmark, blind listening.
Part 2: the voices passed, the conversation failed
October 3, 2026. Both models installed cleanly and passed every automated check. Then I listened, and it sounded like two people reading a script at each other. The full write-up is on X: The Voices Passed. The Conversation Failed.
What happened
- The install was the easy part. Everything pinned and hash-checked, both models running natively on the 5090. The lab runs offline, which caught VibeVoice quietly fetching an unpinned tokenizer from another model's repository at load time. It's now pinned locally. Any hidden download is a stop, not a warning.
- Both models passed. Both failed. Every line sounded fine on its own. The conversation didn't: the second host never sounded like they'd heard the first. Two different architectures failed the same way, which told me the problem was my question, not the model.
- A perfect transcript with the wrong voices. On a 14-line stretch with real friction in it, Dia2 got every word right and put every line in the wrong voice, because it assigns voices by whoever speaks first in each chunk. The transcript check would have passed it. The speaker check caught it.
What changed in the plan
- A new thing to score. Does a reply sound like a reply? Does tension carry across a change of speaker? This is where it stopped being a TTS benchmark and became a conversation-quality problem.
- Licenses became a hard gate. Monetizing moved from "maybe" to "soon," so every model has to be commercially clean from day one. That ruled out ZONOS2 for now: the main repository is MIT licensed, but the Windows build ships with no license file. I've asked the maintainers.
- A director layer. A local language model (Qwen3-14B in llama.cpp) reads the script and writes notes for each line: emotion, energy, pace, pauses, emphasis, and which earlier line it's reacting to. Interpretation and performance are now separate jobs. Same script and seed, byte-identical notes. I logged its misses rather than tune the prompt to pass my own test.
Engines tried so far
- VibeVoice-Large and Dia2: functional pass, conversational fail.
- ZONOS2: the best fit for directed delivery, on hold until the Windows build's license is clear.
- VoxCPM2: excellent cloning, but it ignored its instructions. Across 32 renders, only "much slower" did anything reliable, and changing the random seed moved the audio more than any instruction did.
- CosyVoice 3: the next candidate, put through the same instruction test.
What I learned
Before testing whether a control works, measure how much the system moves when you touch nothing. And a quality check only covers what it actually measures. My ears caught the failure none of the metrics did.
Where Part 2 ends
Same day update: CosyVoice 3 passed the instruction test that VoxCPM2 failed. Pace, volume, and basic emotion instructions all moved the audio in the right direction, in plain English, well beyond its seed-to-seed noise. Whether "happy" sounds happy or just louder is a listening call I haven't made yet. Next: listen, then put directed voices on the frozen eight-minute script and find out whether they can actually react to each other.
Part 3: the engine, the episode that failed, and the pipeline that came out of it
October 27, 2026. I picked the engine, and it went against my own design. Then the first full episode failed, and the one honest piece of listener feedback changed what I'm building. What came out of it is a pipeline that stops after every step and turns every failure into a check. The full write-up is on X: You Can't Direct Your Way Into a Conversation.
Picking the engine
- CosyVoice 3 takes direction. "Very tiredly," "very skeptically," and "very determinedly" all worked. "In a very tired voice" failed every time. Combinations don't add up: "skeptically and quickly" did both, "tiredly and quickly" lost the speed. Every combination has to be tested on its own.
- Dia2 needed to hear the conversation. Instead of a fixed voice sample, I gave each host their own last few lines from the conversation so far. The handoffs that failed in Part 2 finally sounded like replies.
- The head-to-head. Same five-minute script on both, loudness matched, five things scored out of 10. Dia2 won responsiveness (9.2 vs 8.5), naturalness (9.1 vs 8.2), voice consistency (9.3 vs 8.6), and stamina over the full length (9.0 vs 7.9). CosyVoice won one: you can tell exactly what each line is feeling (9.1 vs 7.8). Dia2 is the production engine, and the director I built in Part 2 barely plugs into it.
- A license correction. In Part 1 I listed VibeVoice-Large as MIT. The code is. The model card for the Large weights says research use only, and they've been withdrawn. It stays a research benchmark in my lab, not part of anything I ship.
The first episode failed
- My verdict: super poor. The first full episode, about 13 minutes, came out muffled and inconsistent. Only 6 of 17 chunks passed my own checks. The new female voice I'd picked was much darker than the old one, and one host's voice drifted further from itself chunk by chunk.
- The listener test. I sent the first 68 seconds to a small group. The one reply said it sounded like two robots talking at the listener, not to each other, and they tuned out at about 30 seconds.
- That changed the goal. I'd been building a show for an audience. What I actually want is a personal podcast: a weekly episode on the AI and product stories I care about, because audio is easier for me to take in than reading. The first 30 seconds still decide everything.
What changed: five agents with a stop after each
- A taste profile first. Before any audio, I wrote down what makes a story worth my time: a mechanism, a measured result, a surprising explanation, or an experiment I could run. Model news only if it unlocks something.
- Five agents, one at a time. A curator reads the week's sources (web reading only, no accounts), a director writes a brief per story, a scriptwriter writes the dialogue for the two hosts, Nina and Theo, a generator renders and masters it locally, and a quality checker runs technical checks and builds a listening pack for me. Each one stops when it's done. Nothing moves to the next step until I say go.
- Every failure becomes a check. When I hear something wrong, it doesn't get fixed once. It becomes a row in a failure register, a tool, a regression test, and a line in the instructions of the agent that should have caught it. By October 27 the register had 18 rows. Two examples: Dia2 sometimes makes a non-word noise just after the last word of a chunk, which is now muted automatically, but only when six safety checks pass. Otherwise it goes on my listening list. And "sixty" kept coming out garbled, so the script now spells it "siksty" for the voice engine.
What I learned
You can't direct your way into a conversation. The engine that won needed to hear what was just said, not instructions on how to feel. And the pipeline only got trustworthy once every problem I caught by ear had to become something that runs every time.
Where Part 3 ends
On October 26 I swapped the female host to a brighter voice and re-rendered the whole first weekly episode: six segments, 16:45 long. No speaker swaps, dropouts, or missing lines, and the transcript matched the script to within 0.5% to 3% per segment. I listened on October 27 and kept the voice. Next: tighten the episode structure, automate the assembly, and get to the first episode I'd actually choose to listen to.