Speech AI: Whisper and TTS

Speech AI covers the two core technologies that let machines "understand human speech" (speech recognition, ASR) and "speak like humans" (speech synthesis, TTS), and it is the most natural, lowest-friction interface between humans and machines. In 2022, OpenAI open-sourced the Whisper speech recognition model, which became the benchmark for ASR; over the same period, TTS evolved from concatenative and neural systems into end-to-end solutions like ElevenLabs and ChatTTS, with timbre and emotional expression closing in on real human voices. Today, speech AI is the foundation of meeting transcription, subtitles, dubbing, and voice assistants — and it is the "ears and mouth" of AI agents.

1. Background: Two Threads — Listening and Speaking

Speech AI broadly splits into two directions, corresponding to the two human pathways of hearing and vocalization:

Direction	English	What it does	Typical tasks
Speech recognition	Automatic Speech Recognition, ASR	Listen: audio → text	Transcription, subtitles, voice commands
Speech synthesis	Text-to-Speech, TTS	Speak: text → audio	Dubbing, audiobooks, voice assistant responses
(Derived) voice cloning	Voice Cloning	Replicate a voice from a few samples	Personalized dubbing, fraud (a risk)
(Derived) speaker/spoken understanding	Speaker Verification / Spoken Understanding	Determine "who is speaking, and what is being said"	Voiceprint unlock, meeting-minutes material

The two threads share the same underlying model technology: after 2022, the Transformer architecture (see Transformer and Attention) became the de facto foundation for both speech understanding and generation, and speech AI moved from "standalone small models" to "one modality of a multimodal large model" (see Multimodal Models).

A few key milestones:

• 2012–2014: Deep learning displaced the traditional GMM-HMM approach; DNN/HMM hybrid systems drove large drops in ASR word error rate (WER);
• 2018–2020: Transformers entered speech (LAS, Conformer) and end-to-end ASR became mainstream; Tacotron + WaveNet freed TTS from its concatenated, stitched-together feel;
• 2022-09: OpenAI released Whisper, trained on 680,000 hours of multilingual, weakly supervised data and supporting 99 languages — it became the industry baseline the moment it was open-sourced;
• 2023–2025: ElevenLabs and others productized "voice cloning + controllable emotion"; GPT-4o brought native real-time voice conversation; Chinese players such as Doubao, Tongyi, and Zhipu launched LLM-powered speech products; Chinese open-source TTS (ChatTTS, CosyVoice, Fish-Speech) emerged in rapid succession.

The one-sentence verdict: After Whisper was open-sourced in 2022, ASR was effectively "solved" — general-purpose recognition is no longer the bottleneck, and the difficulty has shifted to domain specialization and cost. For TTS, the contest has moved from "does it sound human" to "is it fast, is it stable, can it clone a voice, and can it be abused."

2. Technical Foundation 1: Transformer Encoding for ASR

The typical modern ASR pipeline can be summarized as "audio → features → encoding → decode and align → text":

Audio waveform → spectral features (e.g. 80-dim log-Mel) → Transformer encoder
        → output sequence representation → CTC / attention decoding / language model fusion → text

Take Whisper as an example. Its three key design choices:

1. Weakly supervised big data: trained on 680,000 hours of internet audio, of which 117,000 hours cover 96 languages (the paper claims 99), without relying on fine-grained human annotation — it directly predicts "<|start_of_transcript|> + language + task + timestamps";
2. Encoder-decoder structure: isomorphic to machine translation — the input is a 30-second window of log-Mel spectrogram, and the output is a sequence of text tokens with timestamps;
3. Multi-task unification: the same model handles transcription, translation (X→English), timestamps, and silence detection, making it easy to reuse directly in products.

Whisper's open weights come in five tiers — tiny / base / small / medium / large — and large-v3 achieves a word error rate (WER) on general benchmarks that approaches or beats earlier commercial systems, drastically cutting the cost of building your own transcription. Follow-up derivatives (faster-whisper, distil-whisper) use CTranslate2 inference and distillation to push speed up several-fold; see Inference Optimization and Quantization.

For developers, the key criteria for choosing an ASR system are not "which model is smarter" but:

• Word error rate (WER): compare on the same test set; for Chinese, watch Cantonese, regional dialects, and noisy conditions;
• Latency and throughput: real-time transcription requires streaming capability; offline batch jobs care about throughput;
• Domain adaptation: specialized vocabulary for meetings, healthcare, and customer service calls for fine-tuning open-source models (see Fine-Tuning and PEFT);
• Cost and privacy: local deployment vs. cloud API — sensitive audio often needs to stay on-premises.

The speech-text "alignment" problem

The essence of ASR is alignment: mapping variable-length audio frames onto variable-length text tokens. Whisper aligns implicitly with attention; traditional approaches use CTC and HMM for explicit alignment. This alignment problem makes speech model evaluation trickier than pure text — the "ground truth" for the same utterance can shift depending on colloquial speech, accents, and noise. For evaluation methodology, see LLM Evaluation and Benchmarks.

3. Technical Foundation 2: TTS from Concatenative to End-to-End

The evolution of TTS can be summarized in four stages: "concatenative → parametric/neural → end-to-end → LLM-scale":

Stage	Representative	Principle	Drawbacks
Concatenative	1990s database stitching	Select and splice segments from a recording library	Huge voice databases, unnatural output
Statistical parametric	HMM / early DNN	Model acoustic parameters, then synthesize	A "robotic" sound
Neural TTS	Tacotron 2 + WaveNet	seq2seq + autoregressive vocoder	Slow, error-prone
End-to-end	VITS, NaturalSpeech	Text straight to waveform	Stable, fast, more human-like
LLM-scale	VALL-E, ChatTTS, CosyVoice	Pretraining + conditional generation, capable of cloning	Heavy compute, misuse risk

Today's TTS technology follows two main paths:

• Autoregressive: predicts acoustic features frame by frame, like an LLM "continuing tokens" (VALL-E, Mega-TTS, etc.) — highly natural, but slow and prone to accumulating errors;
• Diffusion / flow-based: treats the waveform or acoustic features as the generation target, sampling in one or a few steps via denoising diffusion or flow matching (NaturalSpeech 2/3, CosyVoice, etc.) — a better balance of speed and quality — with principles rooted in Diffusion Models and Generative AI.

Another core capability of modern TTS is voice cloning: with just a few seconds to a few minutes of reference audio, you can replicate a target voice and accent. VALL-E demonstrated "zero-shot cloning from a 3-second sample," and ElevenLabs turned it into a consumer product. Bear in mind that cloning is a double-edged sword — it is also the technical root of deepfake voice fraud (see the risks section below).

Don't treat "sounds human" as the only goal

What matters more for shipping TTS products: stability (no dropped syllables or voice glitches when generating the same text repeatedly), controllability (speed, pauses, emotion tags, multiple speakers), latency (first packet under 300ms for a conversational feel), and compliance (voice rights and cloning authorization). "Does it sound human" is merely the ticket to entry.

4. Product Landscape (as of mid-2025)

Product/Model	Vendor	Positioning	One-line take
Whisper	OpenAI	Open-source ASR benchmark	The strongest open-source baseline for general recognition, widely embedded in transcription products
ElevenLabs	ElevenLabs	Voice cloning + multilingual TTS	Leading in voice/emotion control, with a mature API ecosystem
GPT-4o Realtime Voice	OpenAI	End-to-end voice conversation	Latency down to sub-second; see ChatGPT and Conversational AI
Doubao Voice / Volcano Engine	ByteDance	Chinese ASR+TTS+cloning	Strong price-performance for Chinese scenarios, covers digital-human livestreaming
Tongyi Voice / Spark	Alibaba / iFlytek	Chinese ASR+TTS	iFlytek has decades of speech expertise and complete industry solutions
ChatTTS / CosyVoice / Fish-Speech	Open-source community	Chinese open-source TTS	Free for commercial use, self-hostable, actively updated

Recommendation: For general transcription, go straight to Whisper or a big tech API; for production-grade dubbing/cloning, use an ElevenLabs-style service; for Chinese real-time conversation, try Doubao/Tongyi/iFlytek first; for private deployment, pick open-source TTS and fine-tune it. Validate with APIs first, then consider building your own — the same philosophy as Deployment and Inference Optimization in Practice.

5. Application Scenarios: The Ears and Mouth of Agents

Speech AI applications long ago outgrew the "voice assistant" label:

• Meeting transcription and minutes: Whisper real-time transcription + LLM summarization of key points and to-dos is now standard in meeting tools;
• Subtitles and localization: automatic video subtitle generation and multilingual dubbing/lip-sync adaptation directly serve content going global;
• Audiobooks and digital humans: TTS narration + virtual avatars have cut content production costs by an order of magnitude;
• Voice assistants and in-car systems: upgraded from "wake-word" to multi-turn dialogue, mixing on-device small models with cloud LLMs;
• Accessibility: screen readers for the visually impaired and voice prosthetics for people who have lost their speech — one of the most human-centered directions for the technology.

Even more noteworthy is speech's role in the agent ecosystem: ASR is the agent's "ears," and TTS is its "mouth" (for the agent concept, see AI Agents). A complete voice agent pipeline looks like this:

User speaks → ASR (listen) → LLM / Agent reasoning (think) → TTS (speak) → user
                                 ↑
                        tool calls, RAG retrieval, database queries

To build this kind of application, split "listen–think–speak" into three independent services and wire them together with an orchestration layer (see Building an Agent from Scratch and Building a RAG App from Scratch). Speech is only the entry point; the real intelligence still comes from the large language model behind it.

6. Risks: Voice-Cloning Fraud and Privacy

The sharpest risk in speech AI is deepfake voice:

• Fraud: since 2023, there have been numerous cases worldwide of phone scams and business fraud that "impersonate the voice of a relative or boss" — cloning requires only a few seconds of public audio;
• Evidence fabrication: forged voice calls and interference with voiceprint forensics can shake the chain of judicial evidence;
• Privacy: voice carries health, emotional, and identity information — biometric data more sensitive than text; both training on recordings and cloud processing require notice and consent;
• Content compliance: dubbing and streamer voice cloning implicate portrait rights and voice rights; commercial use requires authorization.

Mitigation works on three layers: detection (audio watermarking, deepfake detection models), governance (platforms requiring real-name verification and authorization review for cloning services, labeling AI-generated content), and legislation (multiple countries bringing deepfakes and voice rights under regulation). This aligns with the broader framework of AI alignment and safety governance — see AI Safety and Governance. As a developer, at minimum: get consent and give notice before collecting voice data, add synthetic-audio markers to TTS products, and introduce voiceprint verification in high-risk scenarios.

The one-sentence red line

"Cloning someone's voice" is technically a matter of minutes, but using it without authorization likely constitutes unlawful infringement. For any voice product: clear the authorization hurdle first, then talk about the experience.

Further Reading

• Transformer and Attention — the architectural foundation for speech recognition and synthesis
• Multimodal Models — speech as the audio modality of large models
• Diffusion Models and Generative AI — the principle behind diffusion-based TTS
• ChatGPT and Conversational AI — the GPT-4o real-time voice conversation case
• AI Agents — speech as the ears and mouth of agents
• AI Safety and Governance — deepfake voice and privacy governance
• Building an Agent from Scratch — building a "listen–think–speak" voice pipeline
• Model and Leaderboard Quick Reference — index of ASR/TTS models and open-source resources

References

• Radford et al. Robust Speech Recognition via Large-Scale Weak Supervision (Whisper, arXiv:2212.04356) — the original Whisper paper
• Wang et al. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (VALL-E, 2023) — the pioneering work on zero-shot voice cloning
• Shen et al. NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023) — a representative diffusion-based TTS
• van den Oord et al. WaveNet: A Generative Model for Raw Audio (2016) — the progenitor of neural vocoders
• Shen et al. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Tacotron 2, 2018) — classic neural TTS
• ElevenLabs official site — voice cloning and multilingual TTS product
• ChatTTS (open-source Chinese TTS) — open-source Chinese TTS built for conversational scenarios
• OpenAI's official Whisper page