From pressure to prose:
how machines hear speech

Before a machine can recognize speech, we have to be honest about what speech is when it leaves a mouth: not words, not symbols — just organized disturbances in air pressure.

1.1A pressure wave, not a thing that travels

When you speak, your vocal folds open and close hundreds of times per second, chopping the airflow from your lungs into rapid puffs. Each puff shoves the air molecules directly in front of it. Those molecules crowd their neighbors, the neighbors crowd their neighbors, and a zone of slightly higher pressure — a compression — ripples outward at about 343 meters per second. Behind each compression, the molecules spring back and briefly overshoot, leaving a zone of slightly lower pressure: a rarefaction.

Watch the molecules below carefully. No individual molecule travels to your ear. Each one just oscillates around its home position, like a fan doing the wave in a stadium. What travels is the pattern — and that pattern is the only thing your listener, or a microphone, ever receives.

1.2Two numbers rule the wave

Freeze the probe's trace and two properties describe almost everything about a simple tone. Frequency — how many pressure cycles pass per second, measured in hertz (Hz) — is what your brain perceives as pitch. Amplitude — how far the pressure swings from its resting value — is what you perceive as loudness. Human hearing spans roughly 20 Hz to 20,000 Hz, but speech is far more modest: the vocal folds of most adults vibrate between about 80 and 300 Hz, and nearly all the information that distinguishes one speech sound from another lives below 8,000 Hz. That number will quietly decide an engineering choice in the next chapter.

1.3Real speech is a sum of simple waves

A tuning fork produces the clean sinusoid above. A voice never does. When your vocal folds buzz at, say, 120 Hz (the fundamental frequency, F0), they simultaneously generate energy at whole-number multiples — 240 Hz, 360 Hz, 480 Hz — called harmonics. Your throat, mouth, and lips then act as a filter, boosting some harmonics and muting others. Move your tongue and the boosted regions move; that is literally the difference between "ee" and "ah".

Build a complex wave yourself below. Notice two things: the sum looks nothing like its ingredients, and yet the ingredients are perfectly recoverable from the sum. That recoverability is Fourier's insight — any repeating wave can be decomposed into sinusoids — and it is the mathematical trapdoor the entire field of speech recognition will crawl through in Chapter 3.

A computer cannot store a continuous wave — it can only store lists of numbers. Digitization makes two brutal simplifications, and the miracle is that, done carefully, neither one loses anything your ear could hear.

2.1The microphone: pressure becomes voltage

Inside a microphone, a thin diaphragm gets pushed and pulled by the very compressions and rarefactions from Chapter 1, and its motion is converted into a continuously varying electrical voltage. The voltage is an analog of the pressure — same shape, different medium. But it is still continuous in two directions: it exists at every instant of time, and it can take any value. Digitization attacks both, one at a time.

2.2Sampling: slicing time

Sampling measures the voltage at strictly regular intervals and throws everything between measurements away. The number of measurements per second is the sample rate. Intuition says discarding "in-between" moments must destroy information — but the Nyquist–Shannon sampling theorem proves otherwise:

Sample too slowly, though, and something worse than blur happens: the samples become perfectly consistent with a different, slower wave, and the true one is unrecoverable. This impostor phenomenon is called aliasing. Drag the sample rate below twice the signal frequency in the instrument and watch a phantom wave appear — the machine will sincerely believe the phantom is what you said.

This is why the engineering standards are what they are. Music on a CD is sampled at 44,100 Hz — comfortably more than twice the 20,000 Hz edge of human hearing. Telephone speech was sampled at 8,000 Hz, capturing only up to ~4,000 Hz, which is why voices sound thin on a phone line yet remain intelligible. Modern ASR systems overwhelmingly standardize on 16,000 Hz: twice the ~8,000 Hz ceiling under which speech distinctions live. Recall that number from §1.2 — the physiology of speech dictated the spec sheet.

2.3Quantization: slicing amplitude

Each sampled measurement is still a continuous voltage. Quantization snaps it to the nearest rung on a ladder of allowed levels, so it can be stored as an integer. With b bits per sample the ladder has 2^b rungs: 8 bits gives 256 levels, 16 bits gives 65,536. The snapping introduces a small error — audible as a faint hiss called quantization noise — and every extra bit roughly halves it (about 6 dB of added signal-to-noise per bit). At the 16-bit standard used for speech, the ladder is so fine that the noise sits far below the room tone of any real recording.

2.4What the machine actually holds

After both cuts, one second of speech has become a plain array: 16,000 signed integers, each between −32,768 and +32,767. This format — sample-by-sample integers, no compression — is called PCM (pulse-code modulation), and it is what lives inside a WAV file. It deserves a moment of awe: your name, your accent, your mood, the room you stood in, and the words you chose are all encoded in nothing but this torrent of integers.

The PCM stream is faithful but illegible: sample #7,201 alone says nothing about speech. Feature extraction re-describes the signal so that nearby numbers mean nearby sounds — and for seventy years, every recognizer has begun the same way: chop, window, transform.

3.1Framing: speech holds still — briefly

Speech changes constantly, but not instantly. Your tongue and lips are physical objects with mass; over any stretch of about 10–30 milliseconds, the vocal tract is nearly frozen and the signal is effectively periodic. So we cut the stream into short frames — conventionally 25 ms long (400 samples at 16 kHz) — and slide the cutting window forward by a hop of 10 ms, so consecutive frames overlap and nothing falls between the cracks. The result: speech becomes a sequence of 100 frames per second, each one a snapshot we can analyze as if it were a steady tone.

One repair before analysis: a frame's edges are arbitrary cliffs — the wave is sliced mid-swing, and Fourier analysis mistakes those cliffs for high-frequency energy that was never in the voice (spectral leakage). The fix is to multiply the frame by a window function (typically a Hamming window) that fades the edges gently to near-zero.

3.2The Fourier transform: interrogating a frame

Chapter 1 promised that any repeating wave is a sum of sinusoids. The discrete Fourier transform (DFT) is the interrogation that recovers the recipe. Its mechanism is disarmingly simple: to ask "how much 220 Hz is in this frame?", multiply the frame, point by point, by a probe sinusoid at 220 Hz and sum the products. If the frame contains that frequency, the products reinforce and the sum is large; if not, positives and negatives cancel toward zero. Ask the same question at every frequency and the answers, laid side by side, form the frame's spectrum.

3.3The spectrogram: sound made visible

Compute a spectrum for every frame, stand each one upright as a colored column, and line the columns up in time: this is the spectrogram — time across, frequency up, energy as color. It is the single most important picture in speech science. Vowels appear as stacked horizontal bands: the dark ridges are formants (F1, F2, F3…), the resonances of your vocal tract from §1.3, and their positions literally spell out which vowel is being said. Fricatives like "s" appear as broadband high-frequency hiss. Silence is a dark gap. A trained phonetician can read a spectrogram; ASR's entire project is to teach a machine to do the same.

3.4Mel & MFCCs: hearing like a human, storing like a machine

The linear spectrogram wastes resolution: your ear can tell 100 Hz from 200 Hz effortlessly, but 7,100 Hz from 7,200 Hz not at all. Perceived pitch follows a roughly logarithmic law, formalized as the mel scale:

Classical pipelines apply the scale with a bank of ~26 overlapping triangular filters spaced evenly in mels: each filter sums the energy in its band, converting a 257-bin spectrum into 26 perceptually honest numbers. Take the logarithm (loudness is perceived logarithmically too), then apply one final decorrelating transform — a discrete cosine transform — and keep only the first ~13 coefficients. These are the MFCCs (mel-frequency cepstral coefficients): the smooth shape of the spectrum, with pitch detail and redundancy squeezed out. Append their frame-to-frame velocities and accelerations (Δ and ΔΔ) and you get the 39-dimensional vector that fed essentially every recognizer from the 1980s to the 2010s.

With features in hand, the recognition problem could finally be attacked. Three generations of ideas followed — circuits that matched patterns, algorithms that stretched time, and statistics that embraced uncertainty — and each one's failure mode taught the field what the next generation had to fix.

4.11952 · Audrey: recognition as circuitry

The first working recognizer, Bell Labs' Audrey (Davis, Biddulph & Balashek), was a rack of analog vacuum-tube circuitry roughly the size of a wardrobe that recognized the ten spoken digits. It worked by tracking formant patterns — precisely the vowel resonances you watched in Instrument 08 — and comparing them against reference patterns tuned to one speaker. For its designer's voice it reportedly exceeded 97% accuracy; for anyone else's, performance collapsed unless the machine was laboriously re-tuned. Audrey established the field's very first lesson: recognition is pattern matching on spectral features — and its first two nemeses, speaker variability and cost.

4.21962 · IBM's Shoebox and the sixteen words

A decade later IBM's Shoebox (William Dersch, demonstrated at the 1962 Seattle World's Fair) recognized sixteen words — the digits plus arithmetic commands — and could total sums spoken to it. Progress was real but the pace was sobering: ten years had bought six words. Through the 1960s and 70s, systems remained speaker-dependent, small-vocabulary, and isolated-word — you had. to. pause. between. words. DARPA's Speech Understanding Research program (1971–76) pushed scale, and CMU's Harpy (1976) reached 1,011 words by searching a network of possible sentences with a pruned "beam" — an idea that quietly survives inside every modern decoder in Chapter 6.

4.3The time problem, and dynamic time warping

Template matching had an obvious core: store a reference recording ("template") of each word as a sequence of feature vectors, and label new audio with the nearest template. But nobody says a word at one fixed speed. Say "seven" quickly and the /ɛ/ shrinks; say it carefully and it stretches — a rigid frame-by-frame comparison misaligns almost immediately. Dynamic time warping (Vintsyuk 1968; refined by Sakoe & Chiba 1978) solved this with dynamic programming: consider every monotonic way of stretching one sequence onto the other, and efficiently find the alignment with the lowest total mismatch. The warped distance, not the rigid one, decides the winning word.

DTW made small-vocabulary, isolated-word recognition genuinely usable, and it ran the first commercial voice dialers. But it scales catastrophically: continuous speech has no word boundaries to cut at, every speaker needs templates, and "distance to a recording" has no principled way to say how confident it is. The field needed a framework that treated speech as what it actually is — variable, noisy, and probabilistic.

4.4The statistical turn: hidden Markov models

That framework arrived when researchers at IBM (Fred Jelinek's group) and CMU (James Baker, later of Dragon) reframed the entire task as probability: of all word sequences W, which one maximizes P(W | audio)? By Bayes' rule this splits into two learnable parts — an acoustic model P(audio | W) and a language model P(W) — a factorization so durable that Chapter 6 still runs on it.

The acoustic model's workhorse was the hidden Markov model. Each phoneme is modeled as a tiny chain of states — canonically three: the sound's onset, steady middle, and offset. The true state sequence is hidden; what we observe are the MFCC vectors it emits. Two ingredients define the model: transition probabilities (at each frame, stay in the current state via a self-loop, or advance — self-loops are how the model absorbs slow speech, replacing DTW's warping) and emission probabilities (how likely is this MFCC vector from this state?), modeled by Gaussian mixture models — weighted sums of bell curves in 39-dimensional feature space. String phone-HMMs together via a pronunciation dictionary and you get word models; string words together and you get sentences.

Finding the best hidden path through this chain, given the observed frames, is the Viterbi algorithm — dynamic programming over a state-by-time grid called the trellis. If the picture below feels familiar, it should: it is DTW's matrix reborn, with a learned probabilistic model standing in for a stored recording.

4.5The empire and its cracks

The HMM-GMM recipe — MFCCs in, context-dependent triphone states, GMM emissions, n-gram language models, Viterbi decoding — ruled for three decades. CMU's Sphinx (1988) proved speaker-independent, continuous, large-vocabulary recognition was possible; toolkits like HTK and Kaldi industrialized it; it answered phone trees and took dictation. Its longevity came from honest engineering: every module was interpretable, trainable from data, and improvable in isolation.

But the cracks were structural, not incremental. The Markov assumption pretends each frame depends only on the current state, though real speech drags context across whole syllables. GMMs are weak models of the correlated, manifold-shaped distributions real features live on. And the pipeline's stages — features, acoustics, pronunciation, language — were each optimized separately, so errors compounded across handoffs and no stage could repair another's mistakes. By the late 2000s, word error rates had plateaued for a decade.

Everything before this point took sixty years. What follows took about ten — and in those ten years, speech recognition went from a technology that mostly didn't work to one that quietly runs beneath a measurable fraction of all human–computer interaction. The neural era is short. It is not small.

5.1The discontinuity, in numbers

The field's hardest public benchmark for two decades was Switchboard: strangers making casual telephone conversation — disfluent, overlapping, badly recorded. In the mid-1990s the best systems misrecognized roughly four words in ten. Fifteen years of classical refinement clawed that down to about one in four, then stalled: from the early 2000s to 2011 the state of the art barely moved. Then a single line of work — deep networks as acoustic models — cut the benchmark by roughly a third in one result (18.5% vs. a 27.4% classical baseline, 2011), and the floor kept falling: ~13% by 2014, 6.3% in 2016, and 5.1% in 2017 — inside the 5–6% range measured for professional human transcribers on the same audio. On cleaner read-speech benchmarks, modern systems sit under 2%. A decade of neural methods bought more accuracy than the entire preceding history of the field combined.

The consequences left the lab almost immediately. Voice assistants became usable rather than embarrassing; live captions appeared on videos, calls, and lectures; dictation became faster than typing for many people; call centers, medical scribing, and courtroom transcription were reorganized around machines with humans checking, instead of the reverse. And — the part that matters most in this course — ASR became the proof of concept that convinced the wider world deep learning was real: the 2012 speech results predate and prefigured the same playbook in vision and language.

5.22012 · The transplant: a network inside the HMM

The revolution began conservatively. The hybrid DNN-HMM kept every part of Chapter 4's machinery — states, lexicon, n-gram LM, Viterbi decoding — and replaced only the GMMs. A deep network reads a frame plus a window of ten-plus neighboring frames (context the GMM never saw) and emits a posterior over every tied HMM state at once. Deep nets don't assume features are Gaussian, don't need them decorrelated (log-mel filterbanks displaced MFCCs — the DCT from §3.4 existed largely to please GMMs), and carve decision surfaces of essentially arbitrary shape. When Toronto, Microsoft, Google, and IBM converged on this recipe around 2012, it was the largest single improvement in the field's history — achieved without changing the pipeline's architecture at all.

5.3CTC: deleting the alignment problem

The hybrid still leaned on the HMM's scaffolding — pronunciation dictionaries, forced alignments, multi-stage training bootstraps, and a decade of accumulated recipe. The first truly end-to-end objective, CTC (connectionist temporal classification, Graves et al. 2006 — years ahead of the hardware that would make it shine), asked a bolder question: why not output characters directly, one distribution per frame? The obstacle is arithmetic: 100 frames per second, but "cat" has three letters. CTC's fix is an extra token — the blank (∅), "nothing to report yet" — plus a collapse rule: merge adjacent repeats, then delete blanks. Training sums probability over every frame-level spelling that collapses to the target, computed by the same forward–backward dynamic programming as the trellis in Instrument 11. Baidu's Deep Speech systems (2014–15) proved the recipe scaled to production; CTC remains the default head bolted onto self-supervised encoders today, prized for its simplicity and natural streaming. Its cost: each frame's output is conditionally independent given the encoder, so CTC models spell what they hear and rely on an external language model to know that "wreck a nice beach" was probably something else.

5.4Attention: alignment as a learned gaze

Sequence-to-sequence with attention — crystallized for speech in Listen, Attend and Spell (2015) — repaired CTC's independence flaw by restructuring the task. An encoder reads all the audio into rich vectors; a decoder writes the transcript one token at a time, conditioned on everything it has already written, at each step computing attention weights — a soft spotlight over the encoder saying "to write this letter, listen here." Alignment stops being a path found by dynamic programming and becomes a differentiable behavior the model learns. Watch the spotlight sweep left to right on its own below: nobody programmed the monotonic drift — it emerges because speech and text run in the same order. The decoder is also, implicitly, a language model: it can output "wreck a nice beach" or "recognize speech" based on its own prior text, no external LM required.

5.5The transducer: recognition while you speak

Attention's catch is causal: the decoder wants the whole utterance before writing — fine for transcribing a file, useless for a live assistant. The neural transducer (RNN-T, Graves 2012) restored streaming without giving back CTC's independence flaw. It runs two networks in tandem — an audio encoder and a prediction network that reads the text emitted so far, acting as a built-in language model — and a small joint network that, at every (frame, text) position, decides: emit the next token now, or emit blank and wait for more audio. Training marginalizes over all such emission schedules on a two-dimensional lattice — the trellis idea's fourth appearance. Because it emits as evidence arrives, the transducer became the production architecture: by 2019 it was running entirely on-device on phones, with tricks like emission- latency regularization tuning exactly how eagerly it commits.

5.6An atlas of neural ASR

Here is the correction to the tidy story so far: these are not stages of one lineage. They are coexisting species, each defining "recognize speech" differently — what the output unit is, when it may be emitted, where the language model lives, and what supervision trains it. A modern practitioner chooses among them the way an engineer chooses a data structure. The atlas below lets you walk the five main body plans; note how the properties row changes under each one.

Beneath all five, the encoder itself evolved in parallel: bidirectional LSTMs gave way to transformers, whose self-attention lets every frame consult every other frame directly, and then to the speech-specialized Conformer (2020), which interleaves convolutions (local spectral texture) with self-attention (global context) and still anchors most production encoders. Two workhorse tricks travel with them: aggressive time-downsampling (100 frames/s is wasteful; encoders quickly compress to 25–40/s) and SpecAugment (2019) — training-time masking of random time and frequency bands of the spectrogram, an absurdly simple augmentation that delivered accuracy gains rivaling architecture changes. Notice what it is: the same masking instinct that §5.7 turns into an entire learning paradigm.

5.7Self-supervision: learning from speech nobody transcribed

Every system above eats transcribed audio — expensive anywhere, and simply nonexistent for most of the world's ~7,000 languages. wav2vec 2.0 (2020) broke the dependency: mask random spans of the latent audio sequence and train a transformer to identify what belonged there, distinguishing the true content from decoys drawn elsewhere in the same utterance. No transcripts involved — yet solving this fill-in-the-blank game across tens of thousands of hours forces the model to discover phonetic structure on its own. Pretrain once, then fine-tune with as little as ten minutes of labeled speech, and you get single-digit error rates on read-speech benchmarks that classical systems needed thousands of transcribed hours to approach. Siblings refined the objective — HuBERT predicts cluster IDs of masked regions (turning audio into pseudo-text, a hinge for §5.9), WavLM adds simulated noise and overlapping speakers so the representation also serves diarization — and self-supervised encoders are now the standard starting point whenever labeled data is thin.

5.8Scale, and the multilingual turn

Self-supervision made unlabeled audio valuable; the next move was to be omnivorous about it. Whisper (2022) trained one attention encoder-decoder on ~680,000 hours of web audio with weak (found, imperfect) transcripts across ~100 languages, multitasking transcription, translation-to-English, language ID, and timestamps as a single token stream — and proved robust out of the box on accents and noise that broke carefully engineered predecessors. Google's USM pretrained on over 12 million hours spanning 300+ languages. Meta's MMS (2023) scaled wav2vec-style pretraining plus a clever data source — recordings of read religious texts, which exist in astonishingly many languages — to ship recognizers for 1,100+ languages and language ID for ~4,000. One honest caveat belongs in every classroom: weakly-supervised seq2seq models hallucinate — during silence, music, or unfamiliar audio they can emit fluent text that was never said, precisely because the decoder is such a good language model. It is the era's signature failure mode, and an open research problem.

5.9The dissolving boundary: speech meets the language model

The newest species may be the last one that is recognizably "an ASR system." Two enabling ideas: neural audio codecs (SoundStream, EnCodec) compress waveforms into short sequences of discrete acoustic tokens, while clustered self-supervised features (HuBERT units) yield discrete semantic tokens — and once audio is tokens, it is grammatically identical to text. Speech-LLMs exploit this: bolt a speech encoder onto a large language model through a thin adapter (or feed audio tokens into its vocabulary directly), and transcription becomes just another next-token prediction task — alongside translation (SeamlessM4T), spoken question answering, summarizing a meeting from raw audio, or reasoning about a sound. The frontier "omni" assistants (GPT-4o-class realtime models, Gemini Live, open efforts like Moshi) go further into full-duplex speech-to-speech: listening and speaking simultaneously, interruptible mid-sentence, with no visible transcript in the loop at all. Inside such a system, ASR is no longer a product — it is a competency, dissolved into a general model of language that happens to have ears.

5.10The horizon

Where is the field actually straining right now? The long tail of speakers: children, the elderly, dysarthric and deaf speech (Project Euphonia), heavy accents, and code-switching mid-sentence — situations where "human parity" claims quietly collapse. The long tail of languages: 1,100 down, ~6,000 to go, bottlenecked on data creation — community corpora (Common Voice and kin) matter as much as any architecture. Rich transcription: real audio has multiple people, so the target is shifting from "what was said" to who said what, when — joint recognition and diarization of overlapping speakers. Trust: hallucination rates, calibrated confidence, and evaluation beyond WER (§6.2's caveats becoming a research agenda). Personalization and context: biasing recognition toward your contact names and jargon on the fly, without retraining. Efficiency: distilled and quantized models pushing Whisper-class accuracy onto phones, cars, and hearing aids, offline and private. And over all of it, the open question §5.9 poses: whether the next great recognizer will be an ASR system at all, or simply what a foundation model does when you talk to it.

A model never outputs text. It outputs scores — probabilities over thousands of competing spellings of what it heard. The last act of every recognizer, from Harpy to Whisper, is a search through that space for the one sentence worth printing.

6.1Beam search: greed, hedged

Bayes' factorization from §4.4 never left: hypotheses are scored by acoustics and by a language model's opinion of the text itself (today usually a neural LM "fused" into the search). Exhaustive search over sentences is impossible; pure greed — commit to the best token at each step — is brittle, because the locally best start is often the globally wrong one. Beam search is the working compromise: at every step, keep the k best partial sentences alive and extend them all. Set the beam to 1 in the instrument below and watch greed print the wrong sentence; widen it and the truth survives long enough to win.

6.2Keeping score: word error rate

The field's common currency for seventy years of claims is word error rate — align the output against a reference transcript (with edit-distance dynamic programming — the DTW family's third cameo) and count the damage:

For scale: the classical HMM-GMM systems of the 2000s sat in the low-to-mid twenties of % WER on hard conversational benchmarks; hybrids cut that sharply after 2012; modern systems report low single digits on those same sets — around the disagreement rate between two careful human transcribers. The gap that remains lives exactly where training data doesn't: heavy accents, distant microphones, code-switching, and the thousands of languages still waiting for their 680,000 hours.

6.3The whole journey, replayed

You now own every stage of the pipeline. Watch it run once more, end to end — and notice how much of the story is one idea in different clothes: an alignment between two sequences, found first by circuitry, then by dynamic programming, then by probability, and finally by attention.