How Speech Recognition Turns Your Voice Into Text

Speech recognition software turns spoken audio into written text. It does this in two broad steps: first the sound is converted into numbers that describe it, then a set of models works out which words most likely produced those numbers. Modern systems use neural networks for most of the work, but the basic problem has stayed the same for decades.

Step one: turning sound into numbers

A Microsoft documentation page describes the front end as isolating segments of sound that are probably speech and converting them into numeric values that characterise the vocal sounds. Wikipedia's article on speech recognition gives more detail on the classic approach: the audio is digitised (telephone speech at 8,000 samples per second), split into short frames computed about every 10 milliseconds, and each frame's frequency content is analysed with a Fourier transform. Extra values capture how the sound changes over time, and normalisation compensates for differences between speakers and recording conditions.

Step two: finding the likely words

The Microsoft page calls the back end a specialised search engine that looks across three sources of knowledge.

• Acoustic model: represents the sounds of a language and can be trained for a particular speaker or environment.
• Lexicon: lists words and how each is pronounced.
• Language model: represents how words combine, so 'recognise speech' is judged more likely than a similar-sounding nonsense phrase.

For any stretch of sound many things could have been said. The page says a recognizer's quality depends on how well it narrows the search, throwing out poor matches and choosing likely ones, which in turn depends on the quality of its models and algorithms.

A short history

• 1952: Bell Labs' 'Audrey' recognised digits spoken by a single speaker.
• Mid-1980s: IBM's Tangora handled a 20,000-word vocabulary using statistical methods based on hidden Markov models, which became the standard approach for decades.
• 1992: Carnegie Mellon University's Sphinx-II became the first speaker-independent, large-vocabulary, continuous recognizer, and won DARPA's 1992 evaluation.
• 2009 and 2010: deep neural networks were applied to acoustic modelling, producing a major accuracy improvement.
• 2017: Microsoft researchers reported reaching human parity on the Switchboard conversational speech benchmark.

All dates above are from Wikipedia's article, which also notes that LSTM networks, published in 1997, trained with a method called Connectionist Temporal Classification began outperforming earlier methods around 2007.

End-to-end systems

Older systems trained the acoustic, pronunciation and language parts separately. Since about 2014, 'end-to-end' systems have learned everything together in one network. Wikipedia credits the first major attempt to Alex Graves and Navdeep Jaitly in 2014, using CTC. CTC is a training method for tasks where the input and output are not aligned in time: audio has far more time slices than there are letters or sounds, and the system does not know in advance which slice matches which label, so CTC lets the network predict a probability at each step and scores all the alignments that give the same text.

Attention-based models, introduced in 2016, learn all components directly and need no separate language model when deployed; 'Listen, Attend and Spell' is an example. Transformers, which also rely on attention, were adopted for speech afterwards.

How accuracy is measured

The usual measure is word error rate, or WER. The recognised text is aligned with a correct reference transcript and the errors are counted: substitutions (wrong word), deletions (missing word) and insertions (extra word). WER is their total divided by the number of words in the reference. A WER of 0 means a perfect match, and it can exceed 1 if there are many insertions.

WER has limits. It treats all errors the same, although some change the meaning more than others; it does not say why errors happen; and a lower WER does not always mean better understanding, according to Wikipedia's article on the measure. It also notes that dictation is generally considered unacceptable below 95 percent accuracy, though the right standard depends on the use.

What still makes it hard

• Accents and unfamiliar pronunciations.
• Background noise.
• Spontaneous speech with pauses and corrections.
• Similar-sounding words.
• Large vocabularies: Wikipedia gives approximate error rates of 3 percent for 200 words, 7 percent for 5,000 words and 45 percent for 100,000 words, which shows why vocabulary size matters.

The short version

Speech recognition converts sound into numbers, then searches for the words most likely to have produced them, using knowledge of sounds, pronunciations and language. The search used to be built from separate hand-designed parts. Today a single neural network often does it all, but the goal is the same: pick the transcript with the fewest errors.

Sources

• Speech recognition, Wikipedia
• How Speech Recognition Works, Microsoft Learn (archived))
• Word error rate, Wikipedia
• Connectionist temporal classification, Wikipedia