Speech-to-Text for AI Voice Agents

How Modern STT Works

Modern speech-to-text systems are built on deep neural networks, primarily transformer architectures that process audio spectrograms to produce text output. The models are trained on thousands to hundreds of thousands of hours of transcribed audio, learning the statistical relationships between acoustic patterns and the words they represent.

The training data determines the model strengths and weaknesses. Models trained on diverse, multilingual datasets handle accents and language mixing well. Models trained primarily on clean, studio-quality audio struggle with phone-quality recordings and background noise. Models with medical, legal, or technical training data recognize domain-specific terminology that general models miss.

OpenAI Whisper was a watershed moment for speech recognition. Released as open source, Whisper was trained on 680,000 hours of multilingual audio data collected from the internet. It demonstrated that a single model could achieve competitive accuracy across multiple languages and acoustic conditions. This raised the bar for the entire industry and spurred rapid improvement across commercial providers.

Current state-of-the-art commercial systems exceed Whisper accuracy on most benchmarks while offering lower latency and streaming capabilities that Whisper (in its original form) did not support. The competitive landscape has driven continuous improvement in both accuracy and speed.

Streaming vs Batch Processing

Voice agents require streaming speech recognition, where audio is processed in small chunks (typically 100 to 300 milliseconds) as it arrives. The system produces partial transcriptions that update as more audio is received, converging on a final transcription once the utterance is complete. This is fundamentally different from batch processing, where the complete audio is uploaded and processed at once.

Streaming ASR is critical for voice agent latency because the language model can begin processing partial transcriptions before the caller finishes speaking. When the caller says "I need to check the status of my order," the LLM receives partial transcriptions like "I need to," "I need to check the," "I need to check the status," and begins formulating a response plan before the complete utterance arrives. This pipelining saves 200 to 500 milliseconds of perceived response time.

The tradeoff is accuracy. Streaming transcriptions are inherently less accurate than batch transcriptions because the system must make predictions with incomplete context. Modern systems mitigate this through look-ahead buffers (waiting slightly longer to get more context before committing to a transcription), and correction mechanisms that revise earlier words as later context arrives.

Endpointing: Detecting When the Speaker Has Finished

Endpointing is the process of determining when the caller has finished speaking so the agent can begin responding. This is one of the most challenging problems in voice agent design because the consequences of errors in either direction are severe.

If the endpoint detection is too aggressive, the agent begins responding while the caller is still speaking, cutting them off mid-sentence. This is the single most frustrating experience in voice agent interactions. If the detection is too conservative, the agent waits in silence after the caller finishes, creating an unnatural pause that makes the conversation feel broken.

Modern endpointing systems use multiple signals. Silence duration is the baseline signal, typically triggering after 500 to 800 milliseconds of quiet. However, silence alone is insufficient because speakers regularly pause mid-sentence to think. Prosodic features like falling intonation, decreased volume, and final lengthening (stretching the last syllable of a sentence) suggest completion. Semantic analysis of the partial transcript determines whether the words so far form a syntactically and semantically complete thought. Some systems also use voice activity detection (VAD) to distinguish between true silence and environmental noise.

The optimal endpoint sensitivity varies by use case. For simple queries where the expected input is short ("yes," "no," "my account number is..."), aggressive endpointing works well. For open-ended questions where callers may think before answering, more conservative settings prevent premature cutoff. Some platforms allow dynamic adjustment of endpoint sensitivity based on the conversation context.

Accuracy Factors

Speech recognition accuracy depends on several factors that vary across real-world calling conditions. Audio quality is the most significant factor. Phone calls use narrowband audio (8kHz sampling) which captures less acoustic detail than wideband connections. Cellular calls introduce additional compression artifacts and signal quality variation. Speakerphones, Bluetooth headsets, and car audio systems each create different acoustic challenges.

Speaker characteristics affect accuracy. Accents that differ significantly from the training data produce higher error rates. Speaking rate matters, with very fast or very slow speech reducing accuracy. Age-related voice characteristics, speech impediments, and non-native speaker patterns all increase difficulty. The best systems show minimal accuracy variation across demographic groups, but this remains an active area of improvement.

Environmental noise, including traffic, wind, office chatter, machinery, and television audio, degrades accuracy. Modern systems use noise suppression algorithms that filter out background sounds before recognition, but extreme noise conditions still challenge current technology. Call center-grade deployments sometimes use echo cancellation and noise gating on the telephony side to improve the audio quality before it reaches the ASR system.

Leading Providers for Voice Agents

Deepgram specializes in real-time speech recognition with extremely low latency, making it popular for voice agent applications where speed is critical. Their Nova-2 model achieves strong accuracy on conversational speech while maintaining streaming latency under 200 milliseconds. Pricing is competitive at /bin/bash.0043 per minute for their base tier.

AssemblyAI offers high accuracy with Universal-2, their latest model, which performs well across accents, noise conditions, and audio qualities. They provide rich features beyond basic transcription, including speaker diarization, sentiment analysis, and content moderation. Their streaming API works well for voice agent pipelines.

Google Cloud Speech-to-Text offers broad language support (over 125 languages) and tight integration with the Google Cloud platform. Their chirp model series provides competitive accuracy, and their healthcare-specific model handles medical terminology well. Pricing is higher than specialized providers but includes enterprise support and compliance certifications.

Amazon Transcribe integrates seamlessly with AWS services, making it a natural choice for organizations already on the AWS platform. It offers medical transcription, call analytics features, and custom vocabulary support. Streaming latency is reasonable but not as fast as providers that specialize exclusively in real-time applications.