Voice AI Architecture: Real-Time Voice Pipelines, ASR, TTS & Conversational AI (2026)

How to diagram voice AI architectures. Covers ASR, LLM, TTS, WebRTC, the OpenAI Realtime API, Twilio voice AI, latency optimization, and streaming pipeline patterns — with prompt templates to generate diagrams in seconds.

Ryan·Senior AI Engineer

·Last updated June 24, 2026

A voice AI architecture diagram visualizes how an AI-powered voice system processes spoken input, reasons about it, and synthesizes a spoken response — all within the tight latency budget users expect from a real conversation. Voice AI pipelines combine automatic speech recognition (ASR), large language model inference, and text-to-speech synthesis (TTS) across a real-time audio transport layer. Each stage adds latency, and the architecture must be designed to keep end-to-end response time under ~500ms for natural-feeling conversations.

Voice AI is one of the fastest-growing AI deployment categories in 2026, powering customer support agents, voice-enabled coding assistants, healthcare intake systems, language learning apps, and AI companions. Diagramming your voice architecture is essential for latency analysis, cost modeling, reliability planning, and communicating the system to stakeholders outside your engineering team.

The three-stage pipeline: ASR → LLM → TTS

Most voice AI systems are built on a three-stage pipeline, each stage contributing independently to latency, cost, and accuracy:

Stage 1: Automatic Speech Recognition (ASR)

ASR converts the user's audio into text. The critical architecture decision here is streaming vs. batch transcription. Batch ASR waits for the user to finish speaking (detected by a voice activity detector or silence timeout), then transcribes the full utterance — simpler but adds 500ms–2s of end-of-speech latency. Streaming ASR (Deepgram Nova, Google Streaming STT, OpenAI Whisper Live) returns partial transcripts as the user speaks, allowing the LLM to begin processing before the utterance is complete — cutting first-token latency dramatically.

ASR providers to show in your diagram: OpenAI Whisper (API or self-hosted), Deepgram Nova-3, AssemblyAI Universal-2, Google Cloud Speech-to-Text v2, Amazon Transcribe Streaming, ElevenLabs STT. Mark whether each is streaming or batch, and show the voice activity detection (VAD) component that determines turn boundaries.

Stage 2: LLM inference

The transcribed text is sent to an LLM, which produces the response. Voice AI imposes unique constraints on LLM selection: responses must be short (a few sentences), conversational in tone, and generated quickly. Large reasoning models (o3, Claude Opus with extended thinking) are poorly suited for voice — their 5–60 second latency is incompatible with real-time conversation. Instead, voice pipelines typically use fast models: GPT-4o mini, Claude Haiku, Gemini Flash, or Llama 3 deployed on dedicated GPU infrastructure.

Your diagram should show: the LLM endpoint, the conversation history management system (how previous turns are stored and truncated to fit the context window), the system prompt, any tool-calling layer, and streaming token output. LLM streaming output to the TTS layer is critical — the TTS stage should start synthesizing as soon as a complete sentence arrives, not after the full LLM response is done.

Stage 3: Text-to-Speech (TTS)

TTS converts the LLM's text response to audio. The primary architecture consideration is streaming TTS: modern TTS APIs ( ElevenLabs, OpenAI TTS, PlayHT, Cartesia) support streaming — they begin returning audio chunks before the full text is available. Combined with streaming ASR and streaming LLM output, this produces a fully pipelined voice system where audio begins playing within ~300ms of the user finishing their utterance. Non-streaming TTS adds 500ms–3s of synthesis latency and should be avoided in real-time voice systems.

Key TTS providers for your diagram: ElevenLabs (high quality, streaming), OpenAI TTS-1/TTS-1-HD, Cartesia Sonic (ultra-low latency, ~90ms TTFB), PlayHT 3.0, Google Cloud TTS. Show the voice character/ voice ID selection as a configurable parameter, and mark whether phoneme-level streaming is supported.

Transport and audio infrastructure

WebRTC transport

For browser and mobile voice AI, WebRTC is the standard transport. It provides sub-100ms audio latency, echo cancellation, noise suppression, and automatic bitrate adaptation. Your diagram should show the WebRTC peer connection between the client and your media server, the STUN/TURN servers for NAT traversal, and the audio processing pipeline (codec selection, packet loss concealment, jitter buffer).

A common WebRTC voice AI stack: LiveKit (open-source WebRTC server) or Daily.co on the media server side, paired with browser-native WebRTC APIs or LiveKit client SDKs on the client side. LiveKit agents support tight integration between WebRTC audio and LLM/TTS pipelines, making it a popular choice for production voice AI.

Telephony integration (Twilio, Vonage)

For PSTN phone integration (customer support, IVR replacement, outbound calling), the transport layer is telephony rather than WebRTC. Twilio Media Streams routes audio from a phone call to your application via WebSocket, where it passes through ASR → LLM → TTS before the response is streamed back via Twilio's <Stream> verb. The architecture adds a Twilio webhook layer and bidirectional WebSocket audio channel to the three-stage pipeline.

The OpenAI Realtime API architecture

OpenAI's Realtime API (released in late 2024) offers an alternative architecture that collapses the three-stage pipeline into a single WebSocket connection. Rather than running ASR → LLM → TTS as separate steps, the Realtime API processes audio natively — the model hears audio, reasons about it, and speaks audio output directly, without an explicit text transcription layer. This enables significantly lower end-to-end latency (~300–500ms) and more natural conversation dynamics including interruption handling.

"A voice AI customer support system uses the OpenAI Realtime API. The client (React web app) opens a WebSocket connection to the Realtime API via a short-lived ephemeral key obtained from the app's backend. Audio is streamed bidirectionally over the WebSocket — the client sends microphone audio and receives synthesized speech audio. The backend configures the session with a system prompt, voice character ('alloy'), turn detection (server-VAD mode), and five function tools: lookup_order, update_shipping_address, initiate_return, escalate_to_human, and fetch_faq. When the model calls a function, the WebSocket event is relayed to the backend, which executes the function against the CRM API and returns the result. Audio output is played via the Web Audio API. Conversation transcripts are logged asynchronously after each turn."

Voice AI architecture patterns

Pattern: Full-duplex streaming voice agent

"A real-time voice coding assistant. Audio path: browser microphone → Deepgram Nova-3 streaming STT (WebSocket, returns partial transcripts every 200ms) → Node.js backend → GPT-4o mini via SSE (streaming response, sentence chunking) → Cartesia Sonic TTS streaming (first audio chunk returned within 90ms of receiving sentence) → browser audio player via Web Audio API. Interruption handling: when new ASR input arrives while TTS is playing, the backend cancels the in-flight TTS stream and LLM completion, clears the pending audio queue, and starts a new LLM request. Conversation history: last 10 turns stored in memory, truncated to 4,000 tokens. Tool: execute_code runs code in an E2B sandbox and returns stdout."

Pattern: Twilio AI phone agent

"An inbound phone support agent. Twilio receives a call, plays a brief greeting via TwiML, then opens a Media Stream WebSocket to the Node.js backend. Incoming audio (mulaw 8kHz) is resampled to 16kHz PCM and sent to Deepgram for streaming transcription. Transcripts are sent to Claude Haiku 4.5 with a customer service system prompt and three tools: lookup_account (Postgres), create_ticket (Zendesk), and schedule_callback (Google Calendar). LLM responses are streamed sentence by sentence to ElevenLabs TTS. ElevenLabs audio (MP3) is transcoded to mulaw 8kHz and streamed back to Twilio via the Media Stream WebSocket. Silence detection: if the user is silent for 8 seconds, the agent prompts them. Call recording is enabled via Twilio for quality assurance. Conversation transcript is written to Postgres after call end."

Latency budget reference

Pipeline stage	Typical latency	Optimization
End-of-speech detection (VAD)	200 – 500ms	Tune silence threshold; streaming ASR reduces this
ASR transcription (batch)	300 – 800ms	Switch to streaming ASR; start LLM on partial transcript
LLM first token (GPT-4o mini)	200 – 400ms	Use faster model tier; reduce system prompt length
First sentence generation	300 – 700ms	Prompt model to lead with short first sentence
TTS first audio chunk	80 – 300ms	Cartesia ~90ms; start TTS on first sentence, not full response
Total end-to-end (streaming pipeline)	500 – 900ms	Target <700ms for natural feel
OpenAI Realtime API	300 – 500ms	Native audio-to-audio, no separate ASR/TTS stages

Frequently asked questions about voice AI architecture

What is a voice AI architecture diagram?

A voice AI architecture diagram shows all the components of a real-time voice AI system — the audio capture layer, voice activity detection, ASR engine, LLM inference backend, TTS synthesis engine, audio transport (WebRTC or telephony), conversation history management, and tool execution layer. It traces the audio path from the user's microphone to the AI's spoken response and back, and annotates each stage with its latency contribution and technology choice.

Should I use the OpenAI Realtime API or a traditional ASR → LLM → TTS pipeline?

The OpenAI Realtime API offers lower latency and more natural conversation (including interruption handling) but is limited to OpenAI models and has a higher per-minute cost than a custom pipeline. A traditional three-stage pipeline gives you full flexibility — you can swap ASR, LLM, and TTS providers independently, use open-source models, and fine-tune each stage for cost and quality. Choose the Realtime API for rapid prototyping and consumer-grade voice UX; choose a custom pipeline for cost optimization, fine-tuned models, or multi-provider redundancy.

How do I handle interruptions in a voice AI system?

Interruption handling requires the architecture to monitor incoming ASR in parallel with outgoing TTS audio. When new speech is detected while the AI is speaking, the system must: (1) cancel the in-flight TTS stream, (2) abort the pending LLM generation if it hasn't finished, (3) flush the audio playback queue on the client, and (4) start a new LLM request with the partial context. The OpenAI Realtime API handles this natively. In a custom pipeline, interruption handling is typically implemented via a state machine in the backend WebSocket handler.

Ready to try it yourself?

Start Creating - Free