Build realtime voice agents on AI Gateway
By Steven Van ·
Realtime voice, text to speech, and speech to text launch in beta on AI Gateway with OpenAI and xAI models, via AI SDK 7.
Vercel's Vercel AI Gateway now supports audio, adding realtime voice, text to speech, and speech to text alongside the text, image, and video calls it already handles. The audio launch ships with models from OpenAI and xAI, is in beta, and works with AI SDK 7. Every audio call gets the same provider routing, observability, spend controls, and bring-your-own-key support as other modalities.
- Realtime voice: live audio in and out for two-way conversation, for voice assistants and customer support agents.
- Text to speech: text in, audio file out, for voiceovers and spoken responses.
- Speech to text: recorded audio in, text out, for transcribing voice notes and call recordings.
Realtime voice agents use a single model that hears and produces audio directly, instead of chaining speech-to-text, a language model, and text-to-speech. In the browser, the useRealtime hook handles the WebSocket connection, microphone capture, and playback, while a server-minted, short-lived token keeps the Gateway API key off the client. Sessions are capped at 25 minutes, close after 5 minutes idle or if the client sends nothing in the first 30 seconds, and reject messages over 256 KB. Image input isn't supported in realtime sessions, and reconnecting starts a new session rather than resuming the old one.