Voice Agent
Let AI Assistants serve customers through "listening" and "speaking", supporting real-time conversations, STT/LLM/TTS pipelines, and interruption control
What Is This?
Voice Agent enables AI Assistants to interact through voice: users speak, and the AI responds by speaking back. It integrates real-time voice models, STT (Speech-to-Text), TTS (Text-to-Speech), and interruption control under the hood, making it suitable for phone customer service, IVR, voice assistants, and similar scenarios.
Once enabled, the AI Assistant gains a "Voice Call" interface where users can tap the microphone to start a conversation.
Three Interaction Modes
Different modes suit different needs, depending on latency, voice customization flexibility, and tool support.
Realtime
Uses a voice model for direct real-time voice conversation
Scenarios requiring the lowest latency where the default voice is acceptable
Realtime + TTS
Real-time voice model paired with custom TTS output
Scenarios needing a custom brand voice while maintaining a real-time feel
STT + LLM + TTS
Traditional pipeline: Speech-to-Text → LLM → Text-to-Speech
Scenarios requiring full tool support, where slightly higher latency is acceptable and maximum flexibility is desired
Quick decision: For the fastest and most natural experience → Realtime; for a branded voice → Realtime + TTS; for maximum flexibility and tool support → STT + LLM + TTS.
Interruption Control (Turn Handling)
A critical aspect of the voice conversation experience — can the user interrupt the AI while it's speaking? How does the system determine whether the user is actually speaking versus background noise or filler words?
Adjustable parameters:
Minimum speech duration (seconds)
The user's speech must last at least this many seconds to be recognized as "I want to interrupt"
Minimum word count
The user must say at least this many words to trigger an interruption
Note:
min_durationandmin_wordsare only effective in STT + LLM + TTS mode. In Realtime mode, interruption detection is handled internally by the voice model.
Conversation States
During a voice conversation, the interface displays the current state:
Listening: The AI is receiving the user's voice input
Thinking: The AI is processing (searching the knowledge base, using tools, generating a response)
Responding: The AI is replying with voice output
Initializing: The connection has just been established and is being prepared
How Is This Different from Text Chat?
Input
Keyboard typing, file attachments
Microphone voice input, possibly DTMF key presses
Output
Text, Markdown, images, files
Voice, with a text transcript appended afterward
Latency requirements
Seconds-level is acceptable
Must be milliseconds-level to feel natural
Best for
Detailed information needs, reviewing conversation history
Immediate responses, hands-busy situations, phone channels
Best Use Cases
Phone customer service: Replace traditional IVR — AI directly answers calls, listens to questions, and provides answers
Voice query systems: Customers call to check orders, balances, or insurance policies
Voice FAQ: Common questions answered by simply speaking
Driving / hands-busy scenarios: Users can't type but need AI assistance
Accessibility needs: More friendly for users who have difficulty typing
Limitations
Tool support: Currently, Realtime and Realtime + TTS modes only support MCP Tools; to use API Tools, switch to STT + LLM + TTS mode
Knowledge base search: Voice Agent searches all linked knowledge bases and cannot be restricted to specific documents only
Microphone permissions: Users must grant browser microphone access before the conversation can begin
What Do I Need to Do?
Go to AI Assistant settings — Navigate to the settings page of the Agent you want to enable voice for
Select voice agent mode — Choose one of the three: Realtime / Realtime + TTS / STT + LLM + TTS
Configure provider and settings — Select the corresponding STT / TTS / Realtime provider and JSON configuration for your chosen mode
Adjust interruption control (optional) — In STT + LLM + TTS mode, adjust minimum duration and minimum word count
Test — Open a voice call in the interface and confirm that the Listening, Thinking, and Responding states all work smoothly
Further Reading
Last updated
Was this helpful?