Appearance
For clean Markdown of any page, append .md to the page URL. For a complete documentation index, see For full documentation content, see For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at
Voice Agent API
For the complete documentation index, see llms.txt
Stream microphone audio into a single WebSocket and receive the agent's spoken response back in real time. No separate STT, LLM, or TTS services to wire up.
Under the hood, the Voice Agent API isn't just stitching together STT, LLM, and TTS. It's an orchestrated pipeline of specialized speech models: voice focus (noise cancellation), Universal-3 Pro speech recognition and understanding, intelligent turn detection, intelligent interruption handling, agent reasoning, and TTS. These models work together so the agent listens, decides, and speaks the way a human would in real conversation. Tool calling, barge-in, and turn-taking are built in. Voice focus is enabled by default, so the agent stays accurate in noisy environments and around background speakers without any extra configuration.
Endpoint: wss://agents.assemblyai.com/v1/ws
Jump to the Quickstart below for a complete browser-based agent, or browse the topic guides in the sidebar to learn each piece in depth.
Connection
Endpoint
wss://agents.assemblyai.com/v1/wsAuthentication
Pass your API key as a Bearer token in the HTTP upgrade request:
Authorization: Bearer YOUR_API_KEYFor client-side apps (where you can't set custom headers or expose your API key), generate a short-lived temporary token on your server and pass it as a query parameter instead:
wss://agents.assemblyai.com/v1/ws?token=YOUR_TEMP_TOKENSee Browser integration for the full token flow.
Resuming a session
Sessions are preserved for 30 seconds after every disconnection. Reconnect using session.resume with the session_id from the previous session.ready event to preserve conversation context. See session.resume for the exact event.
Quickstart
A complete browser-based demo. Enter your API key, pick a voice, and start talking. Echo cancellation is enabled automatically so you don't need headphones.
This quickstart passes your API key directly to the WebSocket for simplicity. For production apps, never expose your API key in client-side code. Generate temporary tokens on your server instead.
Grab your API key from your AssemblyAI dashboard.
Save the following as voice-agent.html:
html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Voice Agent Quickstart | AssemblyAI</title>
<style>
:root {
--brand: #364DEA; --brand-dark: #2B3EC4; --brand-bg: #EEF1FE;
--green: #12B886; --red: #FA5252;
--s50: #F8FAFC; --s100: #F1F5F9; --s200: #E2E8F0;
--s300: #CBD5E1; --s400: #94A3B8; --s500: #64748B;
--s600: #475569; --s700: #334155; --s800: #1E293B; --s900: #0F172A;
}
*, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; }
html, body { height: 100%; }
body {
font-family: system-ui, -apple-system, sans-serif;
color: var(--s900); display: flex; flex-direction: column;
background:
radial-gradient(1200px 600px at 80% -10%, #DCE3FE 0%, transparent 60%),
radial-gradient(900px 500px at -10% 110%, #E6FCF5 0%, transparent 55%),
var(--s50);
}
header {
background: rgba(255,255,255,.85); backdrop-filter: blur(12px);
border-bottom: 1px solid var(--s200);
padding: 0 1.5rem; height: 3.5rem;
display: flex; align-items: center; gap: .75rem;
flex-shrink: 0;
}
.logo img { height: 22px; display: block; }
.page-title { font-size: .875rem; color: var(--s500); padding-left: .75rem; border-left: 1px solid var(--s200); }
.header-spacer { flex: 1; }
.status {
display: flex; align-items: center; gap: .5rem;
font-size: .8125rem; color: var(--s500);
padding: .375rem .75rem; border-radius: 999px;
background: var(--s100); border: 1px solid var(--s200);
}
.dot { width: 8px; height: 8px; border-radius: 50%; background: currentColor; flex-shrink: 0; }
.status.ok { color: var(--green); background: #E6FCF5; border-color: #C3FAE8; }
.status.ok .dot { animation: pulse 2s ease-in-out infinite; }
.status.err { color: var(--red); background: #FFF5F5; border-color: #FFE3E3; }
@keyframes pulse { 0%,100% { opacity: 1; } 50% { opacity: .3; } }
.layout { flex: 1; display: grid; grid-template-columns: 360px 1fr; min-height: 0; }
@media (max-width: 800px) { .layout { grid-template-columns: 1fr; } }
aside {
border-right: 1px solid var(--s200);
background: rgba(255,255,255,.6); backdrop-filter: blur(8px);
padding: 1.5rem; overflow-y: auto;
display: flex; flex-direction: column; gap: 1rem;
}
aside h2 {
font-size: .6875rem; font-weight: 600; color: var(--s500);
text-transform: uppercase; letter-spacing: .08em; margin-bottom: .5rem;
}
.field { display: flex; flex-direction: column; gap: .375rem; }
label { font-size: .75rem; font-weight: 500; color: var(--s600); }
input, select, textarea {
width: 100%; padding: .5rem .625rem; border: 1px solid var(--s200); border-radius: 8px;
font: inherit; font-size: .875rem; color: var(--s900); background: #fff;
transition: border-color .15s, box-shadow .15s;
}
input:focus, select:focus, textarea:focus { outline: none; border-color: var(--brand); box-shadow: 0 0 0 3px rgba(54,77,234,.12); }
textarea { resize: vertical; min-height: 96px; line-height: 1.5; }
.btn {
width: 100%; padding: .75rem 1rem; border: none; border-radius: 10px;
font-size: .9375rem; font-weight: 600; cursor: pointer; color: #fff; background: var(--brand);
transition: all .15s; display: flex; align-items: center; justify-content: center; gap: .5rem;
box-shadow: 0 1px 2px rgba(54,77,234,.3), 0 4px 12px rgba(54,77,234,.15);
}
.btn:hover { background: var(--brand-dark); transform: translateY(-1px); }
.btn:disabled { opacity: .5; cursor: default; transform: none; }
.btn.on { background: var(--red); box-shadow: 0 1px 2px rgba(250,82,82,.3), 0 4px 12px rgba(250,82,82,.15); }
.btn.on:hover { background: #e03131; }
.btn svg { width: 18px; height: 18px; }
main {
display: flex; flex-direction: column; min-height: 0;
padding: 1.5rem 2rem 2rem;
}
.transcript {
flex: 1; min-height: 0; display: flex; flex-direction: column;
background: #fff; border: 1px solid var(--s200); border-radius: 16px;
overflow: hidden;
box-shadow: 0 1px 2px rgba(15,23,42,.04), 0 4px 16px rgba(15,23,42,.04);
}
.transcript-hd {
padding: .75rem 1.25rem; background: var(--s50); border-bottom: 1px solid var(--s200);
font-size: .6875rem; font-weight: 600; color: var(--s500);
text-transform: uppercase; letter-spacing: .08em;
display: flex; justify-content: space-between; align-items: center;
}
.speakers { display: flex; gap: .375rem; }
.speaker {
display: flex; align-items: center; gap: .375rem;
padding: .25rem .625rem; border-radius: 999px;
background: var(--s100); color: var(--s400);
font-size: .6875rem; font-weight: 600;
text-transform: uppercase; letter-spacing: .05em;
transition: background .2s, color .2s;
}
.speaker .dot { width: 6px; height: 6px; }
.speaker.user.active { background: var(--brand-bg); color: var(--brand); }
.speaker.agent.active { background: #E6FCF5; color: var(--green); }
.speaker.active .dot { animation: pulse 1s ease-in-out infinite; }
#msgs { flex: 1; overflow-y: auto; padding: 1rem 1.25rem; display: flex; flex-direction: column; gap: .5rem; }
.empty {
flex: 1; display: flex; align-items: center; justify-content: center;
color: var(--s400); font-size: .875rem;
}
.msg {
padding: .75rem 1rem; border-radius: 12px;
font-size: .9375rem; line-height: 1.5;
max-width: 85%; animation: slideIn .25s ease;
}
@keyframes slideIn { from { opacity: 0; transform: translateY(4px); } to { opacity: 1; transform: none; } }
.msg .who {
font-size: .6875rem; font-weight: 600; text-transform: uppercase;
letter-spacing: .05em; color: var(--s500); margin-bottom: .25rem;
}
.msg.u { background: var(--brand-bg); align-self: flex-end; }
.msg.u .who { color: var(--brand); }
.msg.a { background: #E6FCF5; align-self: flex-start; }
.msg.a .who { color: var(--green); }
</style>
</head>
<body>
<header>
<a class="logo" href="">
<img src="" alt="AssemblyAI">
</a>
<span class="page-title">Voice Agent Quickstart</span>
<div class="header-spacer"></div>
<div class="status" id="status"><span class="dot"></span><span id="status-text">Ready</span></div>
</header>
<div class="layout">
<aside>
<div>
<h2>Configuration</h2>
<div class="field">
<label for="key">API key</label>
<input id="key" type="password" placeholder="Your AssemblyAI API key">
</div>
</div>
<div class="field">
<label for="mic">Microphone</label>
<select id="mic"><option value="">Default microphone</option></select>
</div>
<div class="field">
<label for="voice">Voice</label>
<select id="voice">
<optgroup label="English">
<option value="ivy" selected>🇺🇸 ivy</option>
<option value="james">🇺🇸 james</option>
<option value="tyler">🇺🇸 tyler</option>
<option value="winter">🇺🇸 winter</option>
<option value="sam">🇺🇸 sam</option>
<option value="mia">🇺🇸 mia</option>
<option value="bella">🇺🇸 bella</option>
<option value="david">🇺🇸 david</option>
<option value="jack">🇺🇸 jack</option>
<option value="kyle">🇺🇸 kyle</option>
<option value="helen">🇺🇸 helen</option>
<option value="martha">🇺🇸 martha</option>
<option value="river">🇺🇸 river</option>
<option value="emma">🇺🇸 emma</option>
<option value="victor">🇺🇸 victor</option>
<option value="eleanor">🇺🇸 eleanor</option>
<option value="sophie">🇬🇧 sophie</option>
<option value="oliver">🇬🇧 oliver</option>
</optgroup>
<optgroup label="Multilingual">
<option value="arjun">🇮🇳 arjun (Hindi/Hinglish)</option>
<option value="ethan">🇨🇳 ethan (Mandarin)</option>
<option value="dmitri">🇷🇺 dmitri (Russian)</option>
<option value="lukas">🇩🇪 lukas (German)</option>
<option value="lena">🇩🇪 lena (German)</option>
<option value="pierre">🇫🇷 pierre (French)</option>
<option value="mina">🇰🇷 mina (Korean)</option>
<option value="ren">🇯🇵 ren (Japanese)</option>
<option value="mei">🇨🇳 mei (Mandarin)</option>
<option value="joon">🇰🇷 joon (Korean)</option>
<option value="giulia">🇮🇹 giulia (Italian)</option>
<option value="luca">🇮🇹 luca (Italian)</option>
<option value="lucia">🇪🇸 lucia (Spanish)</option>
<option value="hana">🇯🇵 hana (Japanese)</option>
<option value="mateo">🇪🇸 mateo (Spanish)</option>
<option value="diego">🇨🇴 diego (Spanish, LatAm)</option>
</optgroup>
</select>
</div>
<div class="field">
<label for="prompt">System prompt</label>
<textarea id="prompt">You are a friendly voice assistant having a casual conversation. Keep replies short and natural, usually one or two sentences. Speak the way a person would in real conversation: relaxed, low-key, no exclamation marks, no over-enthusiastic phrases.</textarea>
</div>
<div class="field">
<label for="greeting">Greeting</label>
<input id="greeting" value="Hey, what's on your mind?">
</div>
<button class="btn" id="btn">
<svg id="btn-icon" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round">
<rect x="9" y="2" width="6" height="10" rx="3"/>
<path d="M19 10v1a7 7 0 01-14 0v-1"/><path d="M12 18v4"/><path d="M8 22h8"/>
</svg>
<span id="btn-label">Connect</span>
</button>
</aside>
<main>
<div class="transcript" id="log">
<div class="transcript-hd">
<span>Transcript</span>
<div class="speakers">
<div class="speaker user" id="spk-user"><span class="dot"></span>You</div>
<div class="speaker agent" id="spk-agent"><span class="dot"></span>Agent</div>
</div>
</div>
<div id="msgs">
<div class="empty" id="empty-msg">Add your API key on the left and click Connect to start the conversation</div>
</div>
</div>
</main>
</div>
<script>
const $ = (id) => document.getElementById(id);
const RATE = 24_000;
// Inline AudioWorklet that captures mic as PCM16 and posts to main thread
const workletUrl = URL.createObjectURL(new Blob([`
class P extends AudioWorkletProcessor {
process(inputs) {
const ch = inputs[0]?.[0];
if (ch) {
const buf = new Int16Array(ch.length);
for (let i = 0; i < ch.length; i++)
buf[i] = Math.max(-32768, Math.min(32767, ch[i] * 32767));
this.port.postMessage(buf.buffer, [buf.buffer]);
}
return true;
}
}
registerProcessor("pcm", P);
`], { type: 'application/javascript' }));
// --- Microphone enumeration ---
async function populateMics() {
if (!navigator.mediaDevices?.enumerateDevices) return;
try {
const devices = await navigator.mediaDevices.enumerateDevices();
const inputs = devices.filter(d => d.kind === 'audioinput');
const sel = $('mic');
const current = sel.value;
while (sel.firstChild) sel.removeChild(sel.firstChild);
const def = document.createElement('option');
def.value = '';
def.textContent = 'Default microphone';
sel.appendChild(def);
inputs.forEach((d, i) => {
const opt = document.createElement('option');
opt.value = d.deviceId;
opt.textContent = d.label || `Microphone ${i + 1}`;
sel.appendChild(opt);
});
if (current && inputs.some(d => d.deviceId === current)) sel.value = current;
} catch (e) { console.warn('enumerateDevices failed', e); }
}
populateMics();
navigator.mediaDevices?.addEventListener?.('devicechange', populateMics);
// --- Voice Agent ---
let ws, ctx, mic;
$('btn').onclick = () => (ws?.readyState <= 1) ? stop() : start();
async function start() {
const key = $('key').value.trim();
if (!key) return setStatus('Enter your API key', 'err');
$('btn').disabled = true;
setStatus('Connecting…');
try {
ctx = new AudioContext({ sampleRate: RATE });
await ctx.resume();
await ctx.audioWorklet.addModule(workletUrl);
const deviceId = $('mic').value;
mic = await navigator.mediaDevices.getUserMedia({
audio: {
echoCancellation: true,
noiseSuppression: false,
...(deviceId ? { deviceId: { exact: deviceId } } : {}),
},
});
populateMics();
const source = ctx.createMediaStreamSource(mic);
const worklet = new AudioWorkletNode(ctx, 'pcm');
const url = new URL('wss://agents.assemblyai.com/v1/ws');
url.searchParams.set('token', key);
ws = new WebSocket(url);
let ready = false, playT = 0;
worklet.port.onmessage = ({ data }) => {
if (!ready || ws.readyState !== 1) return;
const b = new Uint8Array(data);
let s = ''; for (let i = 0; i < b.length; i++) s += String.fromCharCode(b[i]);
ws.send(JSON.stringify({ type: 'input.audio', audio: btoa(s) }));
};
source.connect(worklet).connect(ctx.destination);
ws.onopen = () => ws.send(JSON.stringify({
type: 'session.update',
session: {
system_prompt: $('prompt').value,
greeting: $('greeting').value,
output: { voice: $('voice').value },
},
}));
ws.onmessage = ({ data }) => {
const m = JSON.parse(data);
switch (m.type) {
case 'input.speech.started':
setSpeaker('user', true); break;
case 'input.speech.stopped':
setSpeaker('user', false); break;
case 'reply.started':
setSpeaker('agent', true); break;
case 'session.ready':
ready = true;
setStatus('Connected', 'ok');
$('btn').disabled = false;
$('btn-label').textContent = 'Disconnect';
$('btn').classList.add('on');
clearEmpty();
break;
case 'reply.audio': {
const raw = atob(m.data);
const pcm = new Int16Array(raw.length / 2);
for (let i = 0; i < pcm.length; i++)
pcm[i] = raw.charCodeAt(i * 2) | (raw.charCodeAt(i * 2 + 1) << 8);
const f32 = new Float32Array(pcm.length);
for (let i = 0; i < pcm.length; i++) f32[i] = pcm[i] / 32768;
const buf = ctx.createBuffer(1, f32.length, RATE);
buf.getChannelData(0).set(f32);
const src = ctx.createBufferSource();
src.buffer = buf; src.connect(ctx.destination);
playT = Math.max(playT, ctx.currentTime);
src.start(playT); playT += buf.duration;
break;
}
case 'reply.done':
setSpeaker('agent', false);
if (m.status === 'interrupted') playT = ctx.currentTime;
break;
case 'transcript.user':
addMsg('You', m.text, 'u'); break;
case 'transcript.agent':
addMsg('Agent', m.text, 'a'); break;
case 'session.error':
setStatus('Error: ' + m.message, 'err'); break;
}
};
ws.onclose = () => { setStatus('Disconnected'); resetUI(); };
ws.onerror = () => { setStatus('Connection failed', 'err'); resetUI(); };
} catch (e) {
setStatus(e.message, 'err'); resetUI();
}
}
function stop() {
ws?.close(); mic?.getTracks().forEach(t => t.stop()); ctx?.close();
ws = ctx = mic = null; resetUI(); setStatus('Disconnected');
}
function resetUI() {
$('btn').disabled = false;
$('btn-label').textContent = 'Connect';
$('btn').classList.remove('on');
setSpeaker('user', false);
setSpeaker('agent', false);
}
function setStatus(msg, cls) {
$('status-text').textContent = msg;
$('status').className = 'status' + (cls ? ' ' + cls : '');
}
function setSpeaker(who, active) {
$('spk-' + who).classList.toggle('active', active);
}
function clearEmpty() {
const e = $('msgs').querySelector('.empty');
if (e) e.remove();
}
function addMsg(who, text, cls) {
clearEmpty();
const d = document.createElement('div');
d.className = 'msg ' + cls;
const whoEl = document.createElement('div');
whoEl.className = 'who';
whoEl.textContent = who;
const textEl = document.createElement('div');
textEl.textContent = text;
d.appendChild(whoEl);
d.appendChild(textEl);
$('msgs').appendChild(d);
$('msgs').scrollTop = $('msgs').scrollHeight;
}
</script>
</body>
</html>The AudioWorklet processor is inlined using a Blob URL so this works as a single file, with no extra .js file needed. The key line new AudioContext({ sampleRate: 24000 }) forces the audio context to match the default audio/pcm encoding rate, avoiding any manual resampling.
Browsers require a secure context for microphone access, so serve the file locally:
bash
npx serve .Open ` enter your API key, and click Connect.
Event flow
A typical voice agent session moves through the events in this order:
Client Server
│ │
│── WebSocket connect ─────────────►│
│── session.update ────────────────►│ (system prompt + tools + greeting)
│ │
│◄─── session.ready ────────────────│ (save session_id)
│ │
│── input.audio (stream) ──────────►│ (only after session.ready)
│── input.audio (stream) ──────────►│
│ │
│◄─── input.speech.started ─────────│
│◄─── transcript.user.delta ────────│
│◄─── input.speech.stopped ─────────│
│◄─── transcript.user ──────────────│
│ │
│◄─── reply.started ────────────────│
│◄─── reply.audio ──────────────────│
│◄─── transcript.agent ─────────────│
│◄─── reply.done ───────────────────│
│ │
│ [tool call flow] │
│◄─── tool.call ────────────────────│ (arguments is a dict)
│◄─── reply.done ───────────────────│ ← send tool.result here
│── tool.result ───────────────────►│
│◄─── reply.started ────────────────│
│◄─── reply.audio ──────────────────│
│◄─── reply.done ───────────────────│See the events reference for every event's full payload.
Logging for support
The first event the server sends after a successful connection is session.ready, which includes a session_id. Persist this ID for every session along with a timestamp and the API region, not just when something goes wrong. If you contact support@assemblyai.com about a specific session (audio glitches, unexpected interruptions, tool-call issues, session-resume failures), this ID lets us locate it in our logs immediately.
We recommend logging, at minimum:
session_idfromsession.ready- The WebSocket close code and reason on disconnect
- A timestamp for the start of the session
- Whether you connected to the US (
agents.assemblyai.com) or EU endpoint
See Troubleshooting for common issues and the full list of error codes.
Interruptions (barge-in)
Barge-in is built in and semantic. Back-channels like "uh-huh" don't interrupt, but "wait, stop" does. When the user actually interrupts, the server immediately stops the agent and emits:
reply.donewithstatus: "interrupted"transcript.agentwithinterrupted: trueandtexttrimmed to what was actually spoken
Your client should:
- Flush the audio buffer to stop playing stale speech from before the interruption.
- Restart the output stream so it's ready for the next response.
See Turn detection and interruptions for how the model decides what counts as an interruption, and Handling interruptions for the complete client-side flush pattern.
Next steps
- Configure your agent: system prompt, greeting, and turn detection
- Send and play audio: PCM16 format details and playback
- Add tools to your agent: function calling and MCP integration
- Connect from a browser: token-based auth for production apps
- Troubleshooting: common issues and fixes