Talking avatars

<script src="https://cdn.callers.ai/sdk.js" data-agent="YOUR_AGENT_ID" data-avatar="maya"></script>

​

How it works

​

What drives the face

• Lip sync from the audio. The SDK reads the spectrum of the agent’s voice each frame: loudness sets how far the mouth opens, the low formant bands pick the vowel shape (open, spread or rounded) and the high band catches sibilants and fricatives. This works with any voice and needs no training.
• Lip shapes from the text. On avatar calls the SDK asks Callers for the character timings of each sentence the agent speaks (the X-Speech-Timings call header). Audio alone can’t tell a closed-lip “m” from a quiet moment, so the timings close the lips on m/b/p, bring the lip to the teeth on f/v and shape the vowels, while the audio still decides how far the mouth opens. Voices from providers that don’t return timings fall back to audio-only lip sync.
• Behavior. A behavior controller adds everything the face does besides talking: blinks, small gaze shifts, head motion that follows the energy of the phrase, breathing, and a listening pose when the visitor speaks. When the visitor talks over the agent (barge-in), the face reacts and pending lip timings are dropped.
• Emotions. Each sentence in your script carries an emotion tag (for example [cheerful]). The SDK maps the tags to a few customer-facing looks and applies them when the words are heard, after the voice delay described below.

​

How the face is rendered

​

Keeping lips and voice together

• The engine lag is measured live during the call by comparing the voice envelope with the rendered mouth opening (75 ms is used until enough speech has been heard).
• Inference time, one face interval and one display frame are added; the audio device’s own output latency is subtracted, because it already delays what the visitor hears.
• The delay is capped at 300 ms and changes glide slowly, because fast changes to a delay line audibly warp the voice.

​

When the avatar appears

• Voice calls only. The avatar starts when the visitor clicks to call and stays for the whole call. Text conversations never show it.
• It lives in the call window’s avatar area, or in your own element with avatarContainer.
• A misconfigured avatar (a typo in the name, a blocked download, no WebGL) logs a console warning starting with Avatar and the call continues without it. The avatar can never stop the widget from loading or a call from connecting.

​

Choose an avatar

​

Stock avatars

<script src="https://cdn.callers.ai/sdk.js" data-agent="YOUR_AGENT_ID" data-avatar="john"></script>

import { WebCall, STOCK_AVATARS } from '@callers/web-sdk';

console.log(STOCK_AVATARS.map((avatar) => avatar.id)); // ['maya', 'john']

const webCall = new WebCall({ token, avatar: 'john' });

​

Custom avatars

<script src="https://cdn.callers.ai/sdk.js" data-agent="YOUR_AGENT_ID" data-avatar="clinic-host-v2"></script>

​

Hosting a package yourself

​

Layouts

new WebCall({ token, avatar: 'maya', avatarLayout: 'expanded' });

​

Framing

​

Backgrounds

​

Preset scenes

<script src="https://cdn.callers.ai/sdk.js" data-agent="YOUR_AGENT_ID"
  data-avatar="maya" data-avatar-background="preset:clinic"></script>

new WebCall({ token, avatar: 'maya', avatarBackground: 'preset:clinic' });

// A brand color
new WebCall({ token, avatar: 'maya', avatarBackground: '#0b1f3a' });

// Your own image, written as CSS url()
new WebCall({ token, avatar: 'maya', avatarBackground: 'url(https://example.com/office.jpg)' });

// A file the visitor picked on your page (lives only in that tab)
new WebCall({ token, avatar: 'maya', avatarBackground: `url(${URL.createObjectURL(file)})` });

​

How backgrounds work

​

Render the avatar in your own element

<div id="agent-face"></div>

<style>
  #agent-face {
    width: 360px;
    height: 480px;
    border-radius: 24px;
    overflow: hidden;
    background: radial-gradient(circle at 50% 30%, #f5f1ea, #d9d2c7);
    box-shadow: 0 20px 60px rgba(0, 0, 0, 0.15);
  }
  @media (max-width: 600px) {
    #agent-face { width: 100%; height: 60vw; }
  }
</style>

<script src="https://cdn.callers.ai/sdk.js"></script>
<script>
  const webCall = new CallersWebSdk.WebCall({
    token,
    avatar: 'maya',
    avatarContainer: '#agent-face',
    avatarBackground: 'transparent', // the gradient above shows through
  });
</script>

​

Configuration reference

​

Runtime API

​

webCall.avatarStats

const timer = setInterval(() => {
  const stats = webCall.avatarStats;
  if (stats) console.table(stats);
}, 2000);
webCall.onCallEnded = () => clearInterval(timer);

​

AvatarSession

import { AvatarSession, type AvatarSessionOptions, type AvatarStats } from '@callers/web-sdk';

// Start from a user gesture (a click), so the browser lets the audio context run.
const session = await AvatarSession.start({
  avatarUrl: 'https://cdn.callers.ai/avatars/maya', // a packaged avatar folder
  container: document.querySelector('#stage'),
  playAgentAudio: true,          // play the voice through the avatar, delayed to match the lips
  isMuted: () => speakerMuted,   // polled every frame
  framing: 'photo',              // 'face' | 'photo'
  background: 'preset:studio',   // any avatarBackground value
  onPlaybackChange: () => {},    // the browser suspended or resumed audio
});

session.setAgentStream(remoteStream);    // the agent's MediaStream; null to detach
session.setEmotion('cheerful');          // the emotion of the sentence starting now
session.addSpeechTimings(timings);       // character timings of a sentence (optional)
session.clearSpeechTimings();            // a new reply started; drop pending timings
session.interrupt();                     // the visitor talked over the agent
session.setFraming('face');
console.log(session.stats);              // AvatarStats
console.log(session.playsAgentAudio);    // true while the avatar is the one playing the voice
await session.dispose();                 // release the model, canvas and audio context

​

Events and callbacks

• onCallStarted / onCallEnded bracket the avatar’s lifetime. The model starts loading on the dial click and the avatar is removed when the call ends.
• onTranscriptChanged messages carry llmResponseSentences, each with the sentence’s emotions and, on avatar calls, timings. These are the same data the avatar uses, so you can mirror its state in your own UI.
• onError is not called for avatar problems. They are logged as console warnings starting with Avatar, and the call continues.
• onDeafenChange reflects the speaker toggle; the avatar honors it too, so muting the speaker also silences the avatar’s playback.

​

Privacy and security

• The avatar is rendered from the agent’s audio in the visitor’s browser. No camera, microphone or screen access is involved beyond the call’s own microphone permission, and no video leaves the device.
• Library models are sealed: the files on the CDN can’t be opened or run as models outside the SDK.
• Tell visitors they’re talking to an AI. An avatar makes the agent feel more human, which makes disclosure more important, not less.

​

Performance and browser support

​

What gets downloaded

​

WebGPU and the fallback

​

Guidance

• Desktop with WebGPU: any layout and background works well.
• Recent phones: use compact or minimized; prefer photo or a preset over blur, which costs the most per frame.
• Older or low-end phones: consider leaving the avatar off for that device class, or starting minimized so the visitor opts in. Check inferenceMs against displayFrameMs in avatarStats.
• The audio delay that keeps lips and voice together never exceeds 300 ms, so even a slow device doesn’t make the conversation feel laggy; the face just updates less often.

​

Try it in the playground

​

In the dashboard

​

Complete examples

<!-- John, expanded, on the front-desk scene, launcher bottom-left -->
<script
  src="https://cdn.callers.ai/sdk.js"
  data-agent="YOUR_AGENT_ID"
  data-avatar="john"
  data-avatar-layout="expanded"
  data-avatar-background="preset:reception"
  data-launcher-position="bottom-left"
></script>

import { WebCall } from '@callers/web-sdk';

// `token` comes from your server; see /sdk/authentication
const webCall = new WebCall({
  token,
  avatar: 'maya',
  avatarLayout: 'compact',
  avatarBackground: 'blur',
  onCallStarted: () => console.log('avatar backend:', webCall.avatarStats?.backend),
});

import { useEffect, useRef } from 'react';
import { WebCall } from '@callers/web-sdk';

export function AgentCard({ token }) {
  const faceRef = useRef(null);
  const webCallRef = useRef(null);

  useEffect(() => {
    webCallRef.current = new WebCall({
      token,
      avatar: 'maya',
      avatarContainer: faceRef.current, // an element works as well as a selector
      avatarBackground: 'transparent',
      showFab: false, // we provide our own button
    });
    return () => webCallRef.current?.destroy();
  }, [token]);

  return (
    <div className="agent-card">
      <div ref={faceRef} className="agent-face" />
      <button onClick={() => webCallRef.current?.call()}>Talk to Maya</button>
    </div>
  );
}

​

Troubleshooting

​

FAQ

​

Related