CallServer owns everything that is the same for every framework. A handler owns only what differs: what to do with a caller’s voice, and where the reply comes from.
CallHandler is a typing.Protocol. Nothing inherits from anything, every method is optional, and a missing method is a no-op, so a handler that only wants audio implements only on_caller_audio. A synchronous method is accepted too: a callback with nothing to await should not be forced to declare async.
handler_factory is called with no arguments and builds one handler per call, so per-call state can live on self and configuration is closed over rather than threaded through the server.
Ordering and failure
Three rules govern when your methods are called, and all three are the same in both SDKs.- Nothing arrives before
on_startreturns. Inbound messages are handled serially, so caller audio packed into the same read assession.startwaits for youron_start. A slow start delays the caller rather than dropping frames. - Context that arrives before the handler exists is dropped. The handler is built when
session.startlands, and every dispatch is guarded on it existing. On a fast dial a participants or recording sentence can beat that frame, and there is nobody to give it to. - An exception ends that call alone. It is logged and the call closes with the reason
handler-failure. One bad call must never take the worker with it.
on_start either, because what “ready” means is a framework’s own business. A handler whose provider socket opens inside on_start still gets caller audio and context the moment that returns, which is what StartupBuffer is for: a bounded holder for the caller’s first words and the first context sentence, with the oldest dropped first.
CallSession
on_start receives the session, and it is the handler’s only way to reach the socket. It is valid until the call ends.
Seventeen members, and this is all of them.
recording_active is one flag the server keeps current, so no plugin re-derives it from the context
sentence it happened to see. A reported recording.status wins over the session.start snapshot,
whichever arrives first: the snapshot omits the field when the state was unknown at answer time, and an
omitted field is not “not recording”.
What session.start carries
Both are frozen dataclasses, and both are on the barrel for a handler that wants to annotate them:from standin import Caller, SessionStart.
None, so treat every caller field as optional. A direction
that is neither inbound nor outbound reads as inbound rather than being carried through as
something a branch has to handle.
The server owns the outbound sequence number and the audio timeline, so a handler that swaps or re-publishes its audio source cannot make timestamps jump backwards. You never compute a seq and you never build a frame.
Answering, and the plugin that has to say so
The server runs a stale-call reaper: 120 seconds a call may run with nothing having answered it, after which it ends asno-agent-answered. It is the gap none of the other watchdogs cover, because
session.start arrived, on_start succeeded, and the caller is still talking the whole time.
Sending audio is answering, so a handler that speaks is covered without doing anything. A handler that
answers by a route the server cannot see, joining a room and letting an agent speak there, has to say
so:
send_audio can drop what you hand it
Past 1 MB of unflushed transport buffer (MAX_AUDIO_BUFFER_BYTES), agent audio is shed rather than queued: send_audio advances the timeline and returns without sending, and logs at most once every five seconds. The caller hears a gap rather than the call wedging, and the timeline still advances because a dropped frame is a gap in what they hear, not a rewind.
Control frames are never shed. A call that cannot be ended is the failure this exists to prevent.
session.buffered_bytes is how you see it coming before a continuous stream starts losing frames. See Backpressure.
on_start
on_start_timeout defaults to 15 seconds, and a handler that exceeds it loses the call with the reason handler-start-timeout. A failure inside it closes as handler-start-failure, not transport-failure, so a provider outage in your plugin is never reported as StandIn’s own socket failing.
Refusing a call
session.end() is safe to call from inside on_start. It asks for the close and returns immediately, because awaiting teardown from inside on_start would deadlock: teardown waits for on_start to return before dispatching your aclose.
on_start to unwind before it dispatches aclose, so anything you build after the refusal still gets closed.
on_caller_audio
frame_duration_ms(pcm) rather than counting frames, and never index into a caller frame at a fixed offset. See Inbound audio.
This runs on the receive path of a live call, so it must not block. Hand the bytes to your provider and return. The frame is already validated, so a truncated or malformed payload is dropped by the server and never reaches here.
A live Microsoft Teams call delivers PCM continuously, silence included. If the frames stop, the call is gone on the far side, and the idle watchdog will end it.
on_context
- participant counts and group-call etiquette, for example
"There are 4 human participants on this call. Stay quiet unless directly addressed." - DTMF key presses, for example
'The caller pressed the "5" key on their keypad.' - recording status changes, for example
"The Microsoft Teams call recording is now ACTIVE."
on_goodbye
If your provider guards response creation on “is a response already active”, a plain
say will be swallowed here, because the goodbye almost always arrives mid-answer. Cancel the active response first, then speak.aclose
reason is the first close cause recorded for the call, not the last.
Anything you pass to
session.end(...) yourself arrives here too, which is how caller-not-allowed in the refusal above reaches your own cleanup.
call-duration-limit is the one that needs planning for: by the time it arrives your handler has already been asked to say the goodbye line, so treat on_goodbye as the last chance to speak rather than aclose.
The optional callbacks
Two more the server duck-types. Most handlers implement neither.on_speaker_change(name) fires when a different person starts speaking, and only when StandIn sends
unmixed audio. It is called on CHANGE only, never per frame: the name rides every inbound audio frame,
and a model told forty times a second who is speaking would hear nothing else.
on_video_frame(frame) gives you every sampled frame of the caller’s camera or screen share. Reach for
session.latest_video_frame() instead when the model only looks on demand.
Neither is on CallHandler, which is a runtime-checkable Protocol: a member there would make
isinstance(handler, CallHandler) fail for every handler written before the callback existed, while
the documented rule says each method is optional. Implementing the method is enough, and inheriting
from SpeakerHandler or VideoHandler is never required.
Barge-in and cancel playback
cancel_playback() is the only lever that un-sends audio already handed to the service. It flushes the platform player, so the caller stops hearing the turn they just interrupted.
Call it the moment your provider reports that the caller started speaking, and call it before you cancel the response upstream. The buffered audio is the part the caller can hear, so it goes first.
FrameAligner.reset() matters because the bytes it is holding back belong to the turn the caller just interrupted: flushing them instead would replay a fragment of the abandoned answer after the silence. See Audio.
cancel_playback() is cheap and safe to call when nothing is playing. Calling it on every detected speech start is the right default.
A full handler
A realtime speech-to-speech handler, with the resampling, the frame alignment and the barge-in in place.RealtimeClient here stands in for whatever provider you use.
Next
CallServer
Every constructor option, and what each one defends against.
Audio
Why the resampler and the aligner are in the SDK rather than in your plugin.