What the SDK gives you

This page is a boundary, not a brochure. It says what is in the SDK, what is in the hosted service, and what belongs to your framework, so you can tell the three apart before you design around one. Everything below ships in both languages, unless a line says otherwise. Names are snake_case in Python and camelCase in TypeScript: where only one spelling appears below, the other is the mechanical translation of it, and the handful of names that are not a mechanical translation are listed in Parity, and where it stops at the end. Each section ends where the page that owns it begins.

​

Calls

• Seven optional callbacks. on_start, on_caller_audio, on_context, on_video_frame, on_speaker_change, on_goodbye, aclose, in camelCase on the TypeScript side. Implement what you need and leave the rest out. on_video_frame and on_speaker_change are optional in exactly the same way as the other five even though Python declares them on two protocols of their own: the server calls whichever of the seven your handler actually has, so implementing the method is the whole requirement and inheriting from anything never is. See the seam for why the split exists.
• One handler per call, built by a factory that closes over your configuration, so nothing is threaded through the server.
• cancel_playback() / cancelPlayback(), the only wire-level lever that un-sends audio already handed to the service. This is what makes a barge-in actually stop the bot.
• end(reason), idempotent and first-reason-wins. Safe to call from inside on_start to refuse a call.
• Call context as plain sentences. Participant counts and group etiquette, DTMF key presses and recording status arrive through on_context already phrased to put in front of a model.
• Errors are contained. An exception from any handler method ends that call alone.

​

Audio

• resample_pcm16 / resamplePcm16, linear interpolation, dependency-free. An odd trailing byte is dropped rather than raised: a truncated frame is a glitch, a raised exception in the audio path is a dropped call.
• FrameAligner with push, flush, reset and pending, because a resampled buffer does not divide evenly into the 640-byte wire frame and dropping the remainder clips every turn.
• frame_duration_ms / frameDurationMs, for a playout clock. Use it rather than counting frames: outbound chunk lengths are not fixed, so a frame count drifts against real time.
• The wire constants, named rather than inlined: SAMPLE_RATE_HZ 16000, REALTIME_SAMPLE_RATE_HZ 24000, FRAME_MS 20, FRAME_BYTES 640, BYTES_PER_SAMPLE 2, NUM_CHANNELS 1.

​

Turning speech into a reply

There are two dialogue modes and the SDK carries both. In the realtime one, a single speech-to-speech provider hears the caller and answers in voice, and it does its own turn-taking. In the streaming one, speech becomes text, your agent answers, and the answer becomes speech again. That second shape needs somebody to decide where the caller stopped talking, ask three services in order, and feed the answer out at the rate a call consumes audio. That somebody is in the SDK.

• VoiceLane, built with your transcribe, answer and synthesize. It runs one turn at a time, a new utterance supersedes the one in flight rather than racing it, and a failed step is spoken rather than logged and swallowed. Silence is the one thing a caller cannot interpret.
• UtteranceSegmenter, with a pre-roll so the syllable that trips the gate is not lost, a silence window so a pause inside a sentence is not the end of one, a floor so a cough is not a turn, and a ceiling so a stuck-open microphone does not grow one utterance for the whole call.
• PacedPlayback, and decode_wav / encode_wav for the 32-bit float, 44.1 kHz or extensible WAV a speech engine hands back, every one of which plays as noise on a call unconverted.
• Barge-in is handled inside the lane: buffered audio is dropped the moment a new utterance opens, not when the old one finishes.

Turn-taking has the whole lane. Dialogue modes is where to start if you are still choosing a shape.

​

In a meeting, and who is speaking

An agent that answers every turn of a meeting is worse than one that was never invited, so the etiquette is in the core rather than in each plugin.

• Silent until addressed. GroupGate answers the turn that named the agent through one of your wake phrases, and then keeps the floor open for DEFAULT_FOLLOW_UP_WINDOW_MS, twelve seconds, so the follow-up question does not have to say the name again. GateDecision reports respond and addressed separately, because a turn answered inside the window was not addressed and only an addressed turn reopens it. is_verbal_interrupt sits beside the gate for the whole utterances that are only “stop”, “wait” or “never mind”.
• A 1:1 always answers. The gate arms itself off session.start.thread_id: a meeting or channel conversation begins 19: and arms it, a one-to-one call has no such thread at all, and every turn of one is answered with addressed still reported truthfully. note_participants(count) is a second, weaker signal that can only add certainty: a participant count never arrives on the meeting-join path, so a gate keyed on the count alone is dead on exactly the calls it exists for.
• A gate nothing could open is off. Configure no wake phrase and gate.active reports false and every turn goes through. A gate with nothing that could ever open it would mute the agent for the whole call, and silence is the one failure a caller cannot tell apart from a dead worker.
• Who said it. session.speaker and the optional on_speaker_change(name) callback name the person talking, which is what attributes a transcript line and tells a model who it is answering. Both arrive only where StandIn sends unmixed audio: on the mixed path, which is most calls, session.speaker is None, so build on it as a hint and never as a requirement. on_speaker_change fires on change only, never per frame.

Group calls has the gate, and the optional callbacks have the speaker.

​

Seeing and being seen

• Frames in. The caller’s camera and screen share arrive as VideoFrames over the same socket. They are sparse and best-effort by contract, the server keeps the latest per source, and latest_video_frame() is the common path because a model asks to look long after the frame arrived. on_video_frame is the ambient one, for a plugin that wants to know the moment one lands.
• Watching without being asked. AmbientVision pushes what changed on screen into the conversation between turns, for the caller who shares a deck and says only “what do you think?”. It is off unless you enable it, because it spends money on every scene change, and its recording gate is on by default: continuously sending somebody’s screen to a model is a different promise from glancing at it once. It keeps a reserve so the ambient lane cannot spend the whole vision budget and leave the caller’s own “look at this” with nothing left.
• Looking back at what is already gone. The session keeps the latest frame per source, which answers “what am I looking at now”. KeyframeStore answers “what was on that slide a moment ago”, which is what somebody actually asks after they have moved on, and the look_back tool is how a model reaches it. Sixteen frames by default, latched per source so a screen nobody touched is kept once, and it keeps nothing at all unless the call is being recorded: a history of somebody’s screen is a materially different promise from glancing at it once.
• A ceiling on looking. VisionBudget allows six looks in any rolling minute by default, as a sliding window rather than a total, so a long call is not punished for having been long. A look that failed is refunded rather than charged. Its reserve, a quarter of the window and never less than two, is the part only an explicit look may spend, which is why the ambient lane must be handed the same budget object your VisionTools holds rather than building its own.
• Pictures out. display_image() draws a chart, a page or a photo on the bot’s tile for a few seconds. DISPLAY_MODES is the pair fullscreen and overlay, and the model picks one per picture: an overlay is a picture-in-picture inset that keeps the face on screen, and it is unreadable for a dense screenshot or a page of a document, which is exactly when somebody says “show me”.
• The avatar hints. express() names the emotion the face should wear, send_speech_marks() carries the viseme timeline that drives lip-sync, and estimate_visemes() and TurnLipSync estimate that timeline from the text for the providers that return no timings at all.
• Your own video. send_tile_frame() and TileStream put continuous frames on the tile for the minority of workers that produce video themselves. Latest wins, and it drops under backpressure rather than queueing, because the voice matters more than the picture.
• A vision model is optional. FrameDescriber.from_env() returns nothing when STANDIN_VISION_API_URL is unset, and that nothing is the signal a plugin uses to tell an agent that looking is not available here.

Vision and the avatar and The avatar own this lane.

​

What the agent can do about the call

• Call tools, declared once and rendered into each provider’s own JSON by tool_schemas(). BUILT_IN_TOOLS is end_call, express, show_image, look and look_back, and the descriptions are written for a model rather than for you: they say when to reach for the tool, because that is the only thing the model reads before deciding. An unknown dialect falls back to the flat shape rather than raising, since a tool the model never sees is worse than one described a little loosely.
• Dispatch never raises. The result is read out loud, so a failure has to arrive as a sentence or the agent simply goes quiet and the caller waits.
• Consulting and background work. CONSULT_TOOL and BACKGROUND_TASK_TOOL let a model hand real work to your agent without stalling the call, and keep the promise to deliver it afterwards.

See Call tools and Consulting and background work.

​

After the call, and reaching people

• Meeting recap. Transcript collects the turns, post_meeting_minutes is the tool a model calls to file them mid-call, and post_minutes is the function that writes them up and posts them. The OpenClaw and Hermes plugins also do this at hang-up when you turn meetingRecap / meeting_recap on: they write a restart-recoverable local spool first, then post through a listen-only chat lane. Off by default. Delivery needs the managed-bot chat lane (Tiers) and a summarization consult. Hang-up does not await the consult. Meeting recaps are best-effort. They may be lost if the worker exits during processing, or if the spool directory is on disk that dies with the process. Point STANDIN_RECAP_DIR at a directory that survives a container restart when you need unfinished recaps to post after one. minutes_prompt asks for Key Points, Decisions, Action Items and, when the agent saw something, Presented, and it forbids inferring what was on screen: minutes that invent the numbers on a dashboard are worse than minutes with a gap.
• Minutes that say who said it. Each Turn keeps its speaker apart from its words, and write_minutes_docx appends the transcript attributed line by line rather than filing the whole call under one name. That is the half a transcript-only recap tool cannot produce, and it is worth the most on the meeting calls where unmixed audio gives a real speaker per utterance. has_speaker_prefix keeps a line that already carries its own name from being relabelled or prefixed twice, because re-labelling destroys the one thing worth keeping and prefixing again reads as a transcription fault. The document is Word-openable with no dependency on anything: a .docx is a zip of four XML parts, and a document library would be a dependency every install pays for so that the few who ask for minutes get a file. Minutes are delivered as text to one conversation; the document is written to disk, and only when you name a document_dir for it. DOCUMENT_NOT_ATTACHED is the sentence the message then carries, because a document promised into the chat would be a promise that quietly fails. Meeting recap.
• Outbound. The call_me_back tool (CALL_BACK_TOOL) rings somebody who asked to be rung, and OutboundLane with PendingMessages parks the line on disk until they answer, because the answering leg is a different call and may be a different process: an in-memory store loses the message silently and the callee picks up to nothing. The whole lane is off until you switch it on: STANDIN_OUTBOUND_ALLOW lists who this agent may ring, an unset list refuses every call, and STANDIN_OUTBOUND_MAX_PER_HOUR caps the rest at six an hour by default. It also needs the Calls.Initiate.All consent in the Graph permissions table. Reaching people.
• Nobody answers, so it goes to chat. Chat is the one fallback this lane has: a parked answer reaches the person in writing or not at all. When StandIn reports no-answer, declined, busy or failed, the line the agent was holding is posted into the conversation the request came from, prefixed so it reads as a missed call and worded for the person who missed it rather than for an operator. One idempotency key covers both the reported outcome and the answer-timeout sweep, because both can fire for one call and being told twice reads as a second attempt that never happened. The one case with nowhere to go is a call whose thread is not a real conversation, which call_thread_is_postable decides: that one is parked rather than posted somewhere wrong, because an answer delivered into the wrong conversation is the worst outcome this lane has. The thread id is how an unanswered call still reaches them.
• Speaking into a call already up. VoiceDelivery says the line into a live call when there is one and only rings back when there is not, and it never raises: every refusal comes back as a sentence, because the thing reading it is either a host that would mark the whole platform failed or a model that would say the exception out loud.

​

Getting a plugin right

Four pieces most plugins used to carry themselves, each closing a failure that is invisible until a real call finds it. Two of them moved into the core precisely because copies had started to drift. GroupGate above is the first of the four, and these are the other three.

• The echo guard, so a speakerphone does not feed the agent its own playback and leave the model answering itself in a loop while the caller is silent. It is the EchoGuard class in Python and the shouldSuppressEcho function in TypeScript.
• StartupBuffer, for what arrives before your provider is ready. Both halves matter: the caller’s first words are often the reason they called, and the first context sentence is what engages a group gate and a recording gate. Five plugins in this SDK once held one half and silently lost the other.
• run_smoke, which rings this worker’s own handler on loopback and proves audio made the round trip, with no provider bill, no public tunnel and no Microsoft tenant.

See Group calls, Realtime providers and Checking the install.

​

Chat

• ChatChannel, dialed out from your worker. No listener, no open port, no tunnel, and no Bot Framework credential in your agent.
• InboundMessage, already authenticated, resolved to your connection, and with the bot @mention stripped. In group and channel scope only messages that mention the bot are relayed at all.
• parse_inbound / parseInbound and build_reply / buildReply. The reply echoes tenantId and conversationId exactly, which is the cross-tenant leak guard the relay rests on.
• The personal-or-group branch, which nearly every agent needs. Python reads it off the message as the msg.is_personal property; TypeScript exports isPersonal(msg) as a function. Same answer, two shapes, so check the page for your language rather than assuming.
• Attachments become one turn. A pasted screenshot, a dragged-in file and a voice note all land as one ChatTurn your agent can answer, and a reply can carry a picture back. See Attachments in chat and Sending a picture.
• A voice note becomes text. transcribe_voice_messages fetches the clips and runs the Transcriber you supply, and ChatTurn.voice_note is what they said. Supply no transcriber and it returns empty rather than raising, and one clip that fails costs that clip and nothing else: a message with a picture and a voice note must still be answered. Voice notes.
• Per-conversation ordering and at-least-once dedupe, so replies in a conversation cannot overtake each other and a redelivery does not run your agent twice.
• SCHEMA_VERSION is a major version. Additive evolution does not bump it, because the schema already requires receivers to ignore unknown fields.

​

Security

Who may call is the one security decision that is not identical in the two languages. TypeScript ships it: isInboundCallAllowed(inboundPolicy, allowFrom, from) admits everyone under "open", checks the list under "allowlist" and "pairing", and refuses everything else, including "disabled" and an unset policy. "pairing" is enforced as a plain allowlist, because no pairing code is issued for a call: read it as a second name for "allowlist" rather than as a second mechanism. isAllowlistedCaller matches an exact caller id case-insensitively or a digits-only phone number, because a Microsoft Teams caller arrives as a directory object id that phone normalization would reduce to nothing; and describeInboundRejection writes the log line that tells an operator which list to add them to. Python ships no policy module: you write the same check in on_start and call end(reason) to refuse, which is Refusing a call. TypeScript, Python. Two capabilities are off until you turn them on, and that is the security property rather than a default worth changing casually: an unset STANDIN_OUTBOUND_ALLOW refuses every outbound call, and an unset STANDIN_SHOW_ROOTS (a document onto the call tile) or STANDIN_MEDIA_ROOTS (a picture into a chat reply) refuses every local file. The paths reaching those two were chosen by a model that somebody is steering. A URL is the same problem wearing a trusted costume, so the SDK fetches one an agent chose through its own guard rather than with a plain GET: http and https only, no embedded credentials, no host that resolves into private, loopback, link-local or reserved space, the address re-checked at connect time so a name that answers publicly once and privately a moment later is still refused, and at most one redirect hop, re-guarded. 169.254.169.254 is cloud credentials and 127.0.0.1 is whatever else you run, which is why this is in the core rather than in each plugin. The three helpers are top-level exports in TypeScript and live in standin.fetch in Python: Python, TypeScript.

​

Running it

• Capacity and draining. 64 slots by default. A dial that arrives while draining, or with no free slot, gets 503 before any signature is checked, so a flood cannot make the worker spend CPU verifying calls it was never going to take, and a worker winding down does not accept one it will never serve. Refusing beats degrading.
• One live session per call id. A second dial for a call that already has one gets 409, so a retry cannot quietly run two handlers against the same conversation.
• Five bounds on a call nobody closes, four of them on by default: pre-start (10 s), on_start (15 s), caller-audio idle (45 s), the unanswered-call reaper (120 s, closing with no-agent-answered), and an optional per-call ceiling that speaks a goodbye through your on_goodbye before it closes. The reaper is the one the other three cannot catch: an agent dispatch that never lands leaves every other timer satisfied while the caller hears nothing. Answered means audio went out, so send_audio covers every plugin without any code, and mark_answered() is there for an agent that joins and listens first. The table is on Architecture.
• Agent audio is shed, not queued. Past 1 MiB of unflushed outbound buffer the server drops agent audio frames rather than growing a queue, and the outbound timeline still advances, so the caller hears a gap rather than a rewind. Control frames are never shed, because a call that cannot be ended is the failure this exists to prevent. buffered_bytes / bufferedBytes is the number a continuous sender watches.
• Idempotent, shielded teardown that always frees the slot, including on cancellation.
• Environment-only configuration. No config file, nothing to keep in sync, and nothing read at import: a variable is read when the thing that needs it is built, so a worker that never opens a lane never needs that lane’s settings. Every STANDIN_ variable is in one table on the Configuration page, Python and TypeScript.
• Log injection is handled. Ids that reach a log line have control characters replaced and length bounded, so an attacker-influenced value cannot forge log lines.

​

What is not in the SDK

Being clear about this saves you a wasted afternoon. The avatar row is the one people get wrong in both directions, so to say it plainly: express, send_speech_marks, display_image and send_tile_frame go out of your worker, the caller’s frames come back in, and the SDK gives you an API for both directions. It is only the drawing that is not yours. Each plugin states its own scope on its own page, and Dialogue modes compares what the shipped plugins drive.

​

Parity, and where it stops

protocol/schema.yaml generates both SDKs’ protocol modules, a drift check fails the build if a generated file is edited by hand, and shared conformance vectors assert that Python and TypeScript produce identical payloads. For the call wire protocol and the signatures, “the two languages behave the same” is a test result rather than an intention. Everything above the wire is mirrored by convention instead, and convention has gaps. They are small, and they are worth knowing before you assume a missing symbol is a typo: Each SDK page names the difference for its own language where there is one, so read the page for the language you are writing rather than translating the other.