authenticates, normalizes, signs"] G -->|"pushed down the worker's outbound socket"| C["ChatChannel"] C -->|"your async handler returns reply text"| G G --> T2["StandIn sends it to Microsoft Teams"] ``` Only on the StandIn AI Teammate app, and that needs no flag: the socket authenticates with your connection secret, so if it opens at all you are on the StandIn AI Teammate app. A meeting recap that only posts uses `listen_only=True` so it never answers. ## Meeting recap `listen_only=True` is how a recap lane posts without answering. The host must supply that outbound chat lane (`ChatChannel.start()`) and a summarization callback (`post_minutes` or the plugin consult). Hang-up does not await the post. Meeting recaps are best-effort. They may be lost if the worker exits during processing. ## Minimal use ```python theme={null} import asyncio from standin import ChatChannel, InboundMessage async def on_message(msg: InboundMessage) -> str: return f"You said: {msg.text}" async def main() -> None: chat = ChatChannel(respond=on_message) await chat.start() try: await asyncio.Event().wait() finally: await chat.aclose() asyncio.run(main()) ``` `respond` is an async callable taking an `InboundMessage` and returning the reply text. Returning an empty string makes the channel say so rather than leaving the user watching a typing indicator forever. ## Constructor ```python theme={null} ChatChannel(*, respond, secret=None, url=None, chats=None, listen_only=False) ``` | Argument | Default | Meaning | | ------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `respond` | *(required)* | `async (InboundMessage) -> str`. | | `secret` | `STANDIN_CHAT_SECRET`, else `STANDIN_SECRET` | The key this lane signs with. No secret at all raises `StandInError` naming both variables. | | `url` | `STANDIN_CHAT_URL`, else `wss://teams.standin.komaa.com/api/chat/channel` | The chat channel to dial. | | `chats` | `None` | A [`PersonalChats`](#remembering-who-messaged-you) to feed. Every personal message that arrives is remembered in it, which is what later lets a call post back to the caller who sent one. | | `listen_only` | `False` | Take messages without answering them. `chats` is still fed, `respond` is never called, and nothing is sent back, not even the typing indicator. For a lane that only posts, such as a meeting recap, when something else already answers this connection's chat. | `await chat.start()` dials StandIn and begins taking messages. `await chat.aclose()` cancels the in-flight turns and closes the socket. ### Which secret A StandIn AI Teammate app connection may issue a **separate key for chat**. When it does, set `STANDIN_CHAT_SECRET` and it wins; with one key for both lanes, set only `STANDIN_SECRET` and the chat lane falls back to it. A key scoped to one lane cannot be used on the other, which is the point of issuing two.
joins the call, owns the Microsoft side"] S -->|"one HMAC-authenticated WebSocket per call"| P["standin-sdk[hermes-agent]
answers the dial, runs the realtime session"] P --> R["Realtime model
the conversation: hears, answers, interrupts"] P --> H["Hermes agent
the work: lookups, files, web, your skills"] ``` The model is good at conversation, Hermes is good at work. That split is the whole design. ## Install Into the Python environment that runs Hermes: ```bash theme={null} pip install "standin-sdk[hermes-agent]" ``` Hermes loads the plugin **in-process**, through the `hermes_agent.plugins` entry point. There is no HTTP hop to Hermes, no session header, and no second service to run. ```toml theme={null} [project.entry-points."hermes_agent.plugins"] msteams_bridge = "standin.plugins.hermes_agent" ``` The entry point key is `msteams_bridge`, the name shown in `hermes plugins list` and used in `plugins.enabled`. The Hermes host is deliberately **not** a dependency of the package. Hermes is not on PyPI: it is the application the plugin is installed into. Everything that does not need the host works without it, and everything that does raises `HermesUnavailable` with a sentence saying so. ## Setup ```yaml theme={null} #
| Permission | Purpose |
|---|---|
| `Calls.JoinGroupCall.All` | answer / join Microsoft Teams calls and meetings |
| `Calls.JoinGroupCallAsGuest.All` | join meetings as a guest participant |
| `Calls.AccessMedia.All` | access real-time Microsoft Teams call audio/video media |
| `Calls.Initiate.All` | outbound 1:1 "call me back" (skip if unused) |
| `Calls.InitiateGroupCall.All` | outbound group "call me back" (skip if unused) |
| `Chat.Read.All` | resolve chat / thread ids and read message context |
| `ChatMessage.Read.Chat` | read messages in chats the bot is installed in |
| `Sites.ReadWrite.All` | upload files / minutes to SharePoint (OneDrive) for chat attachments (optional; not needed on Hermes - its minutes file card rides the Bot Framework attachment contract with bot credentials only) |
`. Using **Hermes**, an unknown sender gets a
**pairing-code reply** instead of an answer - approve it with `hermes pairing approve teams `
(with `TEAMS_ALLOWED_USERS` set, unlisted senders are dropped silently instead). Details:
[the bot ignores Microsoft Teams chat](/troubleshooting#the-bot-answers-calls-but-ignores-microsoft-teams-chat-dm-pairing).
```
**Hermes: gated too, but visible by default.** With no allowlist, an unknown sender gets a pairing
code back instead of an answer. Approve it once:
```bash theme={null}
hermes pairing approve teams
```
Hermes goes silent only when `TEAMS_ALLOWED_USERS` (or the global `GATEWAY_ALLOWED_USERS`) is set:
unlisted senders are then dropped with an unauthorized-user warning and no pairing code is offered.
Hermes rate-limits pairing replies to one per sender per 10 minutes, so a rapid second test message
getting no answer is expected. Wait, or approve the code you already have.
## A picture is refused on its way into a chat
**Symptom:** the agent tries to attach an image and gets an error back instead, so the reply arrives
without it, or not at all.
Every outbound picture is checked before it is sent, and there are three messages, each meaning
something different:
| What you get | Why |
| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cannot be sent inline; it must be one of image/png, image/jpeg, image/gif, image/webp` | Only those four raster types may be rendered inline. `image/jpg` is normalised to `image/jpeg` first, so that spelling is never the cause. `image/svg+xml` is deliberately absent and is not a gap to be filled later: SVG is scriptable XML, which is precisely what an image must not be. |
| `that image is bytes, over the 1048576 limit` | Decoded bytes are capped at `OUTBOUND_IMAGE_MAX_BYTES`, 1 MiB. Re-encode or downscale rather than raising it. |
| `those bytes are not image/png: they look like image/gif` | The declared type is a claim made by whatever produced the bytes; the signature is what they are. Without this check an HTML document or an SVG labelled `image/png` gets posted into somebody's chat under your bot's name. |
A local file has one gate before all three. Reading one is off until you name a directory, and until
you do, `sending a local file is off until a directory is named in STANDIN_MEDIA_ROOTS` is the
answer. A path outside those roots is refused with `that file is outside the directories this worker
may read`, with symlinks resolved first, because the path reaching that call was chosen by a model
somebody else is steering.
Sending a picture ([Python](/python-sdk/sending-pictures),
[TypeScript](/typescript-sdk/sending-pictures)) covers the checks and the filename rules. For a
picture arriving the other way, see chat attachments
([Python](/python-sdk/chat-attachments), [TypeScript](/typescript-sdk/chat-attachments)).
## An outbound call is refused before it rings
**Symptom:** the agent says it cannot call somebody, and no phone rings anywhere.
That is the outbound policy, and it is working. Everything that can be refused is refused **before**
the call is placed, so a refusal never leaves somebody's phone ringing for a message that was
rejected. Each refusal is one sentence, safe to read out loud:
| What comes back | Why |
| ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `outbound calling is off: set STANDIN_OUTBOUND_ALLOW to the directory ids this agent may ring` | The allowlist is empty, which is the default. Outbound calling stays off until you list somebody. |
| `that person is not on this agent's outbound allowlist` | The target is not on the list. Matching is case-folded on both sides, so a case mismatch is not what you are looking at. |
| `an outbound call needs the person's directory id` | No target at all reached the policy. |
| `this agent has already placed 6 calls in the last hour` | `STANDIN_OUTBOUND_MAX_PER_HOUR`, default `6`, counted over a rolling hour across all targets. `0` means no cap, which is a choice rather than a default. |
**Fix:** put the directory id in `STANDIN_OUTBOUND_ALLOW`, comma separated, and restart the worker.
The list is read when the policy is built.
This list is separate from any inbound allowlist, and stricter, because the two answer different
questions. Inbound asks whether a person may talk to the agent, and that person chose to dial.
Outbound asks whether the agent may ring a person, and the agent was talked into it by whoever is on
the call. An agent with an outbound tool and no allowlist is an agent that can be talked into
cold-calling your directory. Reaching people ([Python](/python-sdk/reaching-people),
[TypeScript](/typescript-sdk/reaching-people)) has the rest of the lane.
## Free identity: bot leaves or will not join (daily budget)
A **free identity** is metered: **5 minutes of call time per day, per identity**. When the budget runs
out the bot leaves and returns the next day.
The cutoff is **abrupt**: at the 5-minute mark the bot leaves immediately, even mid-sentence, with
no warning. A bot that "hung up out of nowhere" at around five minutes is the budget, not a fault.
"The next day" flips at **midnight UTC**, not your local midnight, which is why a bot can work again
"early in the morning" or "only after lunch" without anything having changed.
## A package name will not resolve
There are two package names, one per language, and everything ships inside them:
```bash theme={null}
pip install standin-sdk # Python, imported as standin
npm install @komaa/standin-sdk # TypeScript
```
Framework support is an extra or a subpath of those, never a separate package:
`standin-sdk[livekit]`, `standin-sdk[hermes-agent]`, `@komaa/standin-sdk/openclaw`.
Two Python extras gate a shipped capability rather than a framework, and a base install hits a
missing dependency the first time it uses one: `standin-sdk[tile]` encodes frames for the avatar
tile, and `standin-sdk[render]` draws a document onto that tile. Office formats additionally need
LibreOffice on `PATH`, which is not something pip can fetch. `standin-sdk[all]` pulls all of them in
at once.
Still stuck? Re-read the [quickstart](/quickstart) end to end. Most remaining issues are a variable
that never reached the process, or a Graph permission that was never admin-consented.
## Contact support
Include the `callId` from your worker's logs, which SDK and plugin you are running, and roughly when
the call happened. We do not keep transcripts, so that context is what we work from.
**Support: support \[at] komaa.com**
StandIn is operated by Komaa DigiTech. For privacy requests and data-subject rights, see the
[Privacy Statement](https://standin.komaa.com/privacy).
# Audio
Source: https://docs.komaa.com/typescript-sdk/audio
Sample rates, the 640-byte frame, resamplePcm16 and FrameAligner in the StandIn TypeScript SDK.
The StandIn wire is PCM 16 kHz, 16-bit, mono, little-endian in both directions. Almost nothing else
is. The realtime speech-to-speech models speak 24 kHz, most TTS vendors emit 22.05 or 24 kHz, and none
of them chunk on the wire's frame boundary.
So every plugin that is not a pure passthrough ends up writing the same two things: a resampler,
because the rates differ, and a frame aligner, because a resampled buffer does not divide evenly into
the wire's 640-byte frame. They ship in the SDK because they are properties of the **wire**, not of
any framework, for the same reason the sequence number lives in `CallServer`. A plugin that has to
reimplement them is a plugin the SDK failed.
## Constants
```ts theme={null}
import {
SAMPLE_RATE_HZ, // 16000, the wire
REALTIME_SAMPLE_RATE_HZ, // 24000, what realtime models speak
FRAME_MS, // 20
FRAME_BYTES, // 640 = 16000 * 2 * 20 / 1000
BYTES_PER_SAMPLE, // 2
NUM_CHANNELS, // 1
} from "@komaa/standin-sdk";
```
`REALTIME_SAMPLE_RATE_HZ` is named in the SDK rather than in a plugin because the **ratio** is what
forces the residual buffer below, and that is an audio concern rather than a provider one.
## `frameDurationMs(pcm)`
Duration of a PCM16 mono buffer in milliseconds. Use this for a playout clock rather than counting
frames: outbound chunk lengths are not fixed, so a frame count drifts against real time.
```ts theme={null}
import { frameDurationMs } from "@komaa/standin-sdk";
playbackEndAt = Math.max(playbackEndAt, Date.now()) + frameDurationMs(pcm16k);
```
Knowing when your own audio stops playing is what lets an echo guard tell the caller's voice apart
from your agent's voice coming back up a speakerphone. That clock is the whole basis of the guard, and
[Realtime providers](/typescript-sdk/realtime-providers#a-playout-clock-not-a-send-clock) is where it
is spelled out.
## `pcm16Rms(pcm)`
How loud one frame is, as root mean square normalised to 0.0 to 1.0.
```ts theme={null}
import { pcm16Rms } from "@komaa/standin-sdk";
const loud = pcm16Rms(pcm) >= 0.02;
```
It lives here because three separate things want it and every plugin that wanted it had been writing
it again: the [utterance segmenter](/typescript-sdk/voice#one-utterance-per-phrase) opens its gate on
it, the [echo guard](/typescript-sdk/realtime-providers#the-agent-answering-itself) decides
echo from barge-in with it, and a barge-in check of your own can use it directly. It is
dependency-free and loop-simple for the same reason the resampler is: it runs on every inbound frame
of every call.
An odd trailing byte is dropped rather than throwing, and an empty buffer is `0`.
## Agent audio can be dropped
Everything above gets whole frames onto the wire. The wire can still refuse them.
When the socket has more than 1 MB (`MAX_AUDIO_BUFFER_BYTES`) unflushed, `sendAudio` advances the
timeline and returns **without sending**, logging at most once every five seconds. The caller hears a
gap rather than the call wedging, and the timeline still advances because a dropped frame is a gap in
what they hear rather than a rewind: stalling the clock would make every later frame claim a time
that has already passed. Control frames are never shed, because a call that cannot be ended is the
failure this exists to prevent.
Watch `session.bufferedBytes` before pushing a continuous stream, and treat a zero as no evidence of
backpressure rather than proof of an idle socket, because it reads zero when the transport cannot
report it. See [Call server](/typescript-sdk/call-server#limits).
The avatar tile has its own, tighter budget on the same socket, because a caller forgives a dropped
frame far more readily than a break in the voice. That one is `TileStream`'s, on
[The avatar](/typescript-sdk/avatar).
## Inbound audio
`onCallerAudio` hands you the caller's PCM already decoded and already validated: base64 that is not
canonical is rejected, and so is a payload that is not a whole number of PCM16 samples. A malformed
frame is dropped by the server with a log line and never reaches your handler.
What you do **not** get is a guarantee of 640 bytes. 20 ms is the nominal frame, and it is what
arrives on a normal call, but the only invariant the decoder enforces is a non-empty, even number of
bytes: at least one whole PCM16 sample, and no half sample at the end. Measure with
`frameDurationMs(pcm)` rather than counting frames, and never index into a caller frame at a fixed
offset.
## `resamplePcm16(pcm, srcHz, dstHz)`
Linear-interpolation resample of PCM16 mono. Returns the input untouched when the rates match or the
buffer is empty.
```ts theme={null}
import { REALTIME_SAMPLE_RATE_HZ, SAMPLE_RATE_HZ, resamplePcm16 } from "@komaa/standin-sdk";
const toModel = resamplePcm16(pcm16k, SAMPLE_RATE_HZ, REALTIME_SAMPLE_RATE_HZ);
const toCaller = resamplePcm16(pcm24k, REALTIME_SAMPLE_RATE_HZ, SAMPLE_RATE_HZ);
```
An odd trailing byte is dropped rather than throwing. A truncated frame is a glitch; an exception in
the audio path is a dropped call.
The rounding is deliberately fussy, so that the Python and TypeScript SDKs produce byte-identical
output: `Math.round` for the output length, and `Math.trunc` rather than `Math.floor` when
interpolating, because Python's `int()` truncates toward zero and the difference is audible between a
negative and a less-negative sample. The shared conformance vectors assert the two agree.
## `FrameAligner`
Chops arbitrary-length PCM buffers into whole wire frames, carrying the remainder.
Resampled 24 kHz deltas do not divide evenly into the 640-byte frame. Without a residual, the leftover
bytes are dropped and every turn loses a few milliseconds at the seams. Over a call that is audible as
clipped word endings.
```ts theme={null}
import { FrameAligner } from "@komaa/standin-sdk";
const aligner = new FrameAligner();
// arbitrary lengths in, whole 640-byte frames out
for (const chunk of providerAudio) {
for (const frame of aligner.push(chunk)) {
await session.sendAudio(frame);
}
}
// end of turn: the residual is real audio, not a fragment to discard
const tail = aligner.flush();
if (tail) await session.sendAudio(tail);
```
| Member | Purpose |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `new FrameAligner(frameBytes?)` | Defaults to `FRAME_BYTES`, 640. Pass another size only when you are aligning to something that is not the StandIn wire. |
| `push(pcm)` | Add a buffer, return whatever whole frames are now available. |
| `flush()` | Zero-pad and return the residual at end of turn, or `undefined` when empty. |
| `reset()` | Drop the residual without emitting it. |
| `pending` | Bytes currently held back. |
`flush()` pads rather than drops: the tail of the last word matters more than a few milliseconds of
silence. `reset()` is the barge-in case, where the held-back bytes belong to a turn the caller just
interrupted and must never be played.
### The remainder problem, in numbers
A 24 kHz provider does not hand you round numbers, and whether a chunk divides cleanly is luck:
| From the provider, at 24 kHz | Resampled to 16 kHz | Whole 640-byte frames | Left over | Lost per seam without a residual |
| ---------------------------- | ------------------- | --------------------- | --------- | -------------------------------- |
| 9600 bytes | 6400 bytes | 10 | 0 bytes | nothing |
| 7000 bytes | 4666 bytes | 7 | 186 bytes | 5.8 ms |
Five milliseconds sounds like nothing. It is not one seam: it is one per chunk, for the whole call,
and it always lands at the end of the audio rather than in the middle, which is why it is heard as
clipped word endings rather than as a glitch. The aligner carries those 186 bytes into the next push
and they are never lost at all.
### Resetting on a barge-in
Three steps, and the order is the whole point:
```ts theme={null}
await session.cancelPlayback(); // 1. flush what the service still holds
provider.cancelResponse(); // 2. stop the model generating more
aligner.reset(); // 3. drop the residual of the interrupted turn
```
Do 2 before 1 and the model stops while the bot keeps talking for the length of the buffer. Skip 3
and the first frame of the caller's **next** answer is prefixed with the last few milliseconds of the
turn they interrupted, which sounds like the agent stammering.
## A complete realtime leg
Both directions of a 24 kHz provider, with the interrupt handled properly:
```ts theme={null}
import {
FrameAligner,
REALTIME_SAMPLE_RATE_HZ,
SAMPLE_RATE_HZ,
frameDurationMs,
resamplePcm16,
type CallSession,
} from "@komaa/standin-sdk";
class RealtimeHandler {
#call: CallSession | undefined;
#aligner = new FrameAligner();
#playbackEndAt = 0;
async onStart(session: CallSession) {
this.#call = session;
// model -> caller: 24 kHz down to the wire's 16 kHz, then whole frames only
provider.on("audio", async (pcm24k: Buffer) => {
const pcm16k = resamplePcm16(pcm24k, REALTIME_SAMPLE_RATE_HZ, SAMPLE_RATE_HZ);
this.#playbackEndAt = Math.max(this.#playbackEndAt, Date.now()) + frameDurationMs(pcm16k);
for (const frame of this.#aligner.push(pcm16k)) {
await session.sendAudio(frame);
}
});
// end of turn: send the padded residual so the last word is not clipped
provider.on("turnEnd", async () => {
const tail = this.#aligner.flush();
if (tail) await session.sendAudio(tail);
});
// barge-in: flush the service FIRST, then stop the model, then drop the residual
provider.on("callerStartedSpeaking", async () => {
await session.cancelPlayback();
provider.cancelResponse();
this.#aligner.reset();
});
}
// caller -> model: the wire's 16 kHz up to what the model speaks
async onCallerAudio(pcm: Buffer) {
provider.sendAudio(resamplePcm16(pcm, SAMPLE_RATE_HZ, REALTIME_SAMPLE_RATE_HZ));
}
}
```
## Three traps worth naming
1. **Do not track timestamps yourself.** `sendAudio` owns the sequence number and the outbound
timeline. Advancing your own clock and sending it is how timelines end up jumping backwards after
an audio source is swapped.
2. **Do not drop the residual at end of turn.** That is the clipped-word-endings bug, and it is subtle
enough to survive a demo and fail in production.
3. **Do not reuse one `FrameAligner` across calls.** It carries per-turn state. One per call, and
`reset()` on every interrupt.
## The same bytes in both SDKs
`standin/audio.py` and `src/audio.ts` are the same constants, the same linear resample and the same
carry-the-remainder aligner, and the shared conformance vectors assert that both produce identical
output for the same input. The rounding above is a property of the pair rather than of this language:
it is written the way it is so that Python and TypeScript agree, not because either idiom is nicer.
That matters in practice because a Python worker and a TypeScript worker can share one StandIn
identity, and a call recorded from one has to be indistinguishable from a call recorded from the
other.
## Next
The segmenter, WAV handling and paced playback, for an agent that is not a realtime model.
The playout clock, and why the agent answers itself on a speakerphone.
`sendAudio`, `cancelPlayback` and the session that owns the timeline.
Prove audio made the round trip without placing a real call.
# The avatar
Source: https://docs.komaa.com/typescript-sdk/avatar
Expressions, the viseme timeline that drives lip-sync, and putting your own video on the bot's tile, in the StandIn TypeScript SDK.
StandIn renders the avatar tile. Your worker does not draw it, stream it, or know how it is made. It sends hints, and the service does the rest.
## Two hints and one stream
There are two hints. One names the emotion the face should wear, the other carries the viseme timeline for an utterance, which is what makes the mouth match the words:
```ts theme={null}
await session.express("happy");
await session.sendSpeechMarks([
{ tMs: 0, visemeId: 12 },
{ tMs: 200, visemeId: 4 },
{ tMs: 400, visemeId: 14 },
{ tMs: 800, visemeId: 8 },
]);
```
Everything here is **additive and best-effort**. An unknown emotion renders as neutral, a service that does not implement one of these ignores it, and nothing on this page changes a single sample of the audio the caller hears. That is the whole reason it is safe to send: the worst case is a neutral face over an unchanged voice, which is exactly what the caller would have got had you sent nothing at all.
Neither hint needs a provider that offers it. The emotion can be read straight out of the reply text with no extra model call, and the timeline can be built for a provider that returns no timings whatsoever. Those two are the bulk of this page.
Then there is the stream, for the minority of workers that already produce video of their own: `TileStream` puts that on the tile in place of the rendered avatar. It is the one thing here that is continuous rather than a hint, and the last section is mostly about not letting it hurt the voice.
## The emotion the face wears
```ts theme={null}
import { EMOTIONS, expression } from "@komaa/standin-sdk";
EMOTIONS; // ["neutral", "happy", "sad", "surprised", "thinking"]
```
`EMOTIONS` is an **open set**, not a validation list, and the `Emotion` type says so in the types: it is one of those five, or any other string. Sending something outside the set is allowed and renders as neutral, which is what lets a newer sender and an older service keep interoperating instead of one of them refusing the other's vocabulary.
`session.express(emotion)` is what a handler calls. `expression(emotion)` is the message builder underneath it, exported for a plugin that assembles its own wire messages.
```ts theme={null}
import { MAX_EMOTION_CHARS } from "@komaa/standin-sdk";
MAX_EMOTION_CHARS; // 40
```
`expression()` throws on an empty emotion or one longer than `MAX_EMOTION_CHARS`. Everything in between is the service's to interpret.
The cap is applied **where the message is built**, not at the edge of some plugin, and that placement is the point. An emotion reaches the avatar tile, and the string usually came from a model that whoever is on the call is steering: a tool argument, a token off a transcript. Bounding it at the builder means `session.express` is bounded, every plugin's own `express` tool is bounded, and no future one can forget to do it.
## Emotion with no extra model call
An extra inference per sentence is not something a live call can pay for. `inferEmotion` reads the emotion out of the reply text instead, with no model call and no added latency:
```ts theme={null}
import { inferEmotion } from "@komaa/standin-sdk";
inferEmotion("Congratulations, that is excellent"); // "happy"
inferEmotion("Sorry, that is great news"); // "sad"
inferEmotion("They did what??"); // "surprised"
inferEmotion("the meeting is at four"); // "neutral"
```
It returns `surprised`, `sad`, `happy` or `neutral`, typed as `InferredEmotion` and ready to hand to `expression()`. First match wins in that order rather than a score being totalled: a startled "wow" must not be averaged away by a polite "thanks", and an apology must not be masked by an incidental "nice".
`ExpressionCue` decides **when** one is worth sending, which is the harder half. Ask it on every assistant transcript, partial and final alike, and send whatever comes back:
```ts theme={null}
import { ExpressionCue } from "@komaa/standin-sdk";
const cues = new ExpressionCue();
async function onTranscript(text: string, isFinal: boolean): Promise {
const emotion = cues.cue(text);
if (emotion !== null) await session.express(emotion);
}
```
Two things are happening there. It **re-reads the emotion on every chunk**, so the face self-corrects as the rest of a sentence arrives: "let me check that" cues neutral, and "let me check that, no way!!" cues surprised a moment later. And it **returns `null` when nothing changed**, which is what stops a partial-per-word stream from sending dozens of identical messages for one sentence. Waiting for the final transcript instead would land the cue as the sentence ends, with the face stale for the entire time a happy or apologetic reply was being spoken.
One instance per call. What it remembers is the last emotion that call was told to wear, readable as `cues.lastSent`.
While a tool keeps the caller waiting, hold a thinking face:
```ts theme={null}
async function runTool(name: string): Promise {
const waiting = cues.thinking(true);
if (waiting !== null) await session.express(waiting);
try {
return await tools.run(name);
} finally {
const done = cues.thinking(false);
if (done !== null) await session.express(done);
}
}
```
While thinking is on, `cue()` returns `null` for everything: without that suppression a transcript chunk arriving mid-tool makes the avatar look finished while it is still working. Only a transition acts, so setting the same state twice sends nothing and a retry around the tool is harmless. Leaving sends `neutral` when thinking is still the last thing the call was told to wear, because the model may say nothing at all after a tool result, and with no transcript to re-infer from the face would stick mid-thought for the rest of the call. Call it from a `finally` so a tool that threw still leaves the face behind it.
The lexicon is **English only**, and that is a decision rather than an oversight. An Arabic reply, or a reply in any other language, infers `neutral`. Neutral is always a safe face, and a silent guess at the language would put a confident wrong one on the tile.
Note the asymmetry before you go looking for a bug that is not there: the viseme table in the next sections **is** bilingual. The mouth moves on an Arabic reply. The face stays neutral.
## The viseme timeline
```ts theme={null}
interface SpeechMark {
readonly tMs: number;
readonly visemeId: number;
}
```
One mark is milliseconds from the start of the utterance, and which mouth shape to hold. `tMs` counts from the start of **that turn's audio**, never from the moment the message reaches the service.
This is the one place the two SDKs differ in shape. A mark is an **object** in TypeScript and a **tuple** in Python, where `SpeechMark` is declared as `tuple[int, int]` and the same timeline is written `[(0, 12), (200, 4)]`. Everything else here matches: the same ids, the same ordering rules, the same wire payload. Porting a plugin across rewrites the mark literals and nothing else.
`session.sendSpeechMarks(marks)` sends the timeline for one utterance. `speechMarks(marks)` is the builder underneath, again exported for a plugin assembling its own wire messages. Both take any `Iterable`, so an array or a generator are equally welcome.
```ts theme={null}
import { MAX_VISEME_ID } from "@komaa/standin-sdk";
MAX_VISEME_ID; // 21
```
Visemes use the Azure Speech numbering, 0 to 21, which is what the avatar expects. The builder **sorts the marks ascending and drops anything outside that range**, along with any mark timed before zero, rather than passing it on. The avatar reads the timeline in order, so a mark out of sequence or a shape it cannot render does not cost one frame: it desynchronises the mouth for the rest of the utterance.
## Measured beats still, guessed loses to still
This is the rule that decides whether to send a timeline at all, and it is worth getting right before writing any of the code below.
Send a timeline whenever it is spread over a duration you **measured**: the audio your worker actually sent for that turn, or real per-character timings from the speech provider. That is better than a still mouth, and on a realtime path it is the difference between an avatar that talks and one that never opens its mouth at all.
Do not send a timeline spread over a duration **guessed** from text length or a words-per-minute rate. That is worse than sending nothing. The error in a guess compounds sentence after sentence, so the mouth drifts further from the voice the longer the call runs, and a mouth moving against the voice reads as broken in a way a still mouth never does.
The distinction is the **duration**, not the phonemes. The mouth shapes are approximate either way.
## A timeline from the text and the audio you sent
Every realtime speech-to-speech model streams voice and hands back no phoneme timings, so there is nothing for `sendSpeechMarks` to carry and the mouth never moves on the default path. `TurnLipSync` closes that gap the one honest way open to a worker: it counts the audio the turn actually sent, byte by byte, and spreads the text over that measured length.
```ts theme={null}
import { TurnLipSync } from "@komaa/standin-sdk";
const lipsync = new TurnLipSync();
// the audio sink
for await (const chunk of modelAudio) {
await session.sendAudio(chunk);
lipsync.audioSent(chunk);
}
async function onTranscript(text: string, isFinal: boolean): Promise {
if (isFinal) {
// once per turn, never on a partial
const marks = lipsync.finish(text);
if (marks.length > 0) await session.sendSpeechMarks(marks);
}
}
async function onBargeIn(): Promise {
lipsync.cancel();
}
```
`audioSent(pcm)` takes a PCM16 mono buffer. The constructor's `sampleRateHz` option defaults to the wire's own 16 kHz, which is what a plugin sending frames to the call is holding; pass the real rate when your sink counts at another one, as in `new TurnLipSync({ sampleRateHz: 24_000 })`. For a sink that hands over encoded audio, `audioSentMs(ms)` adds a duration directly, and it ignores a chunk that measures as zero, as negative, or as no number at all rather than taking the turn's count with it. Each chunk is rounded as it is added rather than kept as a running float, so both SDKs accumulate the same integer for the same stream of chunks.
`finish(text)` returns the timeline and resets the counter, so the next turn starts from zero. It returns an empty array when no audio was sent or the text carries no mouth shape, and you send nothing in that case. Emit once per turn, on the final transcript: a partial would send an ever-lengthening timeline several times over and the avatar would restart the mouth mid-sentence.
`cancel()` is not optional. On a barge-in the service drops audio the caller never heard, and a counter that keeps those milliseconds spreads the next turn's text over its own audio **plus the discarded audio**. The mouth runs long for the whole of that turn and every turn after it. Call it wherever you call [`cancelPlayback()`](/typescript-sdk/call-handler#cancelplayback-is-the-barge-in).
Underneath it, and usable on their own:
| Name | What it is |
| ----------------------------------- | ----------------------------------------------------------------------- |
| `estimateVisemes(text, durationMs)` | The spread itself. Returns `SpeechMark[]`, strictly increasing in time. |
| `visemeForChar(ch)` | The mouth shape a character wears, or `undefined`. |
| `CHAR_VISEMES` | The character to shape map, frozen. |
| `SILENCE_VISEME` | `0`, a closed mouth, and what the space between two words becomes. |
`estimateVisemes` returns an empty array rather than throwing on anything it cannot use: no text, a duration that is zero, negative, infinite or not a number, or text with no mouth shape in it at all. `"3.5%"` is the last case, and an empty answer is the right one: mapping digits and punctuation to silence instead would punch a hole of closed-mouth frames into the middle of a spoken number. Only a space becomes silence. A run of one shape is one mark, timed at the first character of the run, because "mmm" is one mouth position and a mark per character would multiply the payload for an identical rendering.
Times come back strictly increasing, and that is not cosmetic either. Spread a long sentence over a very short buffer and the step falls below half a millisecond, so neighbouring marks round onto the same one. The later shape wins, because a shape held for zero milliseconds is not renderable, and because `speechMarks` re-sorts by `(tMs, visemeId)` and would otherwise hand the avatar a different winner than the one the walk ended on.
### What the table covers
`CHAR_VISEMES` covers **all 26 Latin letters and all 28 Arabic letters** in one map, with the eight Arabic variant forms and the three short vowels alongside them:
| Viseme | Latin | Arabic |
| ------ | --------------------------- | ------------------------ |
| 0 | the space between two words | the same |
| 2 | `a` | `ا أ إ آ ى ة`, and fatha |
| 4 | `e` | |
| 6 | `i y` | `ي ئ`, and kasra |
| 7 | `u w` | `و`, and damma |
| 8 | `o` | |
| 12 | `h` | `ه ح ع ء ؤ` |
| 13 | `r` | `ر` |
| 14 | `l` | `ل` |
| 15 | `s z x` | `س ص ز` |
| 16 | `j` | `ش ج` |
| 18 | `f v` | `ف` |
| 19 | `t d n` | `ت د ن ط ض ث ذ ظ` |
| 20 | `k g c q` | `ك ق غ خ` |
| 21 | `m b p` | `م ب` |
Full coverage is not a nice-to-have. One common letter left out thins the timeline unevenly and the mouth stalls on that syllable, and without the Arabic rows at all an Arabic reply produces no tokens, carries no timeline, and the mouth simply does not move for half the people who will be on these calls. The stretch and doubling marks, sukun, shadda, tanween, the tatweel and every presentation form are deliberately absent, because they carry no mouth shape of their own and mapping them would insert mouth changes nobody spoke.
`visemeForChar` lowercases and looks the character up as written, with **no Unicode normalization at all**. Folding presentation forms onto their base letter would change which characters map and the two SDKs would then disagree about the same string. A test reads the table out of the Python source and asserts the two are equal character for character, because a mouth that moves differently in two SDKs is a bug a customer finds by switching language rather than one CI finds by itself.
The map is frozen and a write to it throws. It is also built with no prototype, so `visemeForChar("constructor")` and `visemeForChar("__proto__")` answer `undefined` like any other unmapped input instead of handing back an inherited function where the signature promises a viseme id.
## Real timings when your provider has them
When the speech provider returns per-character timings, use them. They are strictly better than an estimate and they cost nothing, since the synthesis call already produced them:
```ts theme={null}
import { estimateVisemes, visemesFromAlignment } from "@komaa/standin-sdk";
let marks = visemesFromAlignment(
result.characters,
result.characterStartTimesSeconds,
);
if (marks.length === 0) marks = estimateVisemes(text, lipsync.durationMs);
```
Core takes two plain arrays: the characters as the provider spoke them, and when each one starts in seconds. Normalising a vendor's field names is the speech plugin's job, not this function's.
It is deliberately forgiving of the shapes providers actually return. Ragged arrays are walked to the shorter of the two, a timing that is not a finite number costs its own mark and nothing else, and a time before the start of the utterance is clamped to zero. Providers do return mismatched lengths and bare `NaN` values, and throwing there would lose the turn over a cosmetic hint.
Fall back to `estimateVisemes` on an **empty result**, as above, rather than on a missing alignment. A provider that returned timings for punctuation only has an alignment and still needs the estimate.
## When to send the timeline
Where the duration is known before playback, which is a text-to-speech path, send the timeline **ahead of the first audio frame**. The service then has the whole mouth plan before the voice starts.
On a realtime path the duration is only known once the turn ends, so the marks necessarily go out **after the last chunk was handed over**. That is correct while the service still holds that turn's audio buffered for playout, which is the normal case, and it is why the marks are worth sending at all on a path that would otherwise never move the mouth.
## Your own video on the tile
When your agent already produces video of its own (an avatar worker, a rendered face, a camera), that video can go on the tile instead of StandIn's own avatar.
`session.sendTileFrame(jpeg, width, height)` puts one frame there. The SDK owns the sequence number and stamps the frame with the **outbound audio timeline**, `session.mediaTimeMs`, the same clock the voice is stamped on, so the two streams cannot disagree about what time it is. A wall clock keeps ticking through listening silence while the audio clock does not, so a wall-clock stamp would make the two look like they are drifting apart when they are in step.
Sending frames is the easy part. Sending them at a rate that does not hurt the call is what `TileStream` owns:
```ts theme={null}
import { TileStream } from "@komaa/standin-sdk";
const tile = new TileStream(session);
await tile.start();
// ...
tile.offerRgb(rgb, width, height); // as often as you like
// ...
await tile.aclose();
```
| Constructor argument | Default | Meaning |
| -------------------------- | ------------ | ------------------------------------------------------------- |
| `session` | *(required)* | The `CallSession` the frames go to. |
| `options.fps` | `12` | Clamped to 1 through `MAX_TILE_FPS`. |
| `options.encoder` | `undefined` | Yours, or sharp is found for you. |
| `options.maxBufferedBytes` | `327680` | The backpressure budget, in bytes of unflushed outbound data. |
Everything but `session` arrives in one optional `TileStreamOptions`, so `new TileStream(session)` and `new TileStream(session, { fps: 20 })` are both complete calls.
```ts theme={null}
import { MAX_TILE_FPS, TILE_HEIGHT, TILE_WIDTH } from "@komaa/standin-sdk";
MAX_TILE_FPS; // 20
TILE_WIDTH; // 640
TILE_HEIGHT; // 360
```
Frames are encoded to 640 by 360, because shipping an avatar's native resolution only spends bandwidth on pixels the tile will not show. `MAX_TILE_FPS` is a sender-side sanity clamp rather than a protocol limit: a talking head gains nothing above 20, and a higher rate only spends local CPU on encoding and base64.
**Latest wins, and each frame goes at most once.** Frames are offered into a single slot, never a queue. A ticker takes whatever is newest and sends it. So a source producing faster than the wire drops the middle frames rather than falling behind, and a source that stops producing goes quiet rather than repeating one stale frame forever. Silence is how a stream ends.
That is right for video and would be wrong for audio. A dropped frame is a frame the caller never notices missing. A dropped audio buffer is a hole in a sentence, which is why the audio path buffers what it cannot send yet and this path throws it away.
**Video yields to voice.** `TileStream` watches `session.bufferedBytes` and drops a frame rather than adding to it once the budget is exceeded, and it re-checks after an encode resolves, because audio may have filled the socket while the encoder was off the loop. The default budget of 320 KiB is deliberately tighter than the audio buffer cap of `MAX_AUDIO_BUFFER_BYTES`. Both streams share one socket, and a caller forgives a dropped frame far more readily than a break in the voice, so a loaded path goes quiet promptly rather than building up seconds of skew. `framesSent` and `framesDropped` are both readable, and a healthy call has some drops.
`offerRgb` and `offerJpeg` never block and never throw. They are meant to be called from a decode loop that must not be slowed down by the wire. `offerJpeg(jpeg)` skips the encoder entirely, for a source that hands you JPEG already: nothing is re-encoded, no encoder needs to be installed at all, and the width and height default to the tile's own 640 by 360. `aclose()` is safe to call twice and on every teardown path.
### The encoder
Encoding packed RGB to JPEG needs [sharp](https://sharp.pixelplumbing.com/), which is an **optional peer dependency** of the package rather than a dependency:
```bash theme={null}
npm install sharp
```
Pass your own `encoder` to skip it. An `Encoder` is any function taking packed RGB plus a width and a height and returning a promise of JPEG bytes:
```ts theme={null}
type Encoder = (rgb: Buffer, width: number, height: number) => Promise;
```
The promise is a divergence from the Python twin, where an encoder is a plain synchronous callable and the stream hands it to a worker thread. Here the yielding is yours to do: the ticker awaits what you return, so an encoder that spends its time on the loop rather than off it holds up everything else the call is doing, the voice included.
`jpegEncoder()` is what `start()` calls when you pass none. It resolves sharp at runtime and returns an `Encoder`, or `undefined` having logged why. Call it yourself if you would rather branch on the result before building the stream.
Without sharp and without an encoder of your own, the tile relay stays off with one line in the log and **the call is unaffected**. The caller hears your agent and sees StandIn's rendered avatar, which is what they would have seen anyway. sharp is optional precisely because most deployments never put their own video on the tile, and a dependency in every install is the wrong trade for a feature a minority uses. Note that `offerJpeg` still works in that state: it is only `offerRgb` that needs an encoder.
## Next
The other half of the video lane: what the caller shows your agent.
Giving a model the `express` tool, and the rest of the built-in surface.
# Call handler
Source: https://docs.komaa.com/typescript-sdk/call-handler
The seven optional methods a StandIn TypeScript plugin implements, and the CallSession it is handed.
A handler is a plain object with up to seven optional methods. One instance is built per call by your
`handlerFactory`. There is no base class, no registration, and no method you are obliged to write:
the server treats a missing one as a no-op, so a handler that only wants audio implements only
`onCallerAudio`.
```ts theme={null}
import { CallServer, type CallSession } from "@komaa/standin-sdk";
class MyHandler {
#call: CallSession | undefined;
async onStart(session: CallSession) { this.#call = session; }
async onCallerAudio(pcm: Buffer) { await this.#call?.sendAudio(pcm); }
async onContext(text: string) {}
async onGoodbye(text: string) {}
async aclose(reason: string) {}
}
const server = new CallServer({ handlerFactory: () => new MyHandler() });
await server.start();
```
## The seven methods
`onStart`, `onCallerAudio`, `onVideoFrame`, `onSpeakerChange`, `onContext`, `onGoodbye` and `aclose`.
In Python they are the same methods in snake\_case: `on_start`, `on_caller_audio`, `on_video_frame`,
`on_speaker_change`, `on_context`, `on_goodbye`, `aclose`. Porting a handler between the languages
changes the names and nothing else.
One real difference, and it only matters if you are porting: all seven are optional members of one
`CallHandler` interface here, while Python keeps `on_video_frame` and `on_speaker_change` on separate
`VideoHandler` and `SpeakerHandler` protocols. Python's `CallHandler` is `runtime_checkable`, and a
runtime check demands every member, so adding a sixth would have broken `isinstance` for every
handler written before the vision lane existed. Nothing about the wire or the behaviour differs.
### `onStart(session)`
The call is live. Join a room, open a realtime socket, build an agent, whatever your framework needs.
Caller audio does not flow until this resolves, so a slow start delays the caller rather than dropping
frames. It is bounded by `onStartTimeoutMs`, 15 seconds by default, because `onStart` does real
network work and the frame loop is queued behind it. An unbounded one would hold a call slot for the
life of the worker.
Caller audio and context that arrive while your provider is still connecting are **yours to hold**,
not the server's. `StartupBuffer` is the piece that does it, and losing that window is the difference
between an agent that opens by answering the question it was asked and one that opens by asking it
again. See [Realtime providers](/typescript-sdk/realtime-providers#what-arrives-before-your-agent-is-ready).
### `onCallerAudio(pcm)`
One frame of the caller's voice: PCM16, 16 kHz, mono, little-endian, as a `Buffer`.
This runs on the receive path of a live call, so it must not block. The frame is already validated:
a truncated or malformed payload is dropped by the server and never reaches here.
### `onVideoFrame(frame)`
One sampled frame of the caller's camera or screen share.
Frames arrive sparsely and best-effort, and most handlers never implement this: the common shape is
to look only when the model asks, which `session.latestVideoFrame()` already serves without a
callback. Implement it for ambient vision: narrating a slide deck, watching a whiteboard, noticing
that the share stopped. It runs on the receive path too, so a slow model call belongs off the frame
loop exactly as in `onCallerAudio`. See [Vision and the avatar](/typescript-sdk/vision).
### `onSpeakerChange(name)`
A different person started speaking, and only when StandIn sends unmixed audio. Most calls carry
mixed audio and never call it at all.
It is called on CHANGE only, never per frame: the name rides every inbound audio frame, and a model
told forty times a second who is speaking would hear nothing else.
### `onContext(text)`
Non-interrupting context about the call, as a plain sentence ready to put in front of a model:
participant counts and group-call etiquette, DTMF key presses, and recording status changes. For
example, `The caller pressed the "5" key on their keypad.`
Both SDKs publish the same sentences, so an agent written against either reads identical context.
Context is delivered as it arrives, and the server does not queue it on your behalf, because what
"ready" means is your framework's business. If your agent cannot accept context before it is built,
queue it here.
### `onGoodbye(text)`
StandIn is ending the call and wants this line spoken first.
The server has already told StandIn to drop whatever agent audio it had buffered, so the line plays
immediately. Interrupt the current turn and say it. Teardown follows within seconds, and a goodbye
queued behind a long answer is a goodbye the caller never hears.
The same callback carries the line from a `maxCallMs` ceiling, which is why a plugin that honours
`onGoodbye` needs no extra code for the duration limit.
### `aclose(reason)`
Release everything this call holds. Always called exactly once, on every path, before the slot is
freed. It runs before the socket closes, so a handler that wants to say something on the way out
still can.
| Reason | What happened |
| ----------------------- | ---------------------------------------------------------------------------------- |
| `call-ended` | StandIn ended the call normally. |
| `pre-start-timeout` | The socket authenticated and never sent `session.start`. |
| `caller-idle-timeout` | No caller audio for `audioIdleTimeoutMs`. |
| `handler-start-timeout` | `onStart` ran past `onStartTimeoutMs`. |
| `handler-start-failure` | `onStart` threw. |
| `handler-failure` | Some other handler method rejected. |
| `transport-failure` | The frame loop failed. |
| `server-shutdown` | `server.aclose()`. |
| `no-agent-answered` | Nothing sent audio and nothing called `markAnswered()` inside `staleCallReaperMs`. |
| `call-duration-limit` | The call hit `maxCallMs`. |
| `agent-ended-call` | The model called the `end_call` tool. |
| `outbound-no-answer` | An outbound leg rang out. See [Reaching people](/typescript-sdk/reaching-people). |
Anything else is a string you passed to `session.end()`.
`no-agent-answered` is the one a plugin author meets first and has nowhere else to look up: it means
the call connected and authenticated perfectly and no agent ever arrived.
## The session object
`CallSession` is handed to `onStart` and is valid until the call ends. The server owns the socket;
this is your only way to reach it. Seventeen members, and this is all of them.
| Member | Purpose |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `callId` | StandIn's id for this call. Authenticated: it is the value the handshake HMAC signed, not something a caller supplied. |
| `start` | The `session.start` that opened the call: caller identity, direction, thread id, recording status. |
| `await sendAudio(pcm)` | Send the agent's voice. Same PCM16 16 kHz mono format `onCallerAudio` receives. Resolving is not proof it went out: see below. |
| `await cancelPlayback()` | Drop whatever agent audio StandIn still has buffered. |
| `await end(reason)` | Ask for the call to end. Idempotent, and the first reason wins. |
| `recordingActive` | Whether the call is being recorded, right now. |
| `speaker` | Who is speaking, when StandIn sends unmixed audio. `undefined` on the mixed path. |
| `participantCount` | How many people are on the call. Zero until StandIn says. |
| `answered` | Whether anything has actually taken this call yet. |
| `markAnswered()` | Say that an agent has taken it. Stamped once, never re-stamped. |
| `bufferedBytes` | Outbound bytes the socket has not flushed yet. |
| `mediaTimeMs` | The outbound audio timeline, in milliseconds. |
| `latestVideoFrame(source?)` | The most recent frame the caller showed, or `undefined`. |
| `await displayImage(image, options?)` | Draw an image on the bot's video tile for a few seconds. |
| `await sendTileFrame(jpeg, width?, height?)` | Put one frame of continuous video on the tile. |
| `await express(emotion)` | Hint the emotion the avatar's face should wear. |
| `await sendSpeechMarks(marks)` | Send one utterance's viseme timeline, which drives lip-sync. |
Five of those repay a paragraph each.
**`sendAudio` can drop the frame you hand it.** Past 1 MB of unflushed socket buffer it advances the
outbound timeline and returns **without sending**, so the promise resolving is not evidence the audio
reached anyone. The caller hears a gap rather than the loop wedging, and the timeline still advances
because a dropped frame is a gap in what they hear rather than a rewind. Control frames are never
shed. [Audio](/typescript-sdk/audio#agent-audio-can-be-dropped) has the whole rule.
**`markAnswered()` is what keeps a listening plugin alive.** A call counts as answered when the
handler sends audio. A plugin that joins a room and only listens never sends any, so the
[stale-call reaper](/typescript-sdk/call-server#the-watchdogs) ends it after two minutes with
`no-agent-answered`. Call `markAnswered()` when your agent's own audio track appears, not when a
participant connects: monitors, recorders and avatar workers all connect, and none of them is an
agent answering.
**`mediaTimeMs` is the clock, not the wall clock.** It is the same timeline this call's audio frames
are stamped with, and it is what a video frame must be stamped with too. A wall clock keeps ticking
through listening silence while this one does not, so stamping video from a wall clock makes audio
and video drift apart on paper even when they are in step.
**`bufferedBytes` is evidence, not proof.** It is the number a continuous sender watches before
deciding to drop a frame rather than queue it. It reads zero when the transport cannot report it, so
treat a zero as "no evidence of backpressure" rather than as an idle socket.
**`recordingActive` is one flag the server keeps current**, so no plugin re-derives it from the
context sentence it happened to see. A reported `recording.status` **wins over the `session.start`
snapshot**, whichever arrives first: the snapshot omits the field when the state was unknown at answer
time, and an omitted field is not "not recording".
Gate on `recordingActive` before anything that STORES what the caller said or showed with a third
party. A recorded call is one the caller was told is being kept; an unrecorded one is not.
The last four rows are the vision and avatar lane. StandIn draws the tile; these send it what to
draw and what face to wear. [Vision and the avatar](/typescript-sdk/vision) covers what the caller
shows you, and [The avatar](/typescript-sdk/avatar) covers expressions, the viseme timeline and
putting your own video on the tile.
You never track a sequence number or a timestamp. The server owns both, which is why a handler that
swaps or re-publishes its audio source cannot make the outbound timeline jump backwards while the
sequence number keeps climbing.
## `cancelPlayback()` is the barge-in
`cancelPlayback()` is the **only** wire-level lever that un-sends audio already handed to the service.
Without it, a barge-in stops the model but the bot keeps talking for the length of the buffered PCM,
which is the "it kept talking over me" complaint in its entirety.
Call it the moment your provider reports the caller started speaking, and call it **before** you
cancel the response upstream. Cancelling upstream first stops new audio being generated but leaves
everything already queued at the service to play out.
```ts theme={null}
async onProviderSaysCallerStartedSpeaking() {
await this.#call?.cancelPlayback(); // first: flush what the service still holds
this.#provider.cancelResponse(); // then: stop the model generating more
this.#aligner.reset(); // and drop the residual of the interrupted turn
}
```
The two natural trigger sites are the provider truncating its own turn because it heard the caller,
and a deterministic verbal interrupt you match yourself. The second exists because the model is
mid-generation when "stop" arrives, so matching the phrase in code is what makes the cut feel instant.
`isVerbalInterrupt` is that matcher, on [Group calls](/typescript-sdk/group-calls#stop-means-stop).
## Refusing a call
`session.end()` is safe to call from inside `onStart`. It returns immediately there rather than
deadlocking against teardown, and the close runs once `onStart` unwinds. Refusing a call is an
ordinary thing to do, so it is one line:
```ts theme={null}
import { isInboundCallAllowed, type CallSession } from "@komaa/standin-sdk";
async onStart(session: CallSession) {
if (!isInboundCallAllowed(policy, allowFrom, session.start.caller.aadId)) {
await session.end("not-allowed");
return;
}
// ... build the agent
}
```
Give the refusal a real reason string. It reaches StandIn as the close reason, so `busy`,
`not-allowed` and `realtime-unavailable` are visible on the call instead of appearing as silence.
The matcher and the reason it has an id branch as well as a phone branch are on
[Security](/typescript-sdk/security#who-may-call-the-agent). It is TypeScript only: a Python worker
writes the same test by hand.
## Ordering and failure
* **Nothing arrives before `onStart` resolves.** Inbound messages are handled serially, so audio
packed into the same TCP read as `session.start` still waits.
* **Context can arrive before there is a handler** on a fast dial, and is dropped in that window.
There is nothing to receive it, and the server does not buffer on a plugin's behalf. The commoner
case is different and is yours: a handler that exists with a provider not connected yet, which is
what [`StartupBuffer`](/typescript-sdk/realtime-providers#what-arrives-before-your-agent-is-ready)
is for.
* **A rejection from any method ends that call alone.** It is logged, the call closes with
`handler-failure`, and the worker keeps taking other calls. One bad call must never take the worker
with it.
## What `session.start` carries
```ts theme={null}
interface Caller {
readonly aadId?: string;
readonly displayName?: string;
readonly tenantId?: string;
}
interface SessionStart {
readonly callId: string;
readonly threadId: string;
readonly caller: Caller;
readonly direction: "inbound" | "outbound";
readonly recordingStatus?: string;
readonly tenantId?: string;
}
```
Both types are exported, so `import { type Caller, type SessionStart } from "@komaa/standin-sdk"`
resolves and you can name them in your own signatures rather than re-declaring the shape.
Blank or absent values normalize to `undefined`, so treat every caller field as optional. A
`direction` that is neither `inbound` nor `outbound` reads as `inbound` rather than being carried
through as a third value every branch has to handle.
`caller.aadId` is empty for guest and anonymous callers, which means an anonymous caller can never
match an allowlist. That is the intended behaviour, not a gap. Never use it as a bare key for
per-caller memory without checking it first either, or every anonymous caller shares one identity.
The two `tenantId` fields are not the same tenant and the names invite the mistake. The top-level one
is the call's, which is the tenant your worker is bound to. `caller.tenantId` describes the
organisation a guest came from, and reaching for it when you meant the other is the one
plausible-looking source that is actively wrong. See
[Meeting recap](/typescript-sdk/minutes#where-the-minutes-go), where posting to the wrong one is the
failure it prevents.
## Next
Options, watchdogs, capacity and draining.
Rates, frames, and the clipping trap.
What arrives before you are ready, and why the agent answers itself.
Let the model hang up, show a picture and look at the screen share.
# Call server
Source: https://docs.komaa.com/typescript-sdk/call-server
CallServer options, watchdogs, capacity, draining and teardown in the StandIn TypeScript SDK.
`CallServer` answers the socket StandIn dials and hands each call to a
[handler](/typescript-sdk/call-handler). It is the same server in both SDKs, method for method, and
behavioural parity is asserted by the shared conformance vectors.
```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
const server = new CallServer({ handlerFactory: () => new MyHandler() });
await server.start();
// ... later
await server.aclose();
```
## What it owns
Every item below is framework-independent, which is exactly why it lives here and not in a plugin:
* the HMAC handshake, its freshness window and its single-use replay guard
* capacity, draining, and the one-live-session-per-`callId` rule
* the frame loop and the wire protocol
* outbound sequence numbers and the audio timeline, which the video tile is stamped from too
* [five timers](#the-watchdogs): the pre-start watchdog, the caller-audio idle watchdog, the `onStart`
timeout, the stale-call reaper and the optional call-duration ceiling
* idempotent teardown that always frees the slot
## Options
Every option falls back to an environment variable, then to a default.
| Option | Environment | Default | Meaning |
| -------------------- | ----------------- | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `handlerFactory` | none | required | Builds one handler per call. Called with no arguments, so a plugin closes over its own config. |
| `secret` | `STANDIN_SECRET` | required | The connection secret from the portal. Must byte-match or the handshake is rejected with 401. |
| `host` | `STANDIN_HOST` | `0.0.0.0` | Bind address. |
| `port` | `STANDIN_PORT` | `9442` | Port. |
| `wsPath` | `STANDIN_WS_PATH` | `/msteams/calling` | Path StandIn dials. |
| `maxConnections` | none | `64` | Concurrent live calls, checked before any crypto runs. |
| `preStartTimeoutMs` | none | `10000` | How long an authenticated socket may stay silent before it is dropped. |
| `audioIdleTimeoutMs` | none | `45000` | Time without caller audio before a live call is declared dead. `0` disables. |
| `onStartTimeoutMs` | none | `15000` | How long `onStart` may take before the call is given up. `0` disables. |
| `staleCallReaperMs` | none | `120000` | How long a call may run with nothing having answered it. `0` disables. |
| `maxCallMs` | none | `0` | Hard ceiling on one call, from `session.start`. `0` disables, so there is no ceiling unless you set one. |
| `goodbyeText` | none | `"We are out of time on this call, so I have to stop here. Goodbye."` | Said through `onGoodbye` before the ceiling ends the call. |
| `goodbyeGraceMs` | none | `6000` | How long that line is given to finish. |
| `onCallOutcome` | none | unset | Called with `(callId, outcome)` when StandIn reports how an outbound call ended with nobody answering. Setting it mounts one extra route; see [Health and routes](#health-and-routes). |
The default bind is `0.0.0.0` because a worker usually runs in a container behind an ingress. Bind
`127.0.0.1` when only a local tunnel should reach the listener. The upgrade is HMAC-authenticated
either way, so this is defence in depth rather than the authentication itself.
`CallServer` throws a `StandInError` at construction when no secret is available. Starting without
one would mean either refusing every call or accepting anything, and neither is a decision a library
should make quietly.
`maxCallMs` is the only per-call cost ceiling in the SDK, and it is **off** by default. Nothing else
ends a call that is still going: the idle watchdog is satisfied while the caller keeps talking, and a
provider billed by the minute keeps billing. Set it on any deployment that pays per minute.
**The units differ between the SDKs, and the names differ with them.** TypeScript takes milliseconds
(`preStartTimeoutMs`, `maxCallMs`, `goodbyeGraceMs`, `staleCallReaperMs`); Python takes seconds
(`pre_start_timeout`, `max_call_seconds`, `goodbye_grace`, `stale_call_reaper_seconds`). There is no
`maxCallSeconds` in this SDK. The compiler catches a straight port; a configuration file copied
between a Python worker and a TypeScript one does not, and `45` where `45000` was meant disables
nothing, it just ends every call in 45 milliseconds.
## Properties and lifecycle
| Member | Purpose |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `await start()` | Bind the listener. Transactional: either it is listening when this resolves, or nothing of it survives. Throws if it is already running. |
| `await aclose()` | Stop the reaper, close every live call with `server-shutdown`, then stop listening. Awaits the calls' real teardown rather than their close promises. |
| `draining` | Set `true` to refuse new calls with 503 while live ones continue. |
| `activeCalls` | Live call count. |
| `running` | Whether the listener is actually bound. |
| `host` | The interface the listener is bound to, or will be. |
| `port` | The bound port, resolved after `start()` when you passed `0`. |
| `wsPath` | The normalized path, always with a leading slash and no trailing one. |
`running` exists because a host that calls connect twice binds a second listener and leaks the first,
and without it there is no way to ask whether the bind succeeded, so a dead worker reports itself as
connected.
`aclose()` ends live calls rather than waiting for them, so a graceful restart is `draining` first
and `aclose()` after. It waits for real teardown rather than returning on an in-flight closer,
because a close that early-returns lets the process exit with teardown still pending, leaking
sessions.
The seven timing and goodbye members (`preStartTimeoutMs`, `audioIdleTimeoutMs`, `onStartTimeoutMs`,
`maxCallMs`, `goodbyeText`, `goodbyeGraceMs`, `staleCallReaperMs`) are readable but `readonly`: set
them in the constructor. Six of the seven are plain public attributes in Python, assignable after
construction, and `pre_start_timeout` is private there as it is here. So a Python worker that retunes
`audio_idle_timeout` on a running server has no equivalent move in TypeScript, which is one of the
few places the two SDKs genuinely differ.
## Health and routes
The listener answers `GET /healthz` with `{"ok": true, "calls": }`.
One more route exists, and only if you asked for it. Set `onCallOutcome` and the server mounts
`POST /outcome/`; leave it unset and that path is a 404 like everything else, so a
worker that never places a call opens no extra surface. That is the point of it being optional rather
than always on.
It sits **under** `wsPath`, so the `/msteams/calling` mount you already have covers it and there is no
second route to publish. See [Expose your agent](/expose#mount-the-call-path).
When it is mounted:
* it is signed with the **v2** request signature over the method, path and a hash of the body, and
verified inside the same 60 second window as the handshake
* the body is read and capped at 8 KB **before** the signature is checked, because v2 signs a hash of
the body and there is nothing to verify until the bytes are in hand. The cap is what stops that
being a way to make the worker read an unbounded request from an unauthenticated peer
* over the cap is 413, a bad signature or a stale timestamp is 401, and a good one is 204
Your callback throwing is logged and still acknowledged, because a plugin that fails to handle an
outcome must not make StandIn retry it forever.
Everything else that is not the WebSocket path is a 404.
## Draining
A clean rolling restart is two steps: set `draining`, then wait for `activeCalls` to reach zero.
```ts theme={null}
process.once("SIGTERM", async () => {
server.draining = true; // new calls get 503; live calls continue
while (server.activeCalls > 0) {
await new Promise((r) => setTimeout(r, 1000));
}
await server.aclose();
});
```
## Admission order
The upgrade is checked in this order, and the order matters:
1. **Path.** Anything outside `/` is 404. A malformed URL or percent escape is 400 rather
than an exception, because this runs before authentication and must never take the process down.
2. **Draining.** 503, so a worker winding down does not accept calls it will never serve.
3. **Capacity.** 503 at `maxConnections`, checked **before any crypto**, so a flood cannot make the
worker spend CPU on signatures for calls it was never going to accept.
4. **Signature.** 401 unless the HMAC verifies inside the 60 second window.
5. **Replay.** 401 when that exact handshake has already been used once.
6. **Duplicate call.** 409 when that `callId` already has a live session.
## The watchdogs
Five timers can end a call. Three bound a call that has not started properly, one bounds a call
nothing ever answered, and one bounds a call that is going perfectly well and will not stop.
A socket that authenticates and then sends nothing holds a call slot. After `preStartTimeoutMs` the
call is closed with `pre-start-timeout`.
A live Microsoft Teams call delivers PCM continuously, because silence is still frames. Audio going quiet for
`audioIdleTimeoutMs` therefore means the call is gone and nobody said so. That happens: the peer keeps
the socket open, and even keeps pinging, while its own teardown is wedged. The call closes with
`caller-idle-timeout`.
The watchdog is armed **before** `onStart`, not after. While `onStart` is awaited the frame loop is
queued behind it, so `session.end` is never read. Armed after, a hung `onStart` would have no watchdog
at all and the `callId` would 409 forever, leaking one slot per inbound call.
`onStart` does real network work and holds the frame loop. After `onStartTimeoutMs` the call closes
with `handler-start-timeout`, separately from `handler-start-failure`, so a third-party outage inside
a plugin is not reported to StandIn as StandIn's own socket failing. Sending both sides to debug the
wrong system is its own kind of outage.
The gap none of the other three cover. `session.start` arrived, so the pre-start watchdog is
satisfied. `onStart` returned, so its timeout is satisfied. The caller is still talking, so the idle
watchdog is satisfied. And yet no agent is on the call: an agent dispatch that never landed is the
commonest misconfiguration there is, and the caller sits listening to nothing.
The reaper ends such a call after `staleCallReaperMs`, 120 seconds by default, with the reason
`no-agent-answered`. It is on by default because it spends nothing and only frees what is already
lost.
A call counts as answered the moment the handler sends audio. A plugin that joins a room and only
listens sends none, so it must say so itself:
```ts theme={null}
session.markAnswered();
```
Call it when your agent's own audio track appears, not when a participant connects: monitors,
recorders and avatar workers all connect, and none of them is an agent answering. `session.answered`
tells you whether anything has.
Off unless you set `maxCallMs`. A caller who will not hang up, a model looping at itself, an
automated system that dialled and never stopped talking: none of them trips a silence check, and all
of them bill by the minute for as long as the socket lives.
On the limit the server flushes playback first, then delivers `goodbyeText` through your handler's
`onGoodbye`, waits `goodbyeGraceMs`, and closes with `call-duration-limit`. The goodbye goes through
the same callback StandIn's own closing line uses, so a plugin needs no new code to honour it, and
the grace is bounded whatever the handler does with it: a plugin that hangs in `onGoodbye` must not
turn a time-limited call into an endless one.
## Teardown
Teardown is idempotent and the **first** reason wins, because a cascade of close causes must not
overwrite the one that actually ended the call. Every caller awaits the same promise.
The order is fixed: cancel the watchdogs, wait for an in-flight `onStart` to unwind (bounded), dispatch
your `aclose`, write the advisory `session.end`, close the socket with a 2 second bound, then release
the slot unconditionally. The release is in a `finally`, so whatever failed above, the `callId` becomes
usable again.
## Logging
The SDK logs to the console by default, because a library that silently swallows "this worker takes no
Microsoft Teams calls" is a library people debug for an hour. Route it into your own logger:
```ts theme={null}
import { setLogger } from "@komaa/standin-sdk";
setLogger({
debug: (m) => myLogger.debug(m),
info: (m) => myLogger.info(m),
warn: (m) => myLogger.warn(m),
error: (m) => myLogger.error(m),
});
```
This is the one place the two SDKs differ on logging. Python has a standard library logging tree to
hang on, so it logs through the logger named `standin` and has no `set_logger` at all; TypeScript has
no such tree, so it exports this function instead.
Ids that appear in logs are rendered log-safe first: control characters are replaced and length is
bounded, so a crafted `callId` cannot forge a log line.
## Limits
A single inbound message is bounded at 2 MB, matching the Python SDK. Outbound agent audio is dropped
rather than queued past 1 MB of unflushed socket buffer (`MAX_AUDIO_BUFFER_BYTES`), because a wedged
peer turns "send everything" into an unbounded queue. Control frames are never shed: a call that
cannot be ended is the failure that shedding exists to prevent.
Frame types the SDK does not know are ignored by contract, so an older plugin and a newer StandIn
interoperate rather than failing. The honest example is a frame type added to the protocol after your
plugin was published: the avatar and vision frames are **not** an example, because this SDK sends and
parses them. See [Vision and the avatar](/typescript-sdk/vision).
## Next
The seven methods, the session object, and how to refuse a call.
Every `STANDIN_` variable the SDK reads, and what reads it.
Ring your own handler on loopback before anyone places a real call.
The HMAC lanes, the replay guard, and who may call the agent.
# Call tools
Source: https://docs.komaa.com/typescript-sdk/call-tools
One set of call capabilities, declared in every provider's own JSON, in the StandIn TypeScript SDK.
A voice model can talk. What it cannot do, unless you tell it, is hang up, put a picture on its own video tile, react with an expression, or look at what the caller is showing.
Those are properties of being on a Microsoft Teams call, not of any provider, so they live in the SDK. Every plugin gets the same set, and a caller gets the same agent whichever provider answers.
## The capabilities
| Tool | Parameters | What it does |
| ------------ | ----------------------------- | ---------------------------------------------------------------- |
| `end_call` | none | Hang up. |
| `express` | `emotion` | Show an emotion on the avatar's face. |
| `show_image` | `url`, `caption?`, `display?` | Put a picture on the bot's video tile. |
| `look` | `source?`, `question?` | Look at the caller's camera or screen share. |
| `look_back` | `question?` | Look again at something already moved past. Recorded calls only. |
`display` is `"fullscreen"` or `"overlay"`, and it is a schema `enum` rather than a list written into the description, because a model obeys a schema enum far more reliably than the same choice spelled out in prose. [Vision and the avatar](/typescript-sdk/vision#fullscreen-or-beside-your-face) covers which to reach for.
The descriptions are written **for a model**. They say when to reach for a tool rather than what the code does, because that sentence is the only thing the model reads before deciding.
`show_page` is deliberately **not** in that list. It needs a renderer, and a deployment with none would still be telling every model it can show a web page, promising the caller and then apologising on every call. An absent tool is honest; a broken one is not. A plugin that does have a renderer registers `SHOW_PAGE_TOOL` itself.
## More tools the SDK ships
None of these are built in either, for the same reason: an agent with nothing behind a tool should not be told it has one. Each is a `ToolSpec` ready to `register`.
| Tool spec | What it needs behind it | Page |
| -------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| `SHOW_PAGE_TOOL` | A page renderer on `VisionTools`. | [Vision and the avatar](/typescript-sdk/vision#showing-a-web-page) |
| `CONSULT_TOOL` | A `Consultant`, so the slow agent can be asked while the caller waits. | [Consulting](/typescript-sdk/consulting) |
| `BACKGROUND_TASK_TOOL` | A `BackgroundTasks` store, so a promise outlives the call. | [Consulting](/typescript-sdk/consulting) |
| `MINUTES_TOOL` | A resolved minutes target, because whether there is anywhere to post depends on the call. | [Meeting recap](/typescript-sdk/minutes) |
| `CALL_BACK_TOOL`, `CHAT_CALLBACK_TOOL` | An outbound lane and its allowlist. | [Reaching people](/typescript-sdk/reaching-people) |
## Declaring them
Every provider invented its own JSON for "here is a function you may call", so `schemas` renders one list in the shape yours wants.
```ts theme={null}
import { CallTools, VisionTools } from "@komaa/standin-sdk";
const tools = new CallTools(session, { vision: new VisionTools(session, { describer }) });
agent.declare(tools.schemas("flat"));
```
`vision` is the only option, and it is optional: leave it out and `CallTools` builds a `VisionTools`
of its own. Pass one when you want to give it a describer, a renderer or a budget, which is nearly
always, because a `VisionTools` with no describer can put pictures up but cannot say what it sees.
The instance is on `tools.vision` afterwards either way.
| Dialect | Shape | Who takes it |
| ----------- | ----------------------------------- | --------------------------------------------- |
| `flat` | `{name, description, parameters}` | Deepgram's Settings, ElevenLabs' client tools |
| `openai` | the same, tagged `type: "function"` | the Realtime API |
| `anthropic` | `{name, description, input_schema}` | the Messages API |
An unknown dialect falls back to `flat` rather than throwing. A tool a model never sees is a worse outcome than a shape one provider happens to also accept, and a crash at connect time helps nobody.
## Running them
Translate your provider's tool-call frame into a name and an object, and hand it over:
```ts theme={null}
const result = await tools.dispatch(name, params); // never throws
```
**Dispatch never throws.** It returns a sentence, because the result goes back to a model that will read it out. "I could not show that because the image was too large" is worth something to a model; a stack trace is not.
When your provider's tool-result frame also carries an error flag, use `run` instead:
```ts theme={null}
const outcome = await tools.run(name, params);
agent.sendResult(callId, outcome.text, !outcome.ok);
```
`ok` is false only when the SDK **knows** the tool did not do what it was asked: a missing or malformed argument, a rejected value, a handler that threw, or a name no tool answers. It is not a verdict on the vision tools, which answer in sentences by design, so "there is nothing to look at" comes back as ok with the reason in the text.
## Tools of your own
Your own tools go in the same place, so the model sees one list:
```ts theme={null}
tools.register(
{
name: "open_ticket",
description: "Use this when the caller reports a fault that needs following up.",
parameters: { summary: { type: "string", description: "What is wrong, in one line." } },
required: ["summary"],
},
openTicket,
);
```
The handler may return a string or a promise of one, and what it returns is what the model is told. Keep it fast: the caller is listening to silence while it runs.
A tool of your own may not shadow a built-in. That is refused when you register it rather than at the first call, because a shadowed `end_call` is an agent that has quietly lost the ability to hang up, and that is not something to discover mid-conversation.
The provider plugins already do all of this. Reach for `CallTools` when you are writing a plugin of your own, or adding capabilities to a handler you wrote yourself.
## Whether to answer at all
A tool decides what an agent can do. In a meeting, the prior question is whether it should say
anything at all, and the SDK ships that separately: `GroupGate` and the wake-phrase matching around
it. It is inert on a 1:1 call, so a plugin that never configures a phrase behaves exactly as it did
before. See [Group calls](/typescript-sdk/group-calls).
## Next
What `look`, `look_back` and `show_image` actually do.
What `express` puts on the face, and the lip-sync beside it.
The two tools that hand real work to a second agent.
The session every one of these tools is bound to.
# Chat
Source: https://docs.komaa.com/typescript-sdk/chat
Answer Microsoft Teams messages from your Node.js agent with ChatChannel, without holding a Bot Framework credential.
The messages lane puts your agent in Microsoft Teams chat. StandIn owns the Microsoft Teams bot, authenticates
the activity, resolves it to your connection and strips the bot @mention. Your handler returns text and
StandIn performs the Microsoft Teams send.
Two consequences follow, and they are the reason this lane exists in the shape it does:
* **Your agent never holds a Bot Framework credential.** There is nothing to register, rotate or leak.
* **The worker dials out.** The socket is opened from your side, exactly like the call lane, so there
is no listener, no port to expose and no tunnel.
```mermaid theme={null}
flowchart TD
T["Microsoft Teams message"] --> G["StandIn
authenticates, normalizes, signs"]
G -->|"pushed down the worker's outbound socket"| C["ChatChannel"]
C -->|"your async handler returns reply text"| G
G --> T2["StandIn sends it to Microsoft Teams"]
```
This is the twin of the Python SDK's `standin.chat`, method for method, with this language's
casing. The shared conformance vectors assert both parse and build identical wire payloads. One
behaviour genuinely differs, on shutdown, and it is called out below.
## The whole thing
```ts theme={null}
import { ChatChannel } from "@komaa/standin-sdk";
const chat = new ChatChannel({
respond: async (message) => `You said: ${message.text}`,
});
await chat.start();
```
`ChatChannel` throws a `StandInError` at construction when no secret is available. Managed
connections only, and that needs no flag: the socket authenticates with your connection secret, so if
it opens at all you are on the StandIn AI Teammate app. A meeting recap
that only posts uses `listenOnly: true` so it never answers.
## Meeting recap
`listenOnly: true` is how a recap lane posts without answering. The host must supply that outbound chat
lane (`chat.start()`) and a summarization callback (`postMinutes` or the plugin consult). Hang-up
does not await the post.
Meeting recaps are best-effort. They may be lost if the worker exits during processing.
## Options
| Option | Default | Meaning |
| ------------ | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `respond` | required | Async callable taking an `InboundMessage` and returning the reply text. An empty string makes the channel say so rather than leaving the user watching a typing indicator forever. |
| `secret` | `STANDIN_CHAT_SECRET`, then `STANDIN_SECRET` | The key this lane signs its dial with. |
| `url` | `STANDIN_CHAT_URL`, then `wss://teams.standin.komaa.com/api/chat/channel` | The chat channel endpoint. |
| `chats` | none | A [`PersonalChats`](#remembering-who-messaged-you) to feed on every inbound message, so a 1:1 call has somewhere to post to. |
| `listenOnly` | `false` | Take messages without answering them. `chats` is still fed, `respond` is never called, and nothing is sent back, not even the typing indicator. For a lane that only posts, such as a meeting recap, when something else already answers this connection's chat. |
### Which secret
A StandIn AI Teammate app connection can be issued a **separate key for chat**. When it is, set `STANDIN_CHAT_SECRET`
and it wins; with one key for both lanes, set only `STANDIN_SECRET` and this lane falls back to it. A
key scoped to one lane cannot be used on the other, which is the point of issuing two.
**Unset the variable, never blank it.** TypeScript stops at the first variable that is merely
present, so `STANDIN_CHAT_SECRET=""` is taken as the chat secret, is empty, and the constructor
throws even though `STANDIN_SECRET` is set and correct. The Python twin uses a truthiness fallback
and quietly falls through to `STANDIN_SECRET` instead, so the same deployment file works there and
fails here. Worse, the error names `STANDIN_SECRET` rather than the variable that actually caused it,
so the message sends you to the wrong place. Unset it and the two SDKs behave alike. Every variable
is listed on [Configuration](/typescript-sdk/configuration).
## Shutting down
```ts theme={null}
await chat.aclose();
```
**The two SDKs differ here, and it is a silent data-loss difference.** TypeScript lets in-flight turns
settle before closing, so a reply already handed to your agent is delivered on shutdown. Python
cancels them. A Python worker that cares about those replies should stop taking new messages and let
the turns finish before calling `aclose`; a TypeScript worker does not need to, but should expect
`aclose` to take as long as its slowest turn has left to run.
## `InboundMessage`
One user message, already authenticated and resolved to your connection.
```ts theme={null}
interface InboundMessage {
readonly tenantId: string;
readonly conversationId: string;
readonly activityId: string;
readonly scope: string; // "personal" for 1:1, otherwise group or channel
readonly text: string;
readonly senderName?: string;
readonly senderAadId?: string;
readonly senderIsGuest: boolean;
readonly senderIsLinkedOwner: boolean;
readonly attachments: Record[];
readonly mentions: string[];
readonly locale?: string;
readonly cardAction?: Record;
readonly bindingId?: string;
}
```
Reserved bot commands are handled by StandIn and never arrive here. In group and channel scope only
messages that @mention the bot are relayed, and the mention is already stripped from `text`.
`cardAction` carries the submit payload of an `Action.Submit` on a card your agent sent, and `text` is
empty on those messages.
`bindingId` says which StandIn connection this conversation resolved to. One tenant can have several
connections, so the tenant alone no longer identifies who a reply is from. You never set it: read it
if you need it, and let `buildReply` echo it.
`attachments` is raw. A pasted screenshot, a dragged-in file or a voice note all arrive here as plain
objects, and reading a JSON blob to a model is the worst thing you can do with one. `buildChatTurn`
turns the whole message into something to hand a model in one call: see
[Attachments in chat](/typescript-sdk/chat-attachments).
`isPersonal(message)` is true when the message came from a 1:1 chat rather than a group or channel.
It is a free function here and a property on the message in Python, `message.is_personal`, so this is
one of the few lines a port has to rewrite rather than rename. Use it to decide how chatty to be:
```ts theme={null}
import { ChatChannel, isPersonal } from "@komaa/standin-sdk";
const chat = new ChatChannel({
respond: async (message) => {
if (!isPersonal(message)) {
return `In a group, briefly: ${await agent.ask(message.text)}`;
}
return agent.ask(message.text);
},
});
```
## `parseInbound` and `buildReply`
Use these when you are relaying messages yourself rather than through `ChatChannel`, for example
behind your own HTTP endpoint.
```ts theme={null}
import { buildReply, parseInbound } from "@komaa/standin-sdk";
const message = parseInbound(rawBody); // throws StandInError naming the problem, map to 400
const reply = buildReply(message, "Sure, here is that report.");
```
The exception type differs between the SDKs: this one throws `StandInError`, the Python twin raises
`ValueError`. Code mapping the failure to HTTP 400 catches a different class in each.
`buildReply` echoes `tenantId` and `conversationId` **exactly**. StandIn rejects a mismatch, and
that check is the cross-tenant leak guard the whole relay rests on. Never rewrite either field, never
construct a reply from a conversation id you obtained anywhere except the inbound message you are
answering.
`buildReply` also sets `replyToId` from the inbound `activityId`, derives an `idempotencyKey` of
`{activityId}:{kind}`, and echoes `bindingId` when the inbound message carried one.
It takes four arguments, not three:
```ts theme={null}
buildReply(message, text, kind?, image?)
```
`kind` defaults to `message`; `typing` and `error` are the other two the channel itself sends, and a
`typing` reply carries neither text nor image, because it is a state rather than a message. `image`
is an `OutboundImage` rendered inline in the reply. Build one with `outboundImage`, which checks the
declared type against the file's real magic bytes, caps it at 1 MB and refuses `image/svg+xml` by
design: see [Sending a picture](/typescript-sdk/sending-pictures).
That page also covers the `MEDIA:` marker convention, which is how an agent that has written a file
gets it sent: `parseMedia` strips a `MEDIA:/path` line out of the reply text so a caller is never read
a temp path aloud, and `loadMedia` turns one into bytes only under the directories
`STANDIN_MEDIA_ROOTS` names. That gate defaults to **none**, because an agent can be talked into
writing `MEDIA:/etc/passwd`, and it is the kind of thing an operator turns on too widely.
`parseInbound` requires `tenantId`, `conversationId` and `activityId` to be non-empty strings. It
rejects a `schemaVersion` above `SCHEMA_VERSION`, which is currently `1`. That version is a **major**
version: additive evolution does not bump it, because the schema already requires receivers to ignore
unknown fields, so an integer above yours means incompatible semantics rather than new fields.
An unknown `scope` is relayed as a group chat rather than rejected, because the scope is an open enum.
## Authentication
The chat dial signs the literal channel name `"chat"` with `signHandshake`, in the
`X-StandIn-Timestamp` and `X-StandIn-Signature` headers. That is the inbound call handshake running in
the opposite direction: same scheme, same 60 second window, the worker signing instead of verifying.
```ts theme={null}
import { SIGNATURE_HEADER, TIMESTAMP_HEADER, nowMs, signHandshake } from "@komaa/standin-sdk";
const timestamp = nowMs();
const headers = {
[TIMESTAMP_HEADER]: String(timestamp),
[SIGNATURE_HEADER]: signHandshake(secret, timestamp, "chat"),
};
```
`ChatChannel` does this for you. You only need it if you are dialing the channel yourself.
The POST relay lane signs the **body** instead, with `signBody` and a 300 second window. Both lanes
are correct and they are not interchangeable. Never "fix" one into the other. See
[Security](/typescript-sdk/security).
## What the channel does for you
Turns are chained per `{tenantId}:{conversationId}`. The schema promises per-conversation ordering,
and independent tasks would let replies overtake each other. A failed turn does not dam the chain.
StandIn delivers at least once. A redelivery of the same activity must not start a second turn, so
the channel keeps a bounded LRU of seen keys. An aged-out redelivery running again is acceptable
at-least-once behaviour; a fresh double-run is not.
A redelivery is still remembered by `PersonalChats` before the dedupe runs, because it is still
evidence that this person has a chat with this bot, and remembering it twice changes nothing.
A typing indicator is sent before your handler runs, not after, so the indicator still lands first
while the agent thinks. If your handler returns an empty string the channel says so rather than
leaving the user watching a typing indicator forever, and a thrown error becomes a short apology.
After a typing indicator, silence looks exactly like a hang.
One turn may run for five minutes before it is abandoned. Serialization means a hung turn would wedge
its conversation forever. The bound is generous because agent turns legitimately run long. A timeout
sends an error reply rather than silence, because after a typing indicator silence looks exactly like
a hang.
The bound is a module constant here and cannot be changed from outside. The Python twin holds the
same five minutes in a public class attribute, `ChatChannel.TURN_TIMEOUT_S`, which a deployment can
raise or lower.
## Remembering who messaged you
A 1:1 **call** carries no thread and no conversation to post into. So when a caller asks for the
minutes, or for anything else in writing, there is nowhere obvious to send it. The only honest source
of that person's own chat is a message they sent the bot.
`PersonalChats` is that memory. Feed it from the chat lane, read it from the call lane:
```ts theme={null}
import { ChatChannel, PersonalChats } from "@komaa/standin-sdk";
const chats = new PersonalChats();
const chat = new ChatChannel({ respond, chats });
// later, on a call:
const mine = chats.forCaller({
callerAadId: session.start.caller.aadId,
tenantId: configuredTenant,
});
```
Four rules decide whether a chat is admissible, and posting call content into the wrong conversation
is the failure all four exist to prevent:
* **It was a 1:1 chat**, decided by the message's scope and by nothing else. An @mention in a team
channel would otherwise make that channel the caller's "chat" and put a private escalation in front
of their whole team. Testing the id prefix instead inverts the rule exactly: a personal chat's id
begins `a:1...` while `19:...` is precisely the group and channel shape, so a prefix test rejects
every real personal chat and admits nothing useful.
* **It is in this tenant**, the one the worker is bound to, never the caller's own.
* **It was seen inside `CHAT_FALLBACK_WINDOW_MS`**, 12 hours: long enough to cover a working day,
short enough that a conversation from last week is not treated as evidence of who is on the phone
today.
* **The call names a caller whose AAD id is the one that sent it.** Without this last rule every
anonymous caller collapses onto whoever chatted last.
At most 512 people are remembered, keyed by tenant and directory id, oldest evicted first, because a
tenant with many users must not grow without bound inside a worker that is also carrying live audio.
`PersonalChat` is the record itself: `conversationId`, `tenantId`, `aadId`, `displayName` and `atMs`.
A message is remembered **before** the at-least-once dedupe runs, but the dedupe still governs the
turn: a redelivery is more evidence that this person has a chat with this bot, and remembering it
twice changes nothing.
Both lanes have to run in **one process** for this to work, or the memory has to be shared some other
way. `PersonalChats` is in-memory: across processes, a caller who has not messaged the bot inside the
window simply has no personal target.
`forCaller` takes an `allowUnidentified` escape hatch, off by default, for a single-operator install
where the only person who ever chats with the bot is the only person who ever calls it. With it on,
an unidentified caller's minutes go to whoever messaged this bot last. Every use logs a warning
naming that, and it should stay off anywhere a second person exists.
The options object's type, `ForCallerOptions`, is not re-exported from the package barrel today, so
pass the argument inline rather than trying to name its type.
`resolveMinutesTarget` on [Meeting recap](/typescript-sdk/minutes) is what actually consumes this.
## Posting without an inbound message
`chat.send()` posts into a Microsoft Teams conversation with nothing to answer, which is useful from inside a
call:
```ts theme={null}
await chat.send({
tenantId: session.start.tenantId ?? "",
conversationId: threadId,
text: "Here is the summary you asked for on the call.",
idempotencyKey: `${session.callId}:summary`,
});
```
It also takes `image` and `bindingId`. It is best-effort and returns `false` rather than throwing,
because a failed post must never break a live call. A `false` here means the socket was not open or
the send threw, and nothing more: the socket carries no status back, so a post StandIn later refuses
still returns `true`. Pass an `idempotencyKey` whenever a retry is possible.
## Next
Screenshots, dragged-in files and voice notes, as one `ChatTurn`.
`outboundImage`, and `MEDIA:` markers with their filesystem gate.
Where a 1:1 call's minutes are allowed to go.
The handshake this lane signs, and the body lane it is not.
# Attachments in chat
Source: https://docs.komaa.com/typescript-sdk/chat-attachments
Turning a pasted screenshot, a dragged-in file or a voice note into one ChatTurn your agent can answer, in the StandIn TypeScript SDK.
A Microsoft Teams message can carry a pasted screenshot, a file dragged in from disk, or a voice note. Without any of this, your handler gets `InboundMessage.attachments` as raw objects, so the best it can do is read a JSON blob to a model and the worst is answer a message about a picture as though nothing had been sent.
## One call does the whole message
```ts theme={null}
import { buildChatTurn, chatImageDataUrl } from "@komaa/standin-sdk";
const turn = await buildChatTurn(message, { transcribe: myStt });
const answer = await agent.ask(turn.query, turn.images.map(chatImageDataUrl));
```
`buildChatTurn` takes the `InboundMessage` your [chat handler](/typescript-sdk/chat) was given and returns a `ChatTurn`: the text to ask, the pictures to hand over, and a plain sentence naming anything that came with the message.
| Field | Type | What it is |
| ---------------- | ------------- | ------------------------------------------------------------------ |
| `query` | `string` | The text to put in front of the model, including every note below. |
| `images` | `ChatImage[]` | Pictures that were fetched and are ready for a vision model. |
| `voiceNote` | `string` | What the voice notes said, or empty. |
| `attachmentNote` | `string` | The "what was attached" sentence, on its own. |
Everything but the message arrives in one `ChatTurnOptions` object, and every field of it is optional:
```ts theme={null}
const turn = await buildChatTurn(message, {
origin: undefined, // leave it off to derive it; see the origin pin below
images: true, // false to skip image fetching entirely
transcribe: undefined, // your speech-to-text function, or no transcription
});
```
`images: false` and no `transcribe` make it a pure text assembler that opens no sockets at all. A handler that fetches nothing pays nothing. There is a fourth option, `fetchFn`, which substitutes the `fetch` every request goes through: it exists so a test can reach every edge of this page without a socket.
This lives in `attachments.ts`, deliberately **not** in `chat.ts`. That module owns the socket, the duplicate check and the per-conversation ordering, and none of that changes here. Fetching is optional work that must never be able to wedge the transport, so it sits beside it rather than inside it. Everything on this page is re-exported at the package root, with exactly one exception, and that exception is under [Images](#images) below.
## The order is the order a person would say it in
`query` is assembled in a fixed order, and the order is not arbitrary: it is the order somebody would have said the same thing out loud.
1. **What they typed.** `message.text`, trimmed.
2. **What they pressed.** The submit payload of an `Action.Submit` on a card this agent sent, as `cardActionNote`. A card message arrives with **empty text**, so without this the agent is asked nothing at all and answers as though the person said nothing. The payload is serialized and truncated at `CARD_PAYLOAD_MAX_CHARS`, which is 4096 characters, and that bounds this agent's own card template rather than a stranger's message.
3. **What they said out loud.** The transcribed voice notes, under `[They sent a voice message]`.
4. **What they attached.** The note from `attachmentsNote`, under `[Attached to this message]`.
Each part is omitted when it is empty, and the parts are joined by blank lines. A message with nothing attached produces a `query` that is exactly the text the person typed.
## It never throws
`buildChatTurn` does not throw. Anything that will not load is **named in the note** instead of failing the turn, because a failed picture is not a reason to leave a question unanswered.
The note distinguishes the two outcomes per attachment:
```text theme={null}
[Attached to this message]
- plan.png [image] (attached)
- q3.xlsx [file] (unreadable)
```
That distinction is the point of the note. A model told a picture was attached and could not be opened says something useful. A model told nothing answers as if the message were empty. And a model told a picture was attached, when nothing was actually fetched, answers **about a picture it has never seen**, which is the worst of the three.
The note is bounded at `ATTACHMENT_NOTE_MAX_LINES` entries, which is ten, followed by a count of the rest, so a message with forty attachments does not become the whole prompt. `attachmentsNote(attachments, status)` is callable on its own, where `status` is a `Map` keyed by the attachment's index in the message. Both that bound and `CARD_PAYLOAD_MAX_CHARS` above are exported from the package root here. The Python twin keeps its two out of its public surface entirely, so that page spells the numbers rather than naming a constant to import.
## Images
```ts theme={null}
import type { ChatImage } from "@komaa/standin-sdk";
```
| Field | Type | What it is |
| ------------ | --------------------- | --------------------------------------------------------------- |
| `dataBase64` | `string` | The image as it arrived, base64. |
| `mime` | `string` | The real media type. The response's own header wins; see below. |
| `name` | `string \| undefined` | The filename, when there was one. |
| `sizeBytes` | `number` | How big it was. |
`ChatImage` is an interface, and an interface carries no accessors, so the two conveniences the Python twin exposes as properties are plain functions here:
```ts theme={null}
import { chatImageData, chatImageDataUrl } from "@komaa/standin-sdk";
chatImageData(image); // Buffer, for an API that uploads a file
chatImageDataUrl(image); // "data:image/png;base64,..." for most vision APIs
```
`fetchChatImages(attachments, { origin })` is the piece underneath, for a handler that wants the pictures without the rest of the turn. It is best-effort per attachment: one that will not load costs that attachment, never the answer.
The caps are at the package root, with one missing on purpose:
```ts theme={null}
import {
IMAGE_FETCH_ATTEMPTS, // 8
IMAGE_FETCH_TIMEOUT_MS, // 10_000
MAX_IMAGES, // 4
} from "@komaa/standin-sdk";
```
`MAX_IMAGES` is the **accept** cap: four pictures are kept from one message, because each becomes a base64 blob in front of a model. `IMAGE_FETCH_TIMEOUT_MS` is how long one image has to arrive, **in milliseconds**. The Python twin spells the same two budgets in seconds, as `IMAGE_FETCH_TIMEOUT_S` and `CLIP_FETCH_TIMEOUT_S`, so a number carried across unchanged is out by a factor of a thousand. Override either cap on a `fetchChatImages` call with its `maxImages` and `timeoutMs` options.
`IMAGE_FETCH_ATTEMPTS` is a **separate** cap, and it is the one that keeps the worst case arithmetic. The accept cap counts only successes, so without it a message naming fifty attachments that all time out still costs fifty timeouts and blows the whole turn budget while nothing is ever accepted. The attempt cap counts requests whatever the outcome: eight tries at ten seconds, and that is the ceiling.
The one number you cannot import is how much one image may weigh. It is 4 MiB, it is the default for the `maxBytes` option, and it matches the per-attachment ceiling StandIn applies anyway, so a larger local number could never be reached.
`MAX_IMAGE_BYTES` **is** exported from the package root, and it is **not** this cap. The root one comes from `vision.ts` and is `1_400_000`: the **outbound** limit on an image you draw on the bot's video tile. The inbound attachment cap is a different number in a different direction of travel, and it keeps its own `MAX_IMAGE_BYTES` inside `attachments.ts` rather than colliding with the other at the root.
Two consequences. Passing the root `MAX_IMAGE_BYTES` as `fetchChatImages`'s `maxBytes` compiles cleanly, is a perfectly real number, and quietly caps inbound attachments at 1.4 MB instead of 4 MiB. And there is no deep import that reaches the other one: the package's `exports` map lists the root and the plugin subpaths only, so `@komaa/standin-sdk/attachments` does not resolve. Leave `maxBytes` unset to take the 4 MiB default, or pass a number of your own. `buildChatTurn` never takes one at all, so a turn assembled through it always uses the default. See [Vision](/typescript-sdk/vision#fetching-an-image-a-model-chose) for the outbound one.
Two more things the fetch does, both about what arrives rather than what was claimed. The media type is judged **before a byte is read**, and the response's own header wins, because an error page would otherwise be base64'd in front of a model as though it were a picture. What the message declared is the fallback, and only when it is itself a media type: a dragged-in file declares a bare extension, and taking that literally would fail every `image/` and `audio/` check. And the size cap holds **while** the body is read rather than after it, because a `content-length` that lies, or is simply absent, otherwise gets to allocate whatever it likes before a later check objects.
## A dragged-in picture is a file, not an image
A pasted screenshot arrives with `kind` of `image`. The **same file** dragged in from disk arrives with `kind` of `file` and a `contentType` that is a bare extension like `png` rather than a media type.
So gating on the declared kind alone is the trap here, and it is a quiet one: the picture is never fetched, but the note still says one was attached, and the model answers about a picture nobody gave it.
An attachment is therefore worth a request when its kind matches **or** when its kind is exactly `file` and its filename's extension, or a declared type that is itself a bare extension, is one of `png`, `jpg`, `jpeg`, `gif`, `webp`, `bmp`, `heic`, `heif`. The widening is `file` only: it exists to rescue the dragged-in case, not to override a kind the message stated outright. What the file then turns out to be is decided by the response, not by the extension that got it tried.
An attachment StandIn marked as not relayable is skipped without a request.
## Voice notes
```ts theme={null}
import type { ChatAudio, Transcriber } from "@komaa/standin-sdk";
```
`ChatAudio` is one clip before anything has transcribed it: `data` as a `Buffer`, `mime`, and `name`. `fetchChatAudio` returns them. `video/` types are accepted as well as `audio/`, because some clients label a voice note with a container type that speech-to-text reads perfectly well, and the same extension widening applies: a voice note relays as a file in practice, so a plugin gating on `kind === "audio"` transcribes nothing, ever.
`Transcriber` is the function **you** supply:
```ts theme={null}
type Transcriber = (data: Buffer, mime: string) => Promise;
```
The core ships none and reads no provider key. Deciding which speech vendor somebody's voice is sent to is not a decision an SDK gets to make on their behalf, and a default would make it silently.
```ts theme={null}
import { transcribeVoiceMessages } from "@komaa/standin-sdk";
const said = await transcribeVoiceMessages(attachments, {
origin,
transcribe: myStt,
});
```
It returns the clips as one block of text, and empty when there are none, when no transcriber was supplied, or when every one failed. With no transcriber it fetches nothing at all rather than downloading audio that has nowhere to go. A transcriber that throws costs that clip and nothing else.
The clip budgets are deliberately larger than the image ones, and all four are at the package root:
```ts theme={null}
import {
CLIP_FETCH_ATTEMPTS, // 4
CLIP_FETCH_TIMEOUT_MS, // 20_000
MAX_CLIPS, // 2
MAX_CLIP_BYTES, // 16 MiB
} from "@komaa/standin-sdk";
```
A voice note is minutes of audio where a picture is one screen, so it needs four times the bytes and twice the wall clock to arrive. The attempt cap is halved to pay for that: four tries at twenty seconds is the same eighty second worst case as eight tries at ten. And two clips rather than four, because each accepted clip is a transcription call on top of the download.
## An engine that only takes a path
Plenty of transcription engines will not take bytes. `spoolClip` writes the clip to a temporary file, named with the extension its media type implies, and hands your callback the path:
```ts theme={null}
import { spoolClip, type Transcriber } from "@komaa/standin-sdk";
const myStt: Transcriber = async (data, mime) =>
spoolClip({ data, mime, name: "" }, (path) => engine.transcribeFile(path));
```
It resolves to whatever your callback resolved to, and it rethrows whatever your callback threw. The file is removed **on the way out, on every path**, including the one where the engine threw. That is not tidiness. A transcription engine is being handed somebody's voice, and a clip left behind in a temporary directory is a copy nobody decided to keep, sitting on disk long after the conversation it came from ended.
This is a callback rather than a context manager, which is the shape TypeScript has for "hold this open for exactly this long". The other difference from the Python twin is that this one always spools into the system temporary directory: there is no directory argument here.
## The origin pin
Every fetch on this page is pinned to the one origin the messages themselves arrived from:
```ts theme={null}
import { chatAttachmentOrigin } from "@komaa/standin-sdk";
chatAttachmentOrigin(); // from STANDIN_CHAT_URL, else the default
chatAttachmentOrigin("wss://example.com/api/chat/channel"); // "https://example.com"
chatAttachmentOrigin("ws://127.0.0.1:9444/x"); // "http://127.0.0.1:9444"
```
It is derived from the chat channel's own URL rather than configured separately, so it is right by construction for a self-hosted or local setup and there is no second setting to get wrong. `ws` maps to `http` and `wss` to `https`, and a default port is not spelled out, so `wss://host:443` and `wss://host` compare equal.
The name is `chatAttachmentOrigin` here and `attachment_origin` in Python. That is the one name on this page that is not a straight casing translation of its twin, so a search for `attachmentOrigin` finds nothing.
It **fails closed**. A URL it cannot resolve, and anything that is not `http`, `https`, `ws` or `wss`, returns `undefined`, and `undefined` means fetch nothing at all.
That direction is the whole design. An attachment URL is signed, but not by you and not for you: it arrives inside a message somebody else wrote, and the pin is the only thing bounding where this worker can be told to go. An unset origin read as "anywhere" would turn a configuration typo into a fetcher that a stranger's message can point at any address it likes, an internal service or a cloud metadata endpoint included.
Two consequences worth knowing. An attachment whose URL is not on that origin is refused **without a request**, so the log line is the only cost. And no redirect is followed: the URL is same-origin and signed, so a redirect off it is already anomalous, and following one would reopen the door the pin just closed, since the pin is checked on the URL in the message and not on wherever a `302` points.
Pass `origin` explicitly when your handler already knows it. Otherwise `buildChatTurn` derives it for you.
## Next
The channel these messages arrive on, and what goes back out.
The signing lanes, and the rest of the fetch posture.
# Checking the install
Source: https://docs.komaa.com/typescript-sdk/checking-the-install
Ring this worker's own handler on loopback and prove audio made the round trip, without a provider bill, a public tunnel or a Microsoft tenant, in the StandIn TypeScript SDK.
Every plugin has to answer one question before anybody trusts it with a real call: does this actually work here? Until now the only way to find out was to place a real call, which costs a provider bill, a public tunnel, a Microsoft Teams tenant and somebody's afternoon.
`runSmoke` answers it on loopback in under a second. It binds an ephemeral listener, connects a real client that speaks the real call wire, streams a few frames of silence, and reports what came back.
```ts theme={null}
import { runSmoke, smokeReport } from "@komaa/standin-sdk";
import { MyHandler } from "./myHandler.js";
const result = await runSmoke(() => new MyHandler());
console.log(smokeReport(result));
process.exit(result.ok ? 0 : 1);
```
`handlerFactory` is the same `() => CallHandler` that [`CallServer`](/typescript-sdk/call-server) takes, called once. It is called with no arguments, so a handler that needs configuration closes over it in the arrow function.
The printer is on the package as **`smokeReport`**, not `report`. The module names the function `report` and the package renames it on the way out, so `import { report } from "@komaa/standin-sdk"` does not resolve. `SyntheticCall`, `runSmoke`, `SmokeCheck` and `SmokeResult` keep their own names.
## What it prints
```text theme={null}
standin: ok
ok secret (generated for this run)
ok listener (127.0.0.1:55064/msteams/calling)
ok call (10 frames in 235 ms)
ok audio (10 frames came back)
```
And when the handler never sent anything back:
```text theme={null}
standin: NOT ok
ok secret (generated for this run)
ok listener (127.0.0.1:55073/msteams/calling)
ok call (4 frames in 93 ms)
FAIL audio (0 frames came back)
without it: the caller would hear nothing
```
Every failed check prints what stops working without it. A report that names a missing surface and not the consequence sends the operator to the source to find out whether it matters.
`smokeReport` returns the string rather than printing it, so a plugin can put it in a log line, a tool result or an HTTP response. When the run recorded an `error`, a final ` error: ...` line follows the checks.
## The four checks
They run in this order, and each one is the next thing that breaks in the field.
| Check | What it proves | What the failure costs |
| ---------- | --------------------------------------------------------------------------------- | ------------------------------------ |
| `secret` | A credential exists for this run. | nothing could authenticate |
| `listener` | The worker bound its socket and has a call path. The detail is the bound address. | no call could ever reach this worker |
| `call` | Connect, handshake, `session.start`, the frame loop and hang-up all completed. | a real call would fail the same way |
| `audio` | Frames came back out of the handler. | the caller would hear nothing |
All four are mandatory, and `result.ok` is true only when every mandatory check passed.
**`ok` is never true when nothing echoed.** The whole point of the run is that audio made the round trip, so a run that proved nothing must not read as a pass. A listener that binds, authenticates and then says nothing is exactly the install people ship by accident.
The run sends `recording.status: active` before the first frame. Handlers commonly gate their output on the call being recorded, and without it a recording-gated handler stays silent and the run reports a false negative.
Frames are PCM16 mono at 16 kHz, 20 ms each, which is `FRAME_BYTES` (640) at the cadence a real call arrives at. `frames` defaults to 10.
If the listener cannot bind, the run returns there and then with two checks, a failed `listener` carrying the bind error, and `echoFrames` at 0. There is nothing left to ring.
The two results are plain readonly shapes:
```ts theme={null}
interface SmokeResult {
readonly checks: SmokeCheck[];
readonly echoFrames: number;
readonly error?: string;
readonly ok: boolean;
}
interface SmokeCheck {
readonly name: string;
readonly ok: boolean;
readonly detail?: string;
readonly cost?: string;
readonly required?: boolean;
}
```
## It never borrows your configuration
The run generates its own connection secret, `randomBytes(16).toString("hex")`, for that run only. It never reads `STANDIN_SECRET`.
Two reasons, and both are real failures rather than tidiness.
A fixed string would be a predictable credential on a live listener. The check is meant to be safe to run anywhere, including on a host that is answering calls.
Borrowing the operator's secret makes the run pass or fail for reasons that have nothing to do with the wire, and it makes the check impossible to run before the secret is configured. That is exactly when people run it.
## Loopback, on port 0
The listener is built with `host: "127.0.0.1"` and `port: 0`, and the port it actually got is read back from `server.port` after it is listening.
Loopback because a verification run has no business being reachable from the network. Port `0` because the alternative is picking a free port in advance with a throwaway socket, which races whoever binds it next. Two runs in the same CI job, or a run next to a real listener, and the loser fails for a reason that has nothing to do with the install.
The call path is the one this worker is configured to serve, so the address in the report is the real one a call would arrive on.
## Nothing that delivers is started
**This is the most important property on the page.** No pending-message sweep, no no-answer reaper, no outbound caller, no chat lane. Nothing that delivers is started, and the listener's own stale-call reaper is off for the run (`staleCallReaperMs: 0`).
A verification command that quietly resumed durable jobs would place real calls and post real messages as a side effect of somebody checking an install.
Durable work is durable on purpose. Parked messages, background-task promises and unanswered outbound legs all survive a restart, because the person they are for is still owed the answer. That is correct for a worker coming up, and completely wrong for a check somebody ran to find out whether an install works. `runSmoke` starts a listener and a handler, and nothing else.
The listener is closed with `aclose()` in a `finally`, on every path including the failing ones, so a run leaves nothing bound behind it.
## A plugin's own checks
`extra` runs after the call and resolves to more checks, so a plugin can prove its own surfaces in the same report.
```ts theme={null}
import { runSmoke, type SmokeCheck } from "@komaa/standin-sdk";
async function myChecks(): Promise {
return [
{ name: "model", ok: await modelReachable(), cost: "the agent could not answer" },
{
name: "browser",
ok: await browserReachable(),
detail: "no display",
cost: "show_page would apologise",
required: false,
},
];
}
const result = await runSmoke(() => new MyHandler(), 10, myChecks);
```
**Mandatory unless declared advisory.** Leave `required` off and the check counts towards `ok`; set `required: false` for anything the call itself survives without. A degraded extra is worth reporting and is not worth failing an install over.
Give every check a `cost`. It is the sentence printed under a failure, and it is what tells the operator whether to act now or after lunch.
```text theme={null}
FAIL browser (no display)
without it: show_page would apologise
```
An `extra` that rejects is recorded, not fatal. The report gains a non-required `plugin checks` entry carrying the error text, because a broken check must not take down the report of the call that did work.
## Pointing one at your own listener
`SyntheticCall` is the client `runSmoke` uses. It is exported because a plugin with its own listener may want to point one at that instead.
```ts theme={null}
import { SyntheticCall } from "@komaa/standin-sdk";
const callId = "smoke-local";
const call = new SyntheticCall(
`ws://127.0.0.1:${port}/msteams/calling/${callId}`,
secret,
callId,
10,
);
await call.run();
console.log(call.echoFrames);
```
`run()` connects, greets, streams, hangs up, and rejects on anything that fails. `echoFrames` is how many audio frames came back. The constructor takes `url`, `secret`, `callId`, then `frames` (10, clamped to at least 1) and `connectTimeoutMs` (5000).
Three things it does that a hand-rolled client usually gets wrong:
* **The handshake is signed freshly for each connect.** The listener enforces single use inside the freshness window, so a retry that reused the headers would loop on 401 and read as a wrong secret.
* **The `callId` in `session.start` is identical to the one in the path.** A start that disagrees with the authenticated path is refused, and the refusal reads as a wire fault.
* **The message listener is attached before the socket opens.** An echo that arrives while the run is still sending is counted, and nothing blocks the 20 ms cadence, which would otherwise make the call look idle to the server's watchdogs. A frame it cannot parse is not counted as an echo.
The socket is closed in a `finally`, on every path. An error path that skipped the close leaks a socket per run inside a long-lived host that exposes the check.
The client lives in the SDK, next to the wire it speaks, deliberately. A copy kept inside a plugin drifts, and the copy that used to exist kept passing against a call path that no longer existed. That is the exact failure a smoke check is for.
## Timeouts
The whole run is bounded at `timeoutMs`, 15000 by default, and the connect at 5000. A listener that accepts and then stops talking would otherwise hang CI, and hang a status tool call for ever.
A run that times out is a recorded failure, not a hang. The `call` check fails with `the call did not finish within 15000 ms`, and the same sentence becomes `result.error`.
```ts theme={null}
function runSmoke(
handlerFactory: HandlerFactory,
frames?: number, // 10
extra?: () => Promise,
timeoutMs?: number, // 15000
): Promise;
```
## In your own command
Wire it to a subcommand and exit on `result.ok`, so CI fails on a bad install.
```ts theme={null}
#!/usr/bin/env node
import { runSmoke, smokeReport } from "@komaa/standin-sdk";
import { MyHandler } from "./myHandler.js";
const result = await runSmoke(() => new MyHandler(), 4);
console.log(smokeReport(result));
process.exit(result.ok ? 0 : 1);
```
It needs no network, no tenant and no credentials, so it also belongs in the test suite:
```ts theme={null}
import { runSmoke } from "@komaa/standin-sdk";
import { expect, it } from "vitest";
import { MyHandler } from "./myHandler.js";
it("answers a call here", async () => {
const result = await runSmoke(() => new MyHandler(), 4);
expect(result.ok).toBe(true);
expect(result.echoFrames).toBeGreaterThan(0);
});
```
Four frames is enough to prove a round trip and keeps the test well under a second. Raise it when your handler needs more audio before it says anything.
The Python SDK has the same module, check for check: see [Checking the install](/python-sdk/checking-the-install) for the `run_smoke` form.
# Configuration
Source: https://docs.komaa.com/typescript-sdk/configuration
Every STANDIN_ variable the SDK reads, the helpers for reading your own, and how a missing optional peer surfaces, in the StandIn TypeScript SDK.
Everything the SDK reads from the environment is listed here, and nothing else in the package reads a
`STANDIN_` variable. Every one of them can be overridden in code, because a constructor option always
beats an environment value.
**Python and TypeScript do not treat a blank `STANDIN_CHAT_SECRET` or a bad `STANDIN_PORT` the same way.** Unset `STANDIN_CHAT_SECRET` rather than exporting it empty. A non-numeric `STANDIN_PORT` is `NaN` here and a `ValueError` in Python. Details are under [How a value is read](#how-a-value-is-read). The Python twin is [Configuration](/python-sdk/configuration).
## Every variable the SDK reads
| Variable | Default | What reads it, and what it does |
| ------------------------------- | ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `STANDIN_SECRET` | none | Your StandIn connection secret. `CallServer`, `ChatChannel` and `OutboundCaller` all refuse to construct without one. |
| `STANDIN_CHAT_SECRET` | falls back to `STANDIN_SECRET` | The key the chat lane signs with, read before `STANDIN_SECRET`. A StandIn AI Teammate app connection issues a separate key for chat; with one key for both lanes, leave this unset. |
| `STANDIN_HOST` | `0.0.0.0` | Bind address for the call listener. |
| `STANDIN_PORT` | `9442` | Port for the call listener. |
| `STANDIN_WS_PATH` | `/msteams/calling` | The path StandIn dials on your worker. |
| `STANDIN_CHAT_URL` | `wss://teams.standin.komaa.com/api/chat/channel` | The chat lane endpoint. Also the origin an inbound attachment may be fetched from, derived from this rather than configured twice. |
| `STANDIN_WORKER_URL` | none | The StandIn control address `OutboundCaller` posts to, which StandIn gives you with your connection secret. Outbound calling is unavailable until it is set, and a value that is not http or https, has no host, or carries credentials is refused when the caller is built rather than when somebody tries to ring. |
| `STANDIN_OUTBOUND_ALLOW` | empty, meaning off | Comma-separated directory ids this agent may ring, read by `OutboundPolicy.fromEnv()`. |
| `STANDIN_OUTBOUND_MAX_PER_HOUR` | `6` | Outbound calls allowed in any rolling hour, across all targets. Zero means **no cap**, not "no calls", so do not reach for it to turn outbound off: leave `STANDIN_OUTBOUND_ALLOW` unset for that. |
| `STANDIN_TENANT_ID` | none | Which organisation `VoiceDelivery` places calls into. Operator configuration only: never the message, the metadata, the model or the caller. Exported as `TENANT_ENV`. |
| `STANDIN_STATE_DIR` | `~/.standin/state` | Where parked outbound messages are kept, and the default parent of unfinished meeting recaps, created owner-only. |
| `STANDIN_RECAP_DIR` | `$STANDIN_STATE_DIR/recap` | Where unfinished meeting recaps sit until they post, expire (24 hours), or hit the retry limit (8). Restart-recoverable local spool. Best-effort unless this directory is on storage that survives a process or container restart: the default is local process disk, which is ephemeral in most containers. |
| `STANDIN_MEDIA_ROOTS` | empty, meaning off | Directories a `MEDIA:` marker may read a local file from. Exported as `MEDIA_ROOTS_ENV`. |
| `STANDIN_VISION_API_URL` | none | The full chat-completions URL `FrameDescriber` posts a frame to, so a voice model can be asked what it is looking at. It is used exactly as written, not as a base to append a path to. |
| `STANDIN_VISION_MODEL` | none | The model name for that endpoint. `FrameDescriber.fromEnv()` returns `undefined` unless both this and the URL are set, and that `undefined` is the signal a plugin uses to tell an agent that looking is unavailable here. |
| `STANDIN_VISION_API_KEY` | none | Sent to that endpoint as a bearer token when set. A local endpoint usually needs none. |
Two of these names are exported as constants rather than written out, so your code and the SDK cannot
drift apart:
```ts theme={null}
import { MEDIA_ROOTS_ENV, TENANT_ENV } from "@komaa/standin-sdk";
```
Provider credentials are not in this table. Each plugin reads its own vendor prefix, `OPENAI_`,
`ELEVENLABS_`, `DEEPGRAM_`, `CARTESIA_` or `LIVEKIT_`, and each plugin page lists its own. That split
is deliberate: a `STANDIN_` variable configures your worker, never the provider you chose to answer
calls with. The three `STANDIN_VISION_` variables sit on the line because the endpoint they name is
yours to pick, so keep in mind that `STANDIN_VISION_API_KEY` is a credential and belongs wherever the
rest of your secrets live.
The Python SDK reads the same fifteen, plus one this SDK has no use for. `STANDIN_SHOW_ROOTS` fences
the document renderer that Python ships and TypeScript does not, so do not go looking for it here.
See [Configuration](/python-sdk/configuration) for that side.
## How a value is read
The rule is the same everywhere: an explicit option wins, then the environment, then the documented
default.
```ts theme={null}
// secret: the option, else STANDIN_SECRET, else refuse to start
const server = new CallServer({ handlerFactory, secret: fromYourVault });
```
Prefer the option whenever the value already exists somewhere better, such as a platform secret store
or a host that resolved it for you. An environment variable that silently overrides a value an
operator can actually see in a portal is a bad day waiting to happen.
Three specifics are worth knowing before they cost you an afternoon:
`ChatChannel` reads the option, then `STANDIN_CHAT_SECRET`, then `STANDIN_SECRET`. That chain stops at
the first value that is **present**, and an exported empty string is present. So
`export STANDIN_CHAT_SECRET=` shadows a perfectly good `STANDIN_SECRET` and the channel throws at
construction instead. Unset the variable rather than blanking it.
The port is read as `Number(process.env.STANDIN_PORT ?? 9442)`. A value that is not a number becomes
`NaN` rather than an error naming the variable, and you find out at `start()`. If you templated that
value from somewhere, check it before you pass it.
The path is trimmed and its slashes normalized, and a path that reduces to `/` is refused with
`wsPath must be a real path such as /msteams/calling`. A worker listening at the root would answer
anything that reached it, so this fails at construction rather than at the first call.
## Reading your own configuration
Every plugin needs the same things: a value that must be set, one that may be, a boolean, a check that
a vendor host is really that vendor's, and a header map. When each plugin wrote its own, the error a
user saw for a missing key depended on which provider they happened to pick. These five helpers are the
shared version.
```ts theme={null}
import { flag, jsonObject, optional, required, vendorHost } from "@komaa/standin-sdk";
```
| Helper | Signature | Behaviour |
| ------------ | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------ |
| `required` | `(name, purpose = "") => string` | Throws `StandInError` naming the variable when it is unset or blank. |
| `optional` | `(name, fallback?) => string \| undefined` | Blank reads as absent, so an exported empty value takes the fallback. |
| `flag` | `(name, fallback = false) => boolean` | Only the exact string `true` is true. |
| `vendorHost` | `(name, fallback, suffix) => string` | Throws unless the host is the bare suffix or ends with it. |
| `jsonObject` | `(name) => Record` | Parses a JSON object and coerces every value to a string. Blank is an empty object; a JSON array or number throws. |
`purpose` completes the sentence "X is required to ...", so write it as a verb phrase and the message
reads properly:
```ts theme={null}
const key = required("ACME_API_KEY", "answer calls with Acme");
// ACME_API_KEY is required to answer calls with Acme
```
**`flag` is strict on purpose.** The value is trimmed and lowercased, then compared to the single word
`true`, so `TRUE` and `True` work and `1`, `yes`, `on` and `y` are all false. Most flags worth having
turn a guard **off**, and a guard must not be disabled by a plausible-looking typo landing in a config
file. A reader who wanted it on and typed `1` gets the safe direction and a behaviour they will
notice, not a guard quietly removed. An unset variable is not a typo, so it returns the fallback
untouched rather than being forced to false.
**`vendorHost` is the one worth reading twice.** Your API key travels to whatever host the
configuration names, so a mistyped or injected host is not a failed call, it is credential
exfiltration. That is why it throws rather than warning:
```ts theme={null}
const host = vendorHost("ACME_API_HOST", "api.acme.com", ".acme.com");
// "acme.com" passes, "eu.acme.com" passes, "api.acme.com.evil.test" throws
```
The host is accepted when it equals the suffix with its leading dot stripped, or when it ends with the
suffix exactly as you passed it.
Pass the suffix **with** its leading dot. `".acme.com"` accepts `api.acme.com` and rejects
`evilacme.com`; `"acme.com"` accepts both, because `evilacme.com` ends with it. That is a guard that
reads as present and is not.
**`jsonObject` never logs the value it rejected.** It is used for header maps, and a header map
carries **your** credentials to somebody else's endpoint. The error says the variable must be a JSON
object and stops there.
These five live at the package barrel in TypeScript, so `import { required } from "@komaa/standin-sdk"`
is the whole import. The Python SDK does **not** re-export them: they are `standin.config.required`,
`standin.config.optional`, `standin.config.flag`, `standin.config.vendor_host` and
`standin.config.json_object`, imported as `from standin.config import required`. Same behaviour, same
argument order, different import line.
## Optional peer dependencies
The core installs with one runtime dependency, `ws`, and needs Node 20 or newer. That floor is the
**core's** floor, not the highest any plugin wants, so a plugin that needs more says so on its own
page. Everything else this package can reach is an optional peer, which is what lets one package ship
every plugin: the core imports nothing from `plugins/`, so the bare specifier works on a machine with
no framework installed at all.
```ts theme={null}
import { CallServer } from "@komaa/standin-sdk"; // no peer needed
```
| Peer | Reached by | What its absence does |
| -------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `@livekit/rtc-node` | `@komaa/standin-sdk/livekit` | `StandInError`, naming both packages and the `npm install` that fixes it. |
| `livekit-server-sdk` | `@komaa/standin-sdk/livekit` | The same error: the plugin needs both, so it loads both together. |
| `openclaw` | `@komaa/standin-sdk/openclaw` | A bare module-not-found, untranslated: this is the one static import of a peer in the package. The case does not arise, because OpenClaw loads the plugin inside its own process, so the host is always already there. |
| `sharp` | the avatar tile relay | A warning, the relay off, and audio unaffected. Frames cannot be encoded, but nothing about the call breaks. |
### How a missing one surfaces
LiveKit is loaded at **use** time rather than at module load, so importing the plugin on a machine
without LiveKit still succeeds and the failure, when it comes, arrives translated:
```text theme={null}
the livekit plugin needs @livekit/rtc-node and livekit-server-sdk:
run `npm install @livekit/rtc-node livekit-server-sdk`
```
That translation is the point. A reader who gets `Cannot find module '@livekit/rtc-node'` thrown out
of somebody else's package cannot tell a missing optional dependency from a broken install, and will
go looking in the wrong place. Naming the install line ends it in one line.
`sharp` is the other shape. It is looked for, and when it is not there the SDK warns and carries on:
```text theme={null}
standin: the avatar tile relay needs sharp to encode frames (npm install sharp);
the relay is off and audio is unaffected
```
The rule behind the difference is that a missing peer throws when it makes a feature impossible and
warns when it makes an **optional** feature unavailable. Supply your own `encoder` on
`TileStreamOptions` and `sharp` is never looked for at all.
This SDK has no dedicated exception for a missing peer, and `StandInError` is narrower than its name
suggests. It covers configuration and the wire: the constructors that refuse to start, the five
helpers above, `parseInbound`, and the LiveKit peer message, with `OutboundError` extending it for the
outbound lane. It does **not** cover everything the SDK throws. The picture, media and fetch helpers,
`outboundImage` and `loadMedia` among them, throw a plain `Error`, so an `instanceof StandInError`
filter silently drops those. Catch `Error` and narrow inside it.
The Python SDK does have a dedicated one, `standin.PluginNotInstalled`, which subclasses both
`StandInError` and `ImportError` so that `except ImportError` and `importlib.util.find_spec` keep
working. Do not port a `catch (PluginNotInstalled)` from Python code: there is nothing here by that
name.
The two SDKs also do not have the same optional set, because the frameworks are not the same in each
language. TypeScript needs two LiveKit packages and uses `sharp` to encode tile frames. Python reaches
LiveKit through one extra, `standin-sdk[livekit]`, encodes the same frames with Pillow from
`standin-sdk[tile]`, and carries a `render` extra for the document renderer that has no counterpart
here. Read the peers of the SDK you are installing, not the other one's.
## Version
```ts theme={null}
import { SCHEMA_VERSION, VERSION } from "@komaa/standin-sdk";
```
Two numbers, and a bug report wants both. `VERSION` is this package's own version, the same string the
Python twin exposes as `standin.__version__`. `SCHEMA_VERSION` is the chat wire schema, versioned
separately and bumped only for a breaking change, because the schema already requires a receiver to
ignore fields it does not know. Knowing both is what separates "your worker is old" from "the message
was wrong".
## Next
The listener most of these variables configure.
Prove the configuration is right before a real call does.
# Consulting and background work
Source: https://docs.komaa.com/typescript-sdk/consulting
Delegate real work from a live call without stalling it, and keep a promise to deliver later, in the StandIn TypeScript SDK.
A voice model has to answer in under a second or the call sounds broken. Real work does not fit in a second.
Looking something up, reading a file, driving a browser, running a tool: those take ten seconds, or five minutes. A caller listening to silence cannot tell thinking from crashed.
So an agent that does real work is two agents. The fast one talks. The slow one works. This is the seam between them, and it is the same whichever provider does the talking and whichever framework does the working.
## Hold on, let me check
`Consultant` runs the slow agent inside a live call. You supply a factory; what it builds is yours.
```ts theme={null}
import { Consultant } from "@komaa/standin-sdk";
const consultant = new Consultant(() => (query) => myAgent.run(query), 45_000);
const answer = await consultant.ask("what did we bill Contoso last quarter?");
```
The first argument is a **factory**, not the agent: it is called on first use and again after a
timeout, because a timed-out consultation cannot be stopped and whatever state it holds is no longer
yours. The second is the timeout in milliseconds, defaulting to `DEFAULT_CONSULT_TIMEOUT_MS`, which
is 45 seconds. `ask` takes an optional second argument that overrides it for one question, which is
how `resume` below runs the same consultant on a five minute budget instead.
Wrap the method, do not pass it bare. `() => myAgent.run` type-checks and then throws at runtime the
moment `run` reads anything on `this`, because a method detached from its instance has no instance.
The same trap catches the `Poster` on [Meeting recap](/typescript-sdk/minutes) and the `Deliverer`
below. Every callable this SDK takes must arrive already bound.
Two behaviours are not obvious, and both exist because a caller is listening.
**One at a time, and the second asker is told so.** A model that can delegate can delegate twice before the first answer lands. Queueing the second means it waits out both timeouts and answers far too late, so it is refused immediately with a sentence the model can say out loud.
**A timeout is admitted.** The work cannot be cancelled: the promise runs to its end whatever this returns. The answer says it stopped rather than promising a follow-up nothing will send, and the timed-out agent is dropped so the next question builds a fresh one.
Your asker must not block. Node has one thread, and a synchronous agent holds it for its whole duration, stopping the call's audio along with everything else. Do the work in a promise, a worker, or another process.
`ask` never throws. It always returns something speakable, because an exception in a tool result is an agent that goes quiet mid-sentence.
## I'll send you the result
When the work will not finish while somebody waits, it becomes a promise, and a promise made on a call that is about to end has to outlive the process that made it.
`BackgroundTasks` writes each task to disk **before** the work starts and removes it only once the result has actually been delivered.
```ts theme={null}
import { BackgroundTasks } from "@komaa/standin-sdk";
const tasks = new BackgroundTasks();
// during the call
const task = tasks.remember(query, session.start.threadId);
tasks.begin(task);
// ...
tasks.done(task);
// once, at worker startup
await tasks.resume(
() => (query) => myAgent.run(query),
(threadId, text) => postToChat(threadId, text),
);
```
`resume` returns how many tasks it actually delivered. Its third argument is the per-task timeout,
`DEFAULT_TASK_TIMEOUT_MS` or five minutes by default, which is far longer than a consultation's
because nobody is on the line waiting for it. `BackgroundTasks` takes `directory`, `ttlMs` and
`resumeLimit`. The directory defaults to a `tasks` folder under `STANDIN_STATE_DIR`, or under
`~/.standin/state` when that is unset. Deliberately never a temp directory: a temp directory passes
every test and loses every parked promise on the next reboot, which is invisible until somebody is
waiting for an answer that no longer exists.
The two-phase claim is what makes a restart safe. A task waiting to run is a `.json`; a task being run is a `.claimed`. A process that dies mid-run leaves a claim behind, and the next startup takes it back once it is old enough to be sure nobody is still working on it. A delivery that fails puts the task back rather than dropping it.
| Guard | Why |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Two hour expiry | An answer long after the question is worse than no answer. |
| Five tasks per startup | A restart storm must not fan out one agent per promise across the fleet. What is left stays queued, never discarded. |
| No thread, no task | A task with nowhere to deliver cannot be kept, so it is dropped loudly at startup rather than run for nothing. |
| Ten minute claim window | Taking a claim somebody is still working on delivers the same answer twice. |
## The two tools
Both are `ToolSpec` values, ready to register on `CallTools`. They are not built in, because an agent with no second agent behind it should not be told it has one.
```ts theme={null}
import { BACKGROUND_TASK_TOOL, CONSULT_TOOL } from "@komaa/standin-sdk";
tools.register(CONSULT_TOOL, (params) => consultant.ask(String(params.query ?? "")));
tools.register(BACKGROUND_TASK_TOOL, startBackgroundTask);
```
Their descriptions are deliberately distinguishable. Without that, a model picks whichever comes first and every long job is run while somebody waits on the line.
Once the background tool is registered, pass `true` as the `Consultant`'s third argument and a timeout will point the caller at it. Leave it off and it will not, because promising a path that does not exist is worse than admitting the work stopped.
## Next
Where both tools are registered, beside the built-ins.
The other job a `Consultant` is normally handed.
When the answer should ring somebody back rather than land in a chat.
`STANDIN_STATE_DIR` and every other variable the SDK reads.
# Group calls
Source: https://docs.komaa.com/typescript-sdk/group-calls
GroupGate, wake phrases, the follow-up window and verbal interrupts for an agent in a Microsoft Teams meeting, in the StandIn TypeScript SDK.
A 1:1 call is simple: every turn is for you. A meeting is not. People talk to each other, and an agent that answers every turn of a meeting is worse than one that says nothing, because it has to be muted or removed before the meeting can continue.
Two decisions follow from that, and both are taken here rather than by your model:
* **who gets answered**, so the agent stays out of a conversation it was not invited into
* **what stops playback**, so "stop" stops it whether or not the model would have chosen to
Both are taken deterministically in code, on a **finished** caller turn, and they consume the same thing: one transcript, once. Neither is left to the model, for the same two reasons. A model asked "was that for you?" has already been invoked, so a refused turn still costs a round trip and a bill. And a model asked to decide whether to keep talking is a model that is still talking while it decides.
```ts theme={null}
import {
DEFAULT_FOLLOW_UP_WINDOW_MS,
GroupGate,
isAddressed,
isMeetingThread,
isVerbalInterrupt,
type GateDecision,
} from "@komaa/standin-sdk";
```
Every name on this page is exported from the package root. There is no deep import for the gate module: the package publishes the root and the framework plugin subpaths and nothing else, so `@komaa/standin-sdk` is the import.
## How the SDK knows it is a group call
`isMeetingThread(threadId)` is the primary signal, and it is a string test:
```ts theme={null}
import { isMeetingThread } from "@komaa/standin-sdk";
isMeetingThread("19:meeting_abc@thread.v2"); // true
isMeetingThread("8:orgid:0f2a..."); // false
isMeetingThread(""); // false
isMeetingThread(undefined); // false
```
Microsoft's thread id for a meeting or a channel conversation begins `19:`. A 1:1 call has no such thread at all, so the presence of one is the signal. It reaches you as `session.start.threadId`, on every call, before the caller has said a word. See [Call handler](/typescript-sdk/call-handler#the-session-object).
**A participant count never arrives on the meeting-join path.** A gate keyed on the count alone is dead on exactly the calls it exists for: every meeting the agent is invited to reads as a 1:1, and the agent answers every turn of it. The failure looks like a gate that was never switched on rather than one that was keyed to the wrong field, which is what makes it expensive to find.
That is not a hypothetical. The same mistake in the recap path sent every meeting recap to one attendee's private chat instead of to the meeting it summarised, because the count was pinned at 1 there too. See [Meeting recap](/typescript-sdk/minutes).
`noteParticipants(count)` is the second signal, and it is deliberately weaker:
```ts theme={null}
import { GroupGate, type CallSession } from "@komaa/standin-sdk";
class MyHandler {
#call: CallSession | undefined;
#gate: GroupGate | undefined;
async onStart(session: CallSession) {
this.#call = session;
this.#gate = new GroupGate({
wakePhrases: ["assistant"],
threadId: session.start.threadId,
});
}
async onContext(text: string) {
// CallServer already parsed the count for you; read it, do not scrape it.
const count = this.#call?.participantCount ?? 0;
if (count > 0) this.#gate?.noteParticipants(count);
await this.#sendToModel(text);
}
}
```
It corroborates and it never overrules. The count is kept as a running maximum, so a count of `1` arriving on a meeting thread adds nothing and takes nothing away: `gate.isGroup` stays true. The count is the signal that goes missing, so it may add certainty and must not remove it.
`isGroup` is true when either signal says so: a meeting thread, or a count of two or more. That covers the group call that is not a meeting thread as well as the meeting whose count never came.
Context sentences arrive at `onContext`, which is also where the recording status and keypad presses land. The count behind the sentence is on the session as `participantCount`, so there is nothing to parse out of the prose. See [Call handler](/typescript-sdk/call-handler#the-seven-methods).
## Wake phrases and the follow-up window
```ts theme={null}
import { GroupGate } from "@komaa/standin-sdk";
const gate = new GroupGate({
wakePhrases: ["assistant", "مساعد"],
requireAddress: true, // default
followUpWindowMs: DEFAULT_FOLLOW_UP_WINDOW_MS, // 12_000
threadId: session.start.threadId, // default: undefined, read as 1:1
});
```
The constructor takes one options object, and one gate belongs to one call: the follow-up window is per-call state. `wakePhrases` is the only required field.
`decide(transcript, nowMs)` returns a readonly `GateDecision` with two fields:
| Field | What it means |
| ----------- | ---------------------------------------------------------- |
| `respond` | Speak an answer to this turn. |
| `addressed` | This turn named the assistant. Opens the follow-up window. |
```ts theme={null}
gate.decide("what do you all think?", 1_000); // respond: false, addressed: false
gate.decide("assistant, summarise that", 2_000); // respond: true, addressed: true
gate.decide("and the second point?", 7_000); // respond: true, addressed: false
gate.decide("anyway, lunch", 20_000); // respond: false, addressed: false
```
The window is why `addressed` is reported separately from `respond` rather than folded into it. A turn inside the window is answered **without having been addressed**, and only an addressed turn reopens the window. Those are different facts about the same turn, and conflating them costs you both: you cannot log who actually named the agent, and a follow-up would keep the floor open forever by being answered.
The window is stored as a timestamp rather than a latched boolean. A missed wake phrase therefore self-heals by the clock instead of stranding the agent silent, or talkative, for the rest of the meeting.
`nowMs` is yours to supply, and the gate never reads a clock itself. It only ever subtracts one of your timestamps from another, so any consistent millisecond source works: the SDK's own LiveKit plugin passes `Date.now()`, and `performance.now()` is the monotonic option if you would rather a system clock adjustment could not open or close the floor. Pick one and use it for every `decide` on that call. It also makes the whole decision testable without waiting twelve seconds.
If you also run the [echo guard](/typescript-sdk/realtime-providers), do not reuse a `performance.now()` value there. `shouldSuppressEcho` compares against `Date.now()` internally and needs epoch milliseconds. The gate accepts either because it only compares differences; the echo guard does not.
`requireAddress: false` turns the gate off entirely and answers everything, on a meeting thread or not.
On a 1:1 call the gate returns `respond: true` for every turn, and `addressed` still tells you truthfully whether the caller used your name. That is worth having even when nothing is being gated.
## A gate with no phrase never opens
`gate.active` is the question "is this gate muting anything right now?", and it is false unless all three of these hold: the call is a group, `requireAddress` is on, and **at least one non-blank wake phrase is configured**.
```ts theme={null}
const gate = new GroupGate({ wakePhrases: [], threadId: "19:meeting_abc@thread.v2" });
gate.active; // false
gate.decide("hello", 1_000); // respond: true
```
An empty wake list means the gate is off, on purpose. A gate with nothing that could ever open it would mute the assistant for the whole call, and nothing in the transcript would ever explain why. Silence is the one failure a caller cannot interpret and an operator cannot debug: it looks identical to a dead socket, a crashed worker and an agent that is thinking. A blank or whitespace-only phrase counts as no phrase, because a configuration file with an empty string in it is the usual way this happens.
## Matching a name
`isAddressed(transcript, wakePhrases)` is a case-insensitive match on **word boundaries**, not substrings.
```ts theme={null}
import { isAddressed } from "@komaa/standin-sdk";
isAddressed("assistant, what is the plan?", ["assistant"]); // true
isAddressed("ASSISTANT, hello", ["assistant"]); // true
isAddressed("the assistants are ready", ["assistant"]); // false
isAddressed("مرحبا مساعد كيف حالك", ["مساعد"]); // true
```
Boundaries rather than substrings because a substring match on a short name fires inside ordinary words, and the way that failure presents itself is the agent interrupting a meeting, which is the exact thing the gate was added to prevent.
The boundary is written as "not a letter, digit or underscore" with Unicode property escapes rather than as a Latin word character, so an Arabic wake phrase behaves the same as a Latin one and needs no special case in your configuration. Phrases are trimmed and lowercased before matching, regex metacharacters in them are escaped, and an empty phrase never matches.
Choose a wake phrase that is not an ordinary word in the language the call will be held in. Word boundaries stop `assistant` matching `assistants`; they do not stop a name that happens to be a common noun from matching every time somebody uses it.
## Stop means stop
```ts theme={null}
import { isVerbalInterrupt } from "@komaa/standin-sdk";
isVerbalInterrupt("stop by the store"); // false
isVerbalInterrupt("ok stop"); // true
isVerbalInterrupt("Assistant, stop.", ["assistant"]); // true
isVerbalInterrupt("hold on assistant", ["assistant"]); // true
isVerbalInterrupt("توقف"); // true
```
`isVerbalInterrupt(text, wakePhrases?)` matches the **whole normalised utterance**, never a substring. "stop by the store" is a sentence somebody said in a meeting, and a substring rule would cut the agent off mid-answer every time it came up. Whole-utterance is the rule that makes this safe to run on every turn.
Before the comparison the utterance is normalised: lowercased, apostrophes deleted rather than split on so `that's` becomes `thats`, combining marks and the Arabic tatweel deleted so a vocalised `تَوَقَّف` matches `توقف`, and everything that is not a letter collapsed to a space. Letters in any script survive, which is what lets an Arabic interrupt cut as instantly as an English one.
Then filler and wake phrases are peeled off **both ends**, repeatedly until nothing changes, because the name can sit outside the filler or inside it and both orders are things people say. Pass your wake phrases in, or the name you chose will be the thing that defeats the match.
The second parameter is typed `string[]`, while `GroupGate.wakePhrases` is `readonly string[]`. Handing the gate's own list straight in is a compile error. Spread it: `isVerbalInterrupt(text, [...gate.wakePhrases])`.
What is left has to be **at most four words** and has to match the phrase set exactly. The cap is the belt to the whole-utterance braces: a long sentence cannot reduce to an interrupt by accident, whatever the filler lists do.
Interrupt phrases ship for two languages: English and Arabic. `stop`, `stop it`, `stop talking`, `wait`, `hold on`, `hang on`, `never mind`, `pause`, `one second`, `توقف`, `خلاص`, `لحظة` and their close neighbours. The set is a module-private constant in the gate module and there is no hook to extend it, so a phrase your callers use and the set does not is a wrapper around `isVerbalInterrupt`, not a configuration key.
Two consequences worth knowing before you rely on it. The wake phrase on its own is an address and not a cut, because `"assistant?"` peels down to nothing and nothing never matches. And phrases that are merely close to the set are not in it: `"could you stop"` returns `false`, because widening the set to catch it is how a whole-utterance rule turns back into substring matching.
An interrupt is worth nothing on its own. Pair it with `session.cancelPlayback()`. `onCallerTurn` below is **your own** method and not a `CallHandler` one: wire it to whatever your provider calls a finished caller transcript. Nothing in [Call handler](/typescript-sdk/call-handler#the-seven-methods) delivers a transcript, because only your provider produces one, so a method named for it on a handler is never called by anything.
```ts theme={null}
/** The provider reported a finished caller transcript. */
async onCallerTurn(text: string) {
if (isVerbalInterrupt(text, [...this.#wakePhrases])) {
await this.#call?.cancelPlayback(); // first: un-send what StandIn still holds
this.#provider.cancelResponse(); // then: stop the model generating more
return;
}
}
```
`cancelPlayback()` first, because it is the only lever that un-sends audio StandIn already holds. Cancelling the model first stops it producing more, and the caller still hears every buffered sample of the answer they just interrupted. A caller who has said "stop" has stopped listening, and several seconds of the agent carrying on afterwards reads as the agent ignoring them. See [Barge-in](/typescript-sdk/call-handler#cancelplayback-is-the-barge-in).
Suppress the reply to the interruption itself too. "stop" does not want an answer, it wants silence, and an agent that says "of course, I will stop" has not stopped.
### A request for silence gives the floor back
An interrupt cuts the sentence in progress. It does not close the follow-up window, so in a meeting the assistant would still answer the next thing anybody says. That is right after "wait", because whoever said it is about to ask something. It is wrong after "be quiet".
`isDismissal(text, wakePhrases)` tells the two apart, and `gate.closeWindow()` makes the next turn name the assistant again:
```ts theme={null}
if (isVerbalInterrupt(text, [...this.#wakePhrases])) {
if (isDismissal(text, [...this.#wakePhrases])) {
this.#gate?.closeWindow(); // "be quiet": wait to be named again
}
await this.#call?.cancelPlayback();
this.#provider.cancelResponse();
return;
}
```
Every dismissal is an interrupt, so check it inside the interrupt branch. On a 1:1 call there is no window to close and the call does nothing.
## Where to call it
On a finished transcript, once, before you spend a model call.
```ts theme={null}
/** The provider reported a finished caller transcript. Your method, not the SDK's. */
async onCallerTurn(text: string) {
// Interrupts first: "stop" is not a turn to answer, and scoring it for
// wake phrases can only end in answering it.
if (isVerbalInterrupt(text, [...this.#wakePhrases])) {
if (isDismissal(text, [...this.#wakePhrases])) this.#gate?.closeWindow();
await this.#call?.cancelPlayback();
this.#provider.cancelResponse();
return;
}
const decision = this.#gate!.decide(text, Date.now());
if (!decision.respond) {
this.#provider.cancelResponse();
return;
}
this.#provider.createResponse();
}
```
Three things about that order and that placement.
**Interrupts before the gate.** Run the gate first and "stop" is scored as an ordinary turn: inside the follow-up window it is answered, and "of course, I will stop" is the one reply a caller who said stop must never hear. Outside the window the gate refuses it, you return early, and the cut never happens at all while the agent carries on talking.
**Finished transcripts, not partials.** A partial has not necessarily reached the name yet, so a gate run on partials refuses turns it would have accepted a syllable later. If you want the window stamped as early as possible, guard the partial path with `isAddressed` and let only that call `decide`: because the window is a timestamp, a stamp can only ever open the floor, never close it.
```ts theme={null}
#onCallerTranscript(text: string, final: boolean) {
const gate = this.#gate;
if (gate === undefined) return;
if (!final) {
if (isAddressed(text, gate.wakePhrases)) gate.decide(text, Date.now());
return;
}
this.#lastDecision = gate.decide(text, Date.now());
}
```
**Before the model call, not after.** A refused turn should cost nothing. If your provider answers on its own voice detection, put it in manual response mode while `gate.isGroup` is true and create the response yourself when `decision.respond` is true. A response cancelled after it has started generating is one the caller has already heard the beginning of, and a meeting hears the agent start to speak and then stop, which is more disruptive than either answering or staying quiet.
There is one thing to check before any of this: whether your provider gives you a caller transcript at all. Without one there is nothing to gate on and nothing to match an interrupt against. Log that loudly at startup rather than shipping a gate that silently never fires, and leave auto-response on, because an ungated assistant is at least an assistant.
## What is not the same in the Python SDK
The gate itself is. `GroupGate`, `GateDecision`, `is_addressed`, `is_meeting_thread`, `is_verbal_interrupt` and `DEFAULT_FOLLOW_UP_WINDOW_MS` exist there under the same names in snake\_case, with the same twelve second default, the same thread-id rule and the same "no phrase means no gate" catch. Porting a gate between the languages changes the casing, and one thing that is language-shaped rather than a decision: the constructor takes a single options object here and keyword arguments there.
Two things around it are not the same, and both will bite a port that assumes otherwise.
**The interrupt phrase set differs by language and by membership.** This SDK ships English and Arabic; the Python SDK ships English, Arabic, French and German. The lists are not translations of each other either: `"stop talking"`, `"hang on"`, `"pause"` and `"one second"` match here and not there, and `"cancel"`, `"stop stop"` and `"hold up"` match there and not here. There is also no four-word cap in the Python implementation. Test the phrases your callers actually use, in the SDK you are actually running, rather than porting a test suite across.
**The echo guard is a different shape entirely.** The Python SDK has an `EchoGuard` class that owns its own playout clock, and because that clock is monotonic, one monotonic source serves both the gate and the guard there. This SDK has a `shouldSuppressEcho` function, no class, and a playout horizon you keep yourself: it must be on the epoch-millisecond scale, so the source that serves both here is `Date.now()`, and the monotonic option the gate would otherwise accept stops the guard firing at all. [Realtime providers](/typescript-sdk/realtime-providers) spells the difference out in full.
## Next
The startup buffer and the echo guard, for a speech-to-speech agent.
For an agent that is not speech to speech: segmentation, WAV and paced playback.
Where the transcript, the context and `cancelPlayback` reach you.
The other decision the meeting thread id makes for you.
# Meeting recap
Source: https://docs.komaa.com/typescript-sdk/minutes
Minutes of the call, including what was shown on screen, with a Word document, in the StandIn TypeScript SDK.
A recap is the one thing people ask an agent for that it cannot do while the call is happening. It needs the whole conversation, so it happens at the end, and by then the caller has usually gone.
What makes this one worth having is the second track. Every transcript-first recap tool is blind to the screen share. Your agent was on the call and could see it.
## Keeping the record
Feed the transcript as the call runs, from whichever callbacks your plugin already has.
```ts theme={null}
import { Transcript } from "@komaa/standin-sdk";
const transcript = new Transcript();
transcript.add(callerName, "we should push the launch to March");
transcript.add("Assistant", "noted", "assistant");
transcript.addVisual("Sara's shared screen: the Q3 revenue dashboard");
```
The third argument is the `TurnRole`, `"caller"` unless you say otherwise, and it is what tells the
two sides apart in the document. A consecutive turn from the same speaker **and** the same role is
merged into the one before it, because a live transcript arrives as fragments and half-sentences fed
to a model as separate turns make the minutes read like a stutter. Merging never crosses speakers or
roles: filing one person's words under another's name is worse than no attribution, because it is
confidently wrong.
Both tracks are bounded, because a two-hour meeting sits in the memory of a process that is also carrying live audio. What survives is the tail, since the end of a meeting is what minutes are mostly about.
Here are the numbers, so "bounded" is something you can check:
| Bound | Value | What it limits |
| ---------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MAX_TRANSCRIPT_TURNS` | `600` | Turns held at once. Also the default for `new Transcript({ maxEntries })`. |
| `MAX_TRANSCRIPT_VISUALS` | `60` | Things shown. Far fewer than turns, because a screen changes slowly. |
| `MAX_TRANSCRIPT_ENTRY_CHARS` | `1000` | How long one entry may grow before the next turn from the same speaker starts a fresh one. Without it, an hour of one person talking coalesces into a single ever-growing entry the entry count can never trim. |
| `MAX_TRANSCRIPT_ENTRIES` | `40` | Entries that reach the summarising model. |
| `MAX_TRANSCRIPT_CHARS` | `12000` | Characters that reach it. |
| `RECAP_MIN_TURNS` | `4` | Below this there is no meeting to summarise, only a greeting. Advisory: it is exported for your plugin to check before it offers a recap at all. `postMinutes` itself only refuses an **empty** transcript, so a two-turn call still gets summarised if you ask. |
All six resolve from `@komaa/standin-sdk`. In the Python SDK only `MAX_TRANSCRIPT_ENTRIES`,
`MAX_TRANSCRIPT_ENTRY_CHARS` and `RECAP_MIN_TURNS` are on the barrel, and the three hard holding caps
are `from standin.minutes import ...` there. Nothing about the values differs, only where you import
them from.
Consecutive visual repeats are collapsed. The vision lane describes whatever is on screen each time it is asked, and a slide nobody changed would otherwise fill the record with the same line.
## Where the minutes go
Decided **once**, before anything is written, and then passed to every later step.
`resolveMinutesTarget` returns a `DeliveryTarget`, and it holds three fields and nothing else: a
`kind` of `"thread"` or `"caller-dm"`, a `conversationId` and a `tenantId`.
```ts theme={null}
import { ChatChannel, PersonalChats, resolveMinutesTarget } from "@komaa/standin-sdk";
// One instance, shared: the chat lane fills it, the call lane reads it.
const chats = new PersonalChats();
const chat = new ChatChannel({ respond, chats });
// ... and on a call:
const target = resolveMinutesTarget({
threadId: session.start.threadId,
humanCount: participants,
callerAadId: session.start.caller.aadId,
callerChat: chats.forCaller({
callerAadId: session.start.caller.aadId,
tenantId: configuredTenant,
}),
sessionTenantId: session.start.tenantId,
configTenantId: configuredTenant,
});
```
`chats` is a [`PersonalChats`](/typescript-sdk/chat#remembering-who-messaged-you),
and it is the only honest answer to "where does a 1:1 call post?". A 1:1 call carries no thread, so
the caller's own chat has to have been seen on the messages lane first. Both lanes have to be in one
process for that, or the memory has to be shared some other way.
This is the highest-consequence rule in the feature. A send with no pinned recipient falls back to whatever conversation the sending code last saw, and a customer's meeting minutes are the most sensitive thing this product produces. So the target is one immutable value, and no step downstream is allowed to work out a recipient of its own.
Three details are load-bearing, and each one is a real failure that happened:
**A meeting thread counts even when the participant count says one.** The count only arrives on topologies that send a participants frame, so on some deployments it stays pinned at 1. A count-only test delivered the minutes of a group call into one attendee's private chat.
**The caller's own chat is admitted by scope, never by the shape of its id.** A personal chat and a group thread are told apart by what the message said its scope was. Testing the id prefix inverts it, admitting a team channel and rejecting every real private chat.
**The tenant is the session's, then the configured one, then the remembered sender's.** Never the caller's own. A guest's tenant describes the organisation they came from, which is not the one this worker is bound to, and it is the one plausible-looking source that is actively wrong.
A call that identifies nobody gets no target, and therefore no minutes. That is the correct outcome: there is no conversation that can be asserted as theirs.
Pass a list when more than one place is admissible, best first. The walk advances **only on a 404**, because that is the only answer that proves nothing was delivered, so it is the only one where trying again cannot duplicate. A 401 or a 5xx would fail identically elsewhere.
The SDK's own chat lane writes to a socket and gets no status back, so it can report that a message was sent and never that it was refused. A poster built on it never advances the walk, which is the safe behaviour rather than a bug. Supply a poster that surfaces a status if you want the fallback to fire.
## Writing it up
```ts theme={null}
import { postMinutes } from "@komaa/standin-sdk";
const result = await postMinutes(
(prompt) => consultant.ask(prompt),
transcript,
target,
postToChat,
{
documentDir: "/var/standin/minutes",
subtitle: `Call with ${caller} - ~${durationMin} min, ${humans} human participants.`,
},
);
await lane.say(result.spoken);
```
Wrap the summariser in an arrow function rather than passing `consultant.ask` bare. It type-checks
either way, because `Summariser` is just `(prompt: string) => Promise`, and then throws a
`TypeError` at the first private field read: `ask` reads the `Consultant`'s own state, and detached
from its instance there is none. The same applies to any method you hand in as a callable here, and
to your `Poster`.
`lane.say` above is the `VoiceLane` this call is running on. `CallSession` has no `say`: the session
sends PCM, and turning a sentence into PCM is either the lane's job or your provider's. A realtime
plugin has neither, and hands `result.spoken` back as the tool result instead, which is what the model
then reads out.
`documentDir` decides whether a Word document is written at all: omit it and none is. `subtitle` is
the line under the document title. `assistantLabel` and `callerLabel` name the two sides in the
attributed transcript.
`postMinutes` never raises. It normally runs during teardown, where an exception takes the whole teardown with it, so every failure comes back as a sentence instead. It degrades a step at a time: a document that cannot be built sends text only, a send that fails is logged, and the spoken sentence is always there.
Two outcomes look similar and are kept apart on purpose. A call with nothing said is told it was too short. A call with plenty said but no conversation to post into is told exactly that. Conflating them tells people their conversation did not count when it did.
| Field | What it holds |
| ----------- | ----------------------------------------------------------------------------------------- |
| `spoken` | One sentence for the agent to say. Always present, including on failure. |
| `minutes` | The minutes themselves, so a caller can do something else with them when the post failed. |
| `document` | Where the Word document was written, when one was. |
| `delivered` | Whether the minutes actually reached the chat. |
| `target` | Which of the admissible conversations took them. |
Your poster may answer with a plain `true` or with a structured outcome carrying a status. Both are read correctly. Do not hand back an object and expect it to be truth-tested: every object is truthy, and a post StandIn rejected would report as delivered.
## The document
`writeMinutesDocx` emits a Word-openable `.docx` with no dependencies. A document format library would be a dependency every install pays for so that the small fraction who ask for minutes get a file, which is the wrong trade for an SDK.
Give it `sections` from `parseMinutesSections` and it renders real headings instead of a flat wall of lines. A model asked for `### Key points` will write `## Key points` or `**Key points:**` depending on the model and the day, so all of those are accepted; a section that ends up with nothing in it is left out entirely rather than printed as a heading over white space.
Give it `transcript` and it appends an attributed transcript, which is the half a transcript-only tool cannot produce: the call gave you the real speaker per utterance. A turn that already carries its own `Name:` prefix is written through untouched, because relabelling it as a generic caller destroys the attribution and double-prefixing reads as a transcription fault.
Delivery to the chat is text. The document is written to disk beside it, for whoever keeps the record. A meeting chat cannot be sent a file by a bot the way a person can, so a document promised into the chat would be a promise that quietly fails. Say so in the message rather than leaving it as a silent difference: `DOCUMENT_NOT_ATTACHED` is that sentence.
## The prompt
`minutesPrompt` asks for Key Points, Decisions, Action Items and, when anything was shown, Presented.
The instruction not to infer what was on screen is the load-bearing one. A model handed "Sara shared a dashboard" will happily invent the numbers on it, and minutes that invent numbers are worse than minutes with a gap.
`isSummaryRequest` recognises somebody asking for the write-up in conversation. It needs both halves: "summarise" alone is asked about a document, an email, or a page the agent is looking at.
`MINUTES_TOOL` is the tool declaration, ready to register on `CallTools`. It is named `post_meeting_minutes`, not after the function that does the work, because a model handed two things with one name cannot tell which it is calling. It is not built in: whether there is anywhere to post depends on the call, and `resolveMinutesTarget` is what answers that.
Answer the model the moment it calls, then run the recap. A tool call with no result stalls the turn, and the recap is a full model run plus a document write plus a send, so the caller sits in silence wondering whether anything is happening.
## From the OpenClaw and Hermes plugins
You do not have to wire `postMinutes` yourself. Each plugin has a switch (`meetingRecap` / `meeting_recap`), off by default.
When it is on, hang-up writes a **restart-recoverable local spool** first (`STANDIN_RECAP_DIR`, else `STANDIN_STATE_DIR/recap`, else `~/.standin/state/recap`) and posts after the call is already gone. A process restart drains unfinished jobs if that directory survives the process. Treat recap as best-effort unless that directory is on storage that survives a process or container restart: the default path is local process disk, which is ephemeral in most containers. The file stays until the minutes post, the job expires (24 hours), or it hits the retry limit (8).
OpenClaw always opens its own **listen-only** chat lane for recap and never answers on it. Hermes reuses the host `respond` handler when one exists, and opens a listen-only lane only when it does not.
The host must provide an outbound chat lane and a summarization consult. Hang-up does not await the consult. Meeting recaps are best-effort. They may be lost if the worker exits during processing.
Delivery needs StandIn's outbound chat socket, which is the [StandIn AI Teammate app](/tiers#the-standin-ai-teammate-app). A Free / bring-your-own-bot identity answers chat on its own Azure Bot messaging URL, not this socket.
Plugin pages: [OpenClaw](/typescript-sdk/plugins/openclaw#meeting-recap). The Python twin is [Hermes Agent](/python-sdk/plugins/hermes-agent#meeting-recap).
## Next
`PersonalChats`, and the reply guard every post rests on.
The slow agent that writes the minutes while the fast one talks.
Registering `MINUTES_TOOL` alongside the built-ins.
Where the visual track of the transcript comes from.
# TypeScript SDK
Source: https://docs.komaa.com/typescript-sdk/overview
Put your AI agent into a Microsoft Teams call from Node.js. The shared core every StandIn TypeScript plugin builds on.
`@komaa/standin-sdk` is one end of a socket. StandIn handles the Microsoft Teams side and dials your
worker, once per call, over an authenticated WebSocket. This package answers that dial, speaks the
call protocol, and hands each call to your code.
```bash theme={null}
npm install @komaa/standin-sdk
```
One package. The core is the bare specifier and imports nothing from the plugins, so
`import { CallServer } from "@komaa/standin-sdk"` works with no framework installed. Each
plugin is a subpath of that same package: `@komaa/standin-sdk/openclaw`,
`@komaa/standin-sdk/echo`.
## What you actually write
The whole contract between the SDK and your agent is seven optional methods and a session object.
Nothing extends anything, and you implement only the ones you care about.
```ts theme={null}
import { CallServer, type CallSession } from "@komaa/standin-sdk";
class EchoHandler {
#call: CallSession | undefined;
async onStart(session: CallSession) {
this.#call = session;
}
async onCallerAudio(pcm: Buffer) {
await this.#call?.sendAudio(pcm);
}
}
const server = new CallServer({ handlerFactory: () => new EchoHandler() });
await server.start();
```
That is a worker that answers a real Microsoft Teams call. Everything else on this page is detail about what
the server is doing behind those two methods.
## Who owns what
`CallServer` owns everything that is the same whichever agent framework is on the other side. That
list is not marketing, it is the reason the SDK exists:
* the socket StandIn dials, and the HMAC handshake with its single-use replay guard
* capacity, draining, and the one-live-session-per-call rule
* the wire protocol and the frame loop
* outbound sequence numbers and the audio timeline, for voice and for the video tile alike
* [five timers](/typescript-sdk/call-server#the-watchdogs): a pre-start watchdog, a caller-audio idle
watchdog, an `onStart` timeout, a stale-call reaper and an optional call-duration ceiling
* idempotent teardown that always frees the slot
Your handler owns the part that differs: what to do with a caller's voice, and where the reply comes
from. If you find yourself reimplementing a sequence number or a resampler, that is a bug in the SDK,
not in your plugin.
## The two SDKs are one API
The wire protocol, the audio format and the HMAC are byte-identical between the languages, and shared
conformance vectors prove it rather than promising it: both protocol modules are generated from one
schema and gated against drift.
The core seam is the same in both. Same handler methods, same session, same watchdogs, same defaults.
A handler ported between the languages changes only its method names.
```python Python theme={null}
class MyHandler:
async def on_start(self, session: CallSession) -> None:
self._call = session
async def on_caller_audio(self, pcm: bytes) -> None:
await self._call.send_audio(pcm)
```
```ts TypeScript theme={null}
class MyHandler {
#call!: CallSession;
async onStart(session: CallSession) {
this.#call = session;
}
async onCallerAudio(pcm: Buffer) {
await this.#call.sendAudio(pcm);
}
}
```
Python is snake\_case, TypeScript is camelCase. Around the seam a few peripheral helpers exist in one
language only, and the page each one belongs to says so where it applies. Three that matter here:
* **Timers are milliseconds in TypeScript and seconds in Python.** `preStartTimeoutMs: 10000` against
`pre_start_timeout=10.0`. The names differ too, so the compiler catches a straight port, but a
config file copied between two workers does not.
* **The two optional video callbacks sit differently.** Every `CallHandler` member is optional in
TypeScript, so `onVideoFrame` and `onSpeakerChange` are simply two more of them. Python keeps them
on separate `VideoHandler` and `SpeakerHandler` protocols, because `CallHandler` there is
`runtime_checkable` and a sixth member would break `isinstance` for every handler written before
the vision lane existed.
* **Who may call the agent is a TypeScript module.** `isInboundCallAllowed` and its three companions
ship in this SDK only. A Python worker writes the same check in its own handler. See
[Security](/typescript-sdk/security#who-may-call-the-agent).
## Why the plugin lists differ per language
Most plugins exist in both languages, and the ones that do not are settled by the same
rule: **which language a plugin lives in is decided by the framework it integrates, not by
preference.**
ElevenLabs, Deepgram, Cartesia and LiveKit are in both trees. [OpenClaw](/typescript-sdk/plugins/openclaw)
has to be TypeScript because it loads inside the OpenClaw gateway process and consumes in-process
objects there. Hermes Agent is Python for the same reason, in reverse.
[OpenAI Realtime](/typescript-sdk/plugins/openai) is a TypeScript plugin.
Nothing is missing from either side of the wire: both SDKs speak the same protocol, run the same
conformance vectors, and answer the same calls.
## What is in the package
| Source | Import specifier | How it is used |
| ------------------------ | ------------------------------- | ----------------------------------------------------------------------------------- |
| `src/` | `@komaa/standin-sdk` | the core: `CallServer`, the session types, audio helpers, the chat and vision lanes |
| `src/plugins/elevenlabs` | `@komaa/standin-sdk/elevenlabs` | an ElevenLabs agent takes the call, plus `standin-elevenlabs` |
| `src/plugins/deepgram` | `@komaa/standin-sdk/deepgram` | a Deepgram Voice Agent takes the call, plus `standin-deepgram` |
| `src/plugins/cartesia` | `@komaa/standin-sdk/cartesia` | a Cartesia Line agent takes the call, plus `standin-cartesia` |
| `src/plugins/openai` | `@komaa/standin-sdk/openai` | an OpenAI Realtime model takes the call, plus `standin-openai` |
| `src/plugins/livekit` | `@komaa/standin-sdk/livekit` | a LiveKit agent takes the call, plus `standin-livekit` |
| `src/plugins/openclaw` | `@komaa/standin-sdk/openclaw` | loaded by the OpenClaw gateway, not imported by you |
| `src/plugins/echo` | `@komaa/standin-sdk/echo` | the echo handler, plus the `standin-echo` command |
Directory names are short because the repository is already called `standin`. The published name
carries the scope because a registry is global. They are allowed to differ.
## Requirements
* Node.js 20 or newer for the core. The OpenClaw plugin follows its host and wants 22.19 or
newer, a requirement that travels with the `openclaw` peer dependency rather than with the core.
* A StandIn identity and its connection secret, from [standin.komaa.com](https://standin.komaa.com)
* A public `wss://` route to your worker, so StandIn can dial in
## The names you reach for first
Every name below resolves from the bare specifier. This is the handful a first worker needs, not the
whole surface:
```ts theme={null}
import {
CallServer,
type CallHandler,
type CallSession,
type HandlerFactory,
FrameAligner,
resamplePcm16,
SAMPLE_RATE_HZ,
REALTIME_SAMPLE_RATE_HZ,
FRAME_BYTES,
ChatChannel,
parseInbound,
buildReply,
signHandshake,
verifyHandshake,
setLogger,
StandInError,
VERSION,
} from "@komaa/standin-sdk";
```
## The whole surface, by lane
The barrel re-exports thirty-one modules. Nothing here is a subpath import: a plugin subpath is only ever
a framework adapter, and every name below comes from `@komaa/standin-sdk` itself.
| Lane | What is on the barrel | Page |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- |
| The call seam | `CallServer`, `CallServerOptions`, `CallHandler`, `CallSession`, `HandlerFactory`, `StandInError`, `setLogger`, `VERSION` | [Call server](/typescript-sdk/call-server), [Call handler](/typescript-sdk/call-handler) |
| Audio | `FrameAligner`, `resamplePcm16`, `frameDurationMs`, `pcm16Rms`, `FRAME_BYTES`, `FRAME_MS`, `SAMPLE_RATE_HZ`, `REALTIME_SAMPLE_RATE_HZ`, `BYTES_PER_SAMPLE`, `NUM_CHANNELS` | [Audio](/typescript-sdk/audio) |
| Turn-taking | `VoiceLane`, `UtteranceSegmenter`, `PacedPlayback`, `decodeWav`, `encodeWav`, the `TROUBLE_*` sentences | [Turn-taking](/typescript-sdk/voice) |
| Realtime providers | `StartupBuffer`, `MAX_PENDING_AUDIO`, `MAX_PENDING_CONTEXT`, `shouldSuppressEcho`, `ECHO_SUPPRESSION_WINDOW_MS`, `ECHO_BARGE_IN_RMS` | [Realtime providers](/typescript-sdk/realtime-providers) |
| Group calls | `GroupGate`, `isAddressed`, `isVerbalInterrupt`, `isMeetingThread` | [Group calls](/typescript-sdk/group-calls) |
| Vision | `FrameDescriber`, `parseVideoFrame`, `displayImage`, `displayFrame`, `VisionTools`, `KeyframeStore`, `VisionBudget`, `AmbientVision` | [Vision and the avatar](/typescript-sdk/vision) |
| The avatar | `expression`, `inferEmotion`, `speechMarks`, `ExpressionCue`, `EMOTIONS`, `TurnLipSync`, `estimateVisemes`, `visemesFromAlignment`, `TileStream`, `jpegEncoder` | [The avatar](/typescript-sdk/avatar) |
| Call tools | `CallTools`, `toolSchemas`, `BUILT_IN_TOOLS`, `SHOW_PAGE_TOOL` | [Call tools](/typescript-sdk/call-tools) |
| Consulting | `Consultant`, `BackgroundTasks`, `CONSULT_TOOL`, `BACKGROUND_TASK_TOOL` | [Consulting](/typescript-sdk/consulting) |
| Meeting recap | `Transcript`, `postMinutes`, `resolveMinutesTarget`, `minutesPrompt`, `writeMinutesDocx`, `MINUTES_TOOL` | [Meeting recap](/typescript-sdk/minutes) |
| Chat | `ChatChannel`, `parseInbound`, `buildReply`, `isPersonal`, `PersonalChats` | [Chat](/typescript-sdk/chat) |
| Attachments | `buildChatTurn`, `fetchChatImages`, `fetchChatAudio`, `transcribeVoiceMessages`, `spoolClip`, `chatImageDataUrl` | [Attachments in chat](/typescript-sdk/chat-attachments) |
| Sending a picture | `outboundImage`, `sniffImageType`, `sanitizeImageName`, `parseMedia`, `loadMedia`, `mediaRoots` | [Sending a picture](/typescript-sdk/sending-pictures) |
| Reaching people | `OutboundCaller`, `OutboundPolicy`, `OutboundLane`, `PendingMessages`, `VoiceDelivery`, `LiveCalls` | [Reaching people](/typescript-sdk/reaching-people) |
| Security | `signHandshake`, `verifyHandshake`, `signBody`, `verifyBody`, `signRequest`, `canonicalRequest`, `isInboundCallAllowed`, `normalizePhoneNumber`, `assertPublicHttpUrl`, `fetchPublicImage` | [Security](/typescript-sdk/security) |
| The wire itself | `SessionStart`, `Caller`, `parseMessage`, `parseSessionStart`, `decodePcm`, `contextSentences`, `audioFrame`, `assistantCancel`, `sessionEnd`, `pong` | [Security](/typescript-sdk/security#the-generated-wire-builders) |
| Configuration | `optional`, `required`, `flag`, `jsonObject`, `vendorHost` | [Configuration](/typescript-sdk/configuration) |
| Checking the install | `runSmoke`, `smokeReport`, `SyntheticCall` | [Checking the install](/typescript-sdk/checking-the-install) |
## Where the SDK stops
Three boundaries, and each is a real line rather than a disclaimer.
**Speech and reasoning are your framework's.** Speech recognition, speech generation and the model
that decides what to say belong to your provider. The SDK carries PCM in both directions and gives
you the turn-taking pieces if you are assembling three vendors yourself, but nothing in this package
transcribes, synthesizes or thinks.
**The Microsoft side is StandIn's.** StandIn handles it. Your worker never holds a Bot Framework credential and never talks to Microsoft.
**StandIn draws the avatar tile; the SDK sends it hints.** This is the boundary that is most often
described wrongly, so it is worth stating exactly. The SDK does carry the vision and avatar lane:
`session.express` names an emotion, `session.sendSpeechMarks` sends the viseme timeline that drives
lip-sync, `session.displayImage` puts a picture on the tile, `session.sendTileFrame` puts one frame of
your own video on it, and `session.latestVideoFrame` and `CallHandler.onVideoFrame` are how the
caller's camera and screen share reach you. Whole modules back those: `avatar`, `lipsync`, `vision`,
`visionTools`, `tile` and `ambient`, all on the barrel above. What the SDK does not do is render
anything. It never composites a face, never encodes the avatar and never knows how the tile is drawn.
See [Vision and the avatar](/typescript-sdk/vision) and [The avatar](/typescript-sdk/avatar).
A frame type this SDK does not recognise is ignored by contract, which is what lets an older worker
and a newer StandIn interoperate. That rule is about unknown frames, not about the avatar: those are
frames the SDK itself sends and parses.
## Next
From one install to a Microsoft Teams call that answers.
The seven methods and the session object.
Rates, frames, and the clipping trap.
The two HMAC lanes and their replay windows.
# Cartesia plugin
Source: https://docs.komaa.com/typescript-sdk/plugins/cartesia
Answer Microsoft Teams calls with a Cartesia Line agent, using the StandIn TypeScript SDK.
`@komaa/standin-sdk/cartesia` puts [Cartesia](https://docs.cartesia.ai/line) on a real Microsoft Teams call. StandIn answers the call and dials your worker; this plugin answers that dial, opens one session per call, and relays the audio both ways.
## Install and run
There is nothing to install beyond the SDK. Cartesia is reached over an ordinary WebSocket, so this plugin adds no dependency.
```bash theme={null}
npm install @komaa/standin-sdk
```
```bash theme={null}
export STANDIN_SECRET=... # your StandIn connection secret
export CARTESIA_API_KEY=...
export CARTESIA_AGENT_ID=...
npx standin-cartesia
```
Expose port `9442` at the `/msteams/calling` path and register the public `wss://` URL as your StandIn identity's agent voice URL. The full walkthrough is in the [Quickstart](/typescript-sdk/quickstart), and there is a runnable example at [examples/cartesia-msteams-connector](https://github.com/komaa-com/standin/tree/main/examples/cartesia-msteams-connector).
## The audio format
Audio is pinned to `pcm_16000` in both directions, which is exactly what a Microsoft Teams call carries.
## Your agent stays your agent
Cartesia runs the agent, so this plugin is transport and nothing else. There are no call tools to declare here, because whatever the agent does it does in its own code on Cartesia's platform.
Caller details reach that code as stream metadata, call context arrives as `custom` events, and a key press arrives as a real `dtmf` event.
`CARTESIA_SYSTEM_PROMPT` is optional on purpose. Left unset, the agent keeps exactly the prompt you wrote on Cartesia's side. Set it and the caller's details are appended to yours. Nothing here ever silently rewrites a deployed prompt.
## Inside your own worker
The plugin is a `CallHandler` like any other:
```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
import { CartesiaHandler, cartesiaConfigFromEnv } from "@komaa/standin-sdk/cartesia";
const config = cartesiaConfigFromEnv();
const server = new CallServer({
handlerFactory: () => new CartesiaHandler({ config }),
});
await server.start();
```
Read the configuration **once**, then close over it. `handlerFactory` runs once per call, so
`new CartesiaHandler()` with nothing passed reads the environment again on every call, and a key
removed after startup would fail the next caller rather than failing you. `serve()`, which is what
`npx standin-cartesia` runs, does exactly this, then waits for `SIGINT` or `SIGTERM` before calling
`server.aclose()`. `server.start()` returns as soon as the listener is bound, so a script that
ends there exits before a single call arrives, and `aclose()` is what drains live calls and
releases the port.
## Configuration
| Variable | What it is |
| ------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `CARTESIA_API_KEY` | Your API key. Used only to mint a per-call token over HTTPS, so the long-lived key never rides the agent socket. |
| `CARTESIA_AGENT_ID` | Which Line agent answers the call. |
| Variable | What it is |
| ------------------------ | ------------------------------------------------------------------------------------------------------- |
| `CARTESIA_VOICE_ID` | Override the agent's configured voice. |
| `CARTESIA_INTRODUCTION` | What the agent says first. |
| `CARTESIA_SYSTEM_PROMPT` | Appended to, never replacing, the prompt on Cartesia's side. Leave it unset and nothing is added. |
| `CARTESIA_VERSION` | API version header. Defaults to `2025-04-16`. Pin it when Cartesia ships a version you have not tested. |
| `CARTESIA_API_HOST` | Defaults to `api.cartesia.ai`. Must be a cartesia.ai host. |
The configuration is read once when the worker starts, not per call, so a missing key stops the worker at startup rather than surprising the first caller. `CARTESIA_API_HOST` is checked against the cartesia.ai suffix, because your API key travels to it: a wrong host would be credential leakage rather than a failed call. The reasoning behind that shape is on [Configuration](/typescript-sdk/configuration).
## Next
The startup buffer, the echo guard and barge-in that every provider plugin sits on.
Every `STANDIN_` variable, and the helpers for reading your own.
# Deepgram plugin
Source: https://docs.komaa.com/typescript-sdk/plugins/deepgram
Answer Microsoft Teams calls with a Deepgram Voice Agent, using the StandIn TypeScript SDK.
`@komaa/standin-sdk/deepgram` puts [Deepgram](https://developers.deepgram.com/docs/voice-agent) on a real Microsoft Teams call. StandIn answers the call and dials your worker; this plugin answers that dial, opens one session per call, and relays the audio both ways.
## Install and run
There is nothing to install beyond the SDK. Deepgram is reached over an ordinary WebSocket, so this plugin adds no dependency.
```bash theme={null}
npm install @komaa/standin-sdk
```
```bash theme={null}
export STANDIN_SECRET=... # your StandIn connection secret
export DEEPGRAM_API_KEY=...
npx standin-deepgram
```
Expose port `9442` at the `/msteams/calling` path and register the public `wss://` URL as your StandIn identity's agent voice URL. The full walkthrough is in the [Quickstart](/typescript-sdk/quickstart), and there is a runnable example at [examples/deepgram-msteams-connector](https://github.com/komaa-com/standin/tree/main/examples/deepgram-msteams-connector).
## The audio format
Audio is pinned to `linear16` at 16 kHz in both directions, which is exactly what a Microsoft Teams call carries, so the hot path is a copy.
## What the agent can do about the call
Declared automatically when the call starts, so there is nothing to set up on the provider's side.
| Tool | Parameters | What it does |
| ------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `end_call` | none | Hang up. |
| `express` | `emotion` | Set the avatar's expression on the bot's tile. See [The avatar](/typescript-sdk/avatar#the-emotion-the-face-wears). |
| `show_image` | `url`, `caption?`, `display?` | Put a picture on the bot's video tile. |
| `look` | `source?`, `question?` | Look at the caller's screen share or camera. |
| `look_back` | `question?` | Look again at something the caller has already moved past. Recorded calls only. |
`show_image` fetches the URL through the SDK's guard: public hosts only, no private or link-local addresses, and the address is re-checked at connect time. `display` is `fullscreen` or `overlay`, and this plugin routes the call through `CallTools.dispatch`, so the model's choice is honoured. See [Vision and the avatar](/typescript-sdk/vision) for what each placement is for, and [Sending a picture](/typescript-sdk/sending-pictures) for putting a picture into chat instead.
A Voice Agent hears but does not see, so `look` needs a vision model of your choosing. Set `STANDIN_VISION_API_URL` and `STANDIN_VISION_MODEL` to any OpenAI-compatible endpoint that accepts images, including one you run yourself. The frame is sent for inference and **not stored**, and only the description comes back. Without it the agent is told plainly that looking is unavailable, rather than being left with silence. The whole surface is on [Vision and the avatar](/typescript-sdk/vision).
Pass a `describer` in the handler options to point this plugin at a different vision model from the one the environment names, which is how one worker answers two identities with two vision budgets. Build it with `FrameDescriber.fromEnv()` or its constructor, both on [Vision and the avatar](/typescript-sdk/vision#giving-a-voice-model-eyes).
## How context reaches the agent
The Voice Agent API has no non-interrupting context message, so this plugin folds participant counts, key presses and recording changes into the agent's prompt and pushes it with an update.
That has a cost worth knowing: **the prompt is resent in full on every change**, so the context section is a rolling window of the last eight notes rather than a transcript. A meeting where people join and leave repeatedly keeps the recent notes and drops the old ones, which is the right trade when the alternative is a prompt that grows for the length of the call.
Context that arrives before the agent socket is open is held rather than dropped, then folded in once it exists. The "there are N people here, stay quiet" line and the recording change both land in exactly that gap.
## Tools of your own
A custom tool runs in **your** worker, so it can reach whatever your worker can reach:
```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
import {
DeepgramHandler,
deepgramConfigFromEnv,
type CustomTool,
} from "@komaa/standin-sdk/deepgram";
const tools: CustomTool[] = [
{
name: "open_ticket",
description: "Open a support ticket for the caller.",
parameters: {
type: "object",
properties: { summary: { type: "string" } },
required: ["summary"],
},
// ctx.call is the live call; ctx.recording says whether it is being recorded.
handler: async (params, ctx) => `opened ticket for ${String(params.summary)}`,
},
];
const config = deepgramConfigFromEnv();
const server = new CallServer({
handlerFactory: () => new DeepgramHandler({ config, tools }),
});
await server.start();
```
The `description` is what the model reads to decide whether to call it, so write it for a model rather than for a developer. Keep the handler fast: the caller is waiting in silence while it runs.
A custom tool may not shadow a built-in one. That is refused when the handler is built rather than at the first call, because a shadowed `end_call` is an agent that has quietly lost the ability to hang up.
## Inside your own worker
The plugin is a `CallHandler` like any other:
```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
import { DeepgramHandler, deepgramConfigFromEnv } from "@komaa/standin-sdk/deepgram";
const config = deepgramConfigFromEnv();
const server = new CallServer({
handlerFactory: () => new DeepgramHandler({ config }),
});
await server.start();
```
Read the configuration **once**, then close over it. `handlerFactory` runs once per call, so
`new DeepgramHandler()` with nothing passed reads the environment again on every call, and a key
removed after startup would fail the next caller rather than failing you. `serve()`, which is what
`npx standin-deepgram` runs, does exactly this, then waits for `SIGINT` or `SIGTERM` before calling
`server.aclose()`. `server.start()` returns as soon as the listener is bound, so a script that
ends there exits before a single call arrives, and `aclose()` is what drains live calls and
releases the port.
## Configuration
| Variable | What it is |
| ------------------ | -------------------------------------------------- |
| `DEEPGRAM_API_KEY` | Your API key. Never logged, never sent to StandIn. |
| Variable | What it is |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `DEEPGRAM_LISTEN_MODEL` | Speech to text. Defaults to `nova-3`. |
| `DEEPGRAM_SPEAK_MODEL` | Text to speech. Defaults to `aura-2-thalia-en`. |
| `DEEPGRAM_THINK_PROVIDER` | Which vendor reasons. Defaults to `open_ai`. |
| `DEEPGRAM_THINK_MODEL` | Defaults to `gpt-4o-mini`. |
| `DEEPGRAM_THINK_ENDPOINT_URL` | Reason with your own model instead. |
| `DEEPGRAM_THINK_ENDPOINT_HEADERS` | A JSON object of headers for that endpoint. Carries your model credentials, so it is never logged. Anything but a JSON object is refused when the configuration is read. |
| `DEEPGRAM_INSTRUCTIONS` | The agent's base prompt. |
| `DEEPGRAM_GREETING` | What the agent says first. |
| `DEEPGRAM_LANGUAGE` | Defaults to `en`. |
| `DEEPGRAM_AGENT_HOST` | The Voice Agent socket host. Defaults to `agent.deepgram.com`. Must be a deepgram.com host. |
| `DEEPGRAM_API_HOST` | The host the REST calls go to. Defaults to `api.deepgram.com`. Must be a deepgram.com host. |
| `DEEPGRAM_LOG_TRANSCRIPTS` | Exactly `true` to log turns. Gated a second time on the call being recorded. |
The think-endpoint pair is the reason most people pick this plugin: point `DEEPGRAM_THINK_ENDPOINT_URL` at a model of your own and Deepgram keeps the listening and the speaking while your model does the reasoning.
The configuration is read once when the worker starts, not per call, so a missing key stops the worker at startup rather than surprising the first caller. Both hosts are checked against the deepgram.com suffix, because your API key travels to them: a wrong host would be credential leakage rather than a failed call. The reasoning behind that shape is on [Configuration](/typescript-sdk/configuration).
## Next
The startup buffer, the echo guard and barge-in this plugin sits on.
What `look`, `show_image` and `express` reach.
# Echo plugin
Source: https://docs.komaa.com/typescript-sdk/plugins/echo
The smallest StandIn plugin that answers a real Microsoft Teams call. Copy it to start your own.
`@komaa/standin-sdk/echo` is the smallest thing that answers a real Microsoft Teams call. It sends
the caller's voice back, and it exists for two reasons: to be the thing you run **before** you
suspect your own agent, and to be the thing you copy when you start your own.
If the echo answers, your secret, your tunnel and your StandIn identity are all correct, and whatever
breaks next is your agent rather than the transport.
## Run it
Node.js 20 or newer:
```bash theme={null}
npm install @komaa/standin-sdk
STANDIN_SECRET="your-StandIn-connection-secret" npx standin-echo
```
Then expose port 9442 over HTTPS, register the public `wss://` URL with its `/msteams/calling` path as
your StandIn identity's agent voice URL, call the identity from Microsoft Teams, and talk. You should hear
yourself. The [quickstart](/typescript-sdk/quickstart) has the portal steps and
[Expose your agent](/expose) has the mount command and the probes.
## The whole plugin
```ts theme={null}
import { CallServer, type CallSession } from "@komaa/standin-sdk";
export class EchoHandler {
#call: CallSession | undefined;
async onStart(session: CallSession): Promise {
this.#call = session;
console.info(
`call ${session.callId} from ${session.start.caller.displayName ?? "unknown"}`,
);
}
async onCallerAudio(pcm: Buffer): Promise {
// Your agent goes here. PCM16, 16 kHz, mono, little-endian, the same format
// sendAudio expects back.
await this.#call?.sendAudio(pcm);
}
async onGoodbye(text: string): Promise {
// StandIn is ending the call and wants this line spoken first. A real plugin
// would interrupt the agent and say it.
console.info(`goodbye: ${text}`);
}
}
export async function serve(): Promise {
const server = new CallServer({ handlerFactory: () => new EchoHandler() });
await server.start();
await new Promise((resolve) => {
const stop = (): void => resolve();
process.once("SIGINT", stop);
process.once("SIGTERM", stop);
});
await server.aclose();
}
```
That is the whole plugin, minus the CLI wrapper behind `standin-echo`, the licence header and the
doc comments. One line differs from the file in the repository on purpose: that copy reaches the core
by relative path, because it lives inside the package, and yours reaches it by package name. Note
what is **not** there: no sequence number, no timestamp, no handshake, no capacity check, no watchdog. All of that
is [`CallServer`](/typescript-sdk/call-server).
Note also that this handler defines three of the seven methods on `CallHandler`. `onVideoFrame`,
`onSpeakerChange`, `onContext` and `aclose` are absent because it does not need them, and the server
treats a missing method as a no-op: the interface is all-optional, so nothing extends anything. See
[Call handler](/typescript-sdk/call-handler#the-seven-methods).
## Make it yours
Replace `onCallerAudio` with your framework's agent loop. Everything else stays.
```ts theme={null}
async onCallerAudio(pcm: Buffer): Promise {
const reply = await myFramework.respond(pcm);
await this.#call?.sendAudio(reply);
}
```
If your framework speaks at 24 kHz, read [Audio](/typescript-sdk/audio) before you write a resampler.
The SDK ships one, along with the frame aligner that stops every turn losing its last few
milliseconds.
## Write your own handler
You do not need the SDK's source, a fork or a registration to write a handler. Copy the class out of
the block above into your own module, change `onCallerAudio`, and pass it as `handlerFactory`:
```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
const server = new CallServer({ handlerFactory: () => new MyHandler() });
await server.start();
```
There is nothing to declare anywhere. `CallHandler` is an interface of optional methods, so your
class is a handler because it has the right method names, not because it extends or registers. That
is the whole seam, and it is the same one every plugin on this site uses.
## What to add first
Once the echo answers, three things turn the template into something usable:
**Barge-in.** `await session.cancelPlayback()` the moment your provider reports the caller started
speaking. Without it the bot keeps talking over the caller for the length of the buffered audio. See
[`cancelPlayback()` is the barge-in](/typescript-sdk/call-handler#cancelplayback-is-the-barge-in).
**Frame alignment.** If your provider speaks 24 kHz, resample and align before sending. See
[Audio](/typescript-sdk/audio).
**Cleanup.** Add `aclose(reason)` to close your provider socket. It is always called exactly once, on
every path, before the slot is freed.
## Contribute a plugin to the SDK
This part is different: it is for adding a plugin to the `@komaa/standin-sdk` package itself, so it
assumes a clone of the repository rather than an `npm install`. If you are building for your own
deployment, the two sections above are the ones you want.
Every plugin lives inside the one package, so adding one publishes nothing new.
```bash theme={null}
cp -r libraries/typescript/src/plugins/echo libraries/typescript/src/plugins/
```
One entry under `exports` in `libraries/typescript/package.json`, pointing
`@komaa/standin-sdk/` at the emitted `dist/plugins//index.js`. The directory name stays
short; the published specifier carries the scope, because a registry is global.
The copied directory brings a `cli.ts` with it, so add the matching `bin` entry in the same file,
`standin-` pointing at `dist/plugins//cli.js`, or `npx standin-` resolves to
nothing and the install line on your page is wrong from the first day.
A framework the plugin needs is a **peer** dependency, marked optional in `peerDependenciesMeta`.
The core imports nothing from `plugins/`, so somebody installing the SDK for one plugin never drags
in another's framework.
```bash theme={null}
make ts-check
```
`make py-check` is its Python counterpart, and `make check` runs both halves plus the protocol and
documentation checks. Then open a pull request against
[`komaa-com/standin`](https://github.com/komaa-com/standin).
Write a plugin when a framework requires its own plugin registry, private internals, or worker
lifecycle hooks. If the difference is only configuration, prefer a preset or a documentation page.
A plugin needs a named maintainer who will retest it for upstream minor versions.
## Parity
The Python twin of this file is `standin.plugins.echo`, the same handler with the same method
names in that language's casing. Core SDK features land in both languages in the same change, with
matching defaults and behaviour. The full guide is
[CONTRIBUTING.md](https://github.com/komaa-com/standin/blob/main/CONTRIBUTING.md).
## Next
All seven methods, with a full worked handler.
Every option you may want to change once calls are real.
# ElevenLabs plugin
Source: https://docs.komaa.com/typescript-sdk/plugins/elevenlabs
Answer Microsoft Teams calls with an ElevenLabs agent, using the StandIn TypeScript SDK.
`@komaa/standin-sdk/elevenlabs` puts [ElevenLabs](https://elevenlabs.io/docs/agents-platform/overview) on a real Microsoft Teams call. StandIn answers the call and dials your worker; this plugin answers that dial, opens one session per call, and relays the audio both ways.
## Install and run
There is nothing to install beyond the SDK. ElevenLabs is reached over an ordinary WebSocket, so this plugin adds no dependency.
```bash theme={null}
npm install @komaa/standin-sdk
```
```bash theme={null}
export STANDIN_SECRET=... # your StandIn connection secret
export ELEVENLABS_API_KEY=...
export ELEVENLABS_AGENT_ID=...
npx standin-elevenlabs
```
Expose port `9442` at the `/msteams/calling` path and register the public `wss://` URL as your StandIn identity's agent voice URL. The full walkthrough is in the [Quickstart](/typescript-sdk/quickstart), and there is a runnable example at [examples/elevenlabs-msteams-connector](https://github.com/komaa-com/standin/tree/main/examples/elevenlabs-msteams-connector).
## The audio format
In the ElevenLabs dashboard, set the agent's input **and** output audio format to `pcm_16000`. An agent set to anything else is refused at the first frame rather than producing a whole call of garbled audio.
## What the agent can do about the call
Declare these as **client tools** on the agent. Nothing to implement: the plugin answers them.
`clientTools()` returns the declarations, so there is nothing to retype into the dashboard:
```ts theme={null}
import { clientTools } from "@komaa/standin-sdk/elevenlabs";
console.log(JSON.stringify(clientTools(), null, 2));
```
| Tool | Parameters | What it does |
| ------------ | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `end_call` | none | Hang up. |
| `express` | `emotion` | Set the avatar's expression on the bot's tile. See [The avatar](/typescript-sdk/avatar#the-emotion-the-face-wears). |
| `show_image` | `url` **or** `dataBase64` + `mime`, plus `caption?` and `durationMs?` | Put a picture on the bot's video tile. |
| `look` | `source?`, `question?` | Look at the caller's screen share or camera. |
| `look_back` | `question?` | Look again at something the caller has already moved past. Recorded calls only. |
`show_image` is widened here, because ElevenLabs is the one provider that can hand you the bytes inline: either form will do, so neither argument is required. A URL is chosen by the model, which the caller steers, so it is fetched through the SDK's guard: public hosts only, no private or link-local addresses, and the address is re-checked at connect time to close the DNS rebind window.
**The fullscreen or overlay choice is not selectable on this plugin.** The other provider plugins route `show_image` through `CallTools.dispatch`, which honours a `display` parameter. This one has its own dispatch, and it reads `mode`. `clientTools()` still emits `display` with its `fullscreen` / `overlay` enum, because that comes from the shared tool spec, so the declaration you paste will offer the model a choice that is then dropped in silence. Deleting the `display` property from the pasted declaration changes nothing except that the model stops being offered it, which is the honest version. Either way the picture goes up with the default placement. [Fullscreen or beside your face](/typescript-sdk/vision#fullscreen-or-beside-your-face) covers what the two placements are for.
`look` and `look_back` upload the frame into the ElevenLabs conversation, which **stores** the caller's screen with a third party. They therefore work only while the Microsoft Teams call is being recorded. When it is not, the agent is told why rather than being left with silence.
## Inside your own worker
The plugin is a `CallHandler` like any other:
```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
import {
ElevenLabsHandler,
elevenLabsConfigFromEnv,
} from "@komaa/standin-sdk/elevenlabs";
const config = elevenLabsConfigFromEnv();
const server = new CallServer({
handlerFactory: () => new ElevenLabsHandler(config),
});
await server.start();
```
This handler takes its configuration as a **positional** argument, where the other provider plugins
take an options object. Read it **once**, then close over it: `handlerFactory` runs once per call, so
`new ElevenLabsHandler()` with nothing passed reads the environment again on every call, and a key
removed after startup would fail the next caller rather than failing you. `serve()`, which is what
`npx standin-elevenlabs` runs, does exactly this, then waits for `SIGINT` or `SIGTERM` before calling
`server.aclose()`. `server.start()` returns as soon as the listener is bound, so a script that
ends there exits before a single call arrives, and `aclose()` is what drains live calls and
releases the port.
## Configuration
| Variable | What it is |
| --------------------- | -------------------------------------------------- |
| `ELEVENLABS_API_KEY` | Your API key. Never logged, never sent to StandIn. |
| `ELEVENLABS_AGENT_ID` | Which agent answers the call. |
| Variable | What it is |
| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ELEVENLABS_FIRST_MESSAGE` | Override the opening line. Applied only when the agent's security settings allow the override. |
| `ELEVENLABS_ENVIRONMENT` | For an agent deployed to a named environment. |
| `ELEVENLABS_AGENT_BRANCH_ID` | Answer with a specific branch. |
| `ELEVENLABS_HOST` | Defaults to `api.elevenlabs.io`. Must be an elevenlabs.io host. |
| `ELEVENLABS_LOG_TRANSCRIPTS` | Exactly `true` to log turns. Gated a second time on the call being recorded: a transcript in your logs is a recording of the caller that they did not agree to. |
The configuration is read once when the worker starts, not per call, so a missing key stops the worker at startup rather than surprising the first caller. `ELEVENLABS_HOST` is checked against the elevenlabs.io suffix, because your API key and agent id travel to it: a wrong host would be credential leakage rather than a failed call. The reasoning behind that shape is on [Configuration](/typescript-sdk/configuration).
## Next
The startup buffer, the echo guard and barge-in this plugin sits on.
What `look`, `show_image` and `express` reach.
# LiveKit plugin
Source: https://docs.komaa.com/typescript-sdk/plugins/livekit
Answer Microsoft Teams calls with a LiveKit agent, using the StandIn TypeScript SDK.
`@komaa/standin-sdk/livekit` puts a [LiveKit agent](https://docs.livekit.io/agents/) on a real Microsoft Teams call. StandIn answers the call and dials your worker; this plugin answers that dial, creates one room per call, dispatches your agent into it, and relays the audio both ways.
The shape is the opposite of the provider plugins. Those are a socket you send audio to. LiveKit is a room you **join**: the caller is published as a track, and the agent's audio is subscribed to. Your agent therefore needs no Microsoft Teams awareness at all. It sees a participant who talks.
## Install and run
This is the one plugin with dependencies of its own, because joining a room means running LiveKit's client.
```bash theme={null}
npm install @komaa/standin-sdk @livekit/rtc-node livekit-server-sdk
```
```bash theme={null}
export STANDIN_SECRET=... # your StandIn connection secret
export LIVEKIT_URL=wss://...
export LIVEKIT_API_KEY=...
export LIVEKIT_API_SECRET=...
export LIVEKIT_AGENT_NAME=standin-msteams
npx standin-livekit
```
Both packages are **optional peers**: they are imported only when this module is reached, so the core and every other plugin install without them. A missing one raises an error naming the install that fixes it, rather than a bare module-not-found from inside somebody else's package.
Expose port `9442` at the `/msteams/calling` path and register the public `wss://` URL as your StandIn identity's agent voice URL. [Expose your agent](/expose) carries the mount command and the probes, and there is a runnable example, in both languages, at [examples/livekit-msteams-connector](https://github.com/komaa-com/standin/tree/main/examples/livekit-msteams-connector).
## What your agent receives
Job metadata: `source`, `call_id`, `call_direction`, `caller_name`, `user_id`, `tenant_id` and `thread_id`.
Room data on two topics: `msteams.context` for participant counts, key presses and recording changes, and `msteams.goodbye` for the closing line.
Those key names are a **cross-language contract**, not a local choice, and so are the two topic names. `source` is always `"msteams"`, and it is what Python's [`CallInfo.from_job`](/python-sdk/plugins/livekit#callinfo) matches on: a job whose metadata is missing, unparseable or carries any other `source` reads as "not a Microsoft Teams call" and returns a blank record, rather than failing the worker. So an agent dispatched from this plugin and written in Python sees the caller without a line of translation, and an agent dispatched by something else is not mistaken for a caller.
A field StandIn does not know stays **absent** here rather than becoming a shared default, so two anonymous callers never look like one person.
There is no `CallInfo` helper on this side, and no entrypoint hook like the Python plugin's `TeamsCall().start(...)`, because there is no LiveKit worker in this process to hook into: your agent is a separate worker that this one dispatches into a room, so it reads the job metadata and subscribes to the two topics with LiveKit's own API. In Python the agent and the plugin share a worker, which is what makes that helper possible there.
`user_id` is the caller's AAD object id, and it is **empty for guest and anonymous callers** in both SDKs, so it is omitted here. Never use it as a bare key for per-caller memory without checking it first.
**The Python plugin fills an unknown field differently.** It always writes `caller_name`, `tenant_id` and `thread_id`, substituting `"caller"` and `"unknown-tenant"` when StandIn does not know them, where this plugin omits the key. An agent that reads `job.metadata` directly sees `undefined` under this plugin and a literal `"caller"` under the Python one, and one that reads through `CallInfo` sees an empty string here against `"caller"` there. Either way, test for a real name rather than for the key being present.
## Dispatch
Set `LIVEKIT_AGENT_NAME` to the name your worker registered with and each call explicitly dispatches that agent into its room. Leave it unset and the room creation itself assigns the job, which is LiveKit's automatic dispatch. Explicit is the recommended setup: a room that is created with nobody dispatched into it is a caller listening to silence.
**The metadata above rides the explicit dispatch.** This plugin attaches it to the `RoomAgentDispatch` it creates, so an agent picked up by automatic dispatch is handed no call metadata at all and sees no caller. The Python plugin additionally writes the same record as room metadata for exactly that path, so this is a real difference between the two, not a documentation gap. Set `LIVEKIT_AGENT_NAME` whenever your agent needs to know who is calling.
## Rooms
Room names are `{LIVEKIT_ROOM_PREFIX}{callId}`, defaulting to `msteams-`, with the `callId` reduced to a conservative charset and the whole name capped at 100 characters. That derivation is a contract rather than a detail, so a room created for a given call is the same room whichever shipped plugin created it, and the Python plugin computes it identically.
The call id is sanitised before it becomes a room name: it arrives from a decoded URL segment, where a `%2F` would otherwise smuggle a slash into the name.
The prefix is one of the few places the two SDKs genuinely differ in **how** it is set: `LIVEKIT_ROOM_PREFIX` here, and the `room_prefix` constructor argument on the Python plugin's `TeamsCallHandler`, which reads no environment variable for it.
At teardown the plugin disconnects and deletes the room, so the dispatched agent job ends at once instead of idling out: a job whose room still exists sits there until LiveKit's own empty-room timeout, which is minutes of a worker slot doing nothing. Set `LIVEKIT_DELETE_ROOM=off` to keep the room instead.
## The avatar tile
If your agent publishes video of its own, an avatar worker or a rendered face, it is relayed onto the bot's video tile automatically and **on by default**. The caller sees your agent rather than StandIn's avatar. When it does not publish video, [`express` and `sendSpeechMarks`](/typescript-sdk/avatar) still drive StandIn's own avatar, and the caller hears the agent either way.
`LIVEKIT_TILE_VIDEO` takes three kinds of value, and the third is the one worth knowing about:
| Value | What happens |
| --------------- | ---------------------------------------------------- |
| unset or `auto` | Relay whichever participant's video the room offers. |
| `off` | No relay. The caller sees StandIn's avatar. |
| anything else | A participant **identity** to pin the relay to. |
Pin an identity when a separate worker publishes the avatar and LiveKit's publish-on-behalf attribute is not set: without a name the relay takes whichever participant published first, which on a busy room is the wrong one. With the attribute set there is nothing to pin, because that is the first thing the plugin looks at, falling back to the agent's own identity.
Tracks are matched by kind, never by source. An avatar worker publishes untagged video, so a source filter would pick the right participant and then stream nothing, which is the most expensive way to get this wrong.
Frames are paced, latest-wins, and dropped rather than queued when the socket is congested, because both streams share one socket and a caller forgives a dropped frame far more readily than a break in the voice. `LIVEKIT_TILE_VIDEO_FPS` defaults to 12 and is clamped to `MAX_TILE_FPS`, which is 20: a talking head gains nothing above that, and a higher rate only spends local CPU on encoding. A value that is not a whole number above zero is refused when the configuration is read, naming the variable: an operator who typed `twelve` is looking at a setting that is not the one in force, and silence is what makes that take an afternoon to find.
Encoding needs the optional `sharp` peer: `npm install sharp`. Without it the relay stays off with one line in the log and **the call is unaffected**. The Python plugin's equivalent is the `tile` extra, `pip install "standin-sdk[tile]"`.
## Staying out of a meeting
A 1:1 call is simple: every turn is for your agent. A meeting is not. `wakePhrases` arms the SDK's group gate from the call's own thread:
```ts theme={null}
new LiveKitHandler({ wakePhrases: ["assistant", "standin"] });
```
It is **empty by default**, which leaves the gate inert, so an agent that answers everything keeps answering everything and nothing changes for it. `handler.lastDecision` reports what the gate decided about the caller's last finished turn, and is `undefined` until a turn has been decided and on a call where transcripts never arrive. [Group calls](/typescript-sdk/group-calls) is the page for what the gate decides and why a partial transcript may notice a wake phrase but never decide a turn.
## Inside your own worker
```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
import { LiveKitHandler, liveKitConfigFromEnv } from "@komaa/standin-sdk/livekit";
const config = liveKitConfigFromEnv();
const server = new CallServer({
handlerFactory: () => new LiveKitHandler({ config }),
});
await server.start();
```
Read the configuration **once**, then close over it. `handlerFactory` runs once per call, so `new LiveKitHandler()` with nothing passed reads the environment again on every call, and a variable removed after startup would fail the next caller rather than failing you. `serve()`, which is what `npx standin-livekit` runs, does exactly this, then waits for `SIGINT` or `SIGTERM` before calling
`server.aclose()`. `server.start()` returns as soon as the listener is bound, so a script that
ends there exits before a single call arrives, and `aclose()` is what drains live calls and
releases the port.
## Draining and shutdown
This plugin is an ordinary `CallHandler`, so a deployment drains the way every other TypeScript worker does: set `server.draining = true` and new dials are refused with 503 while live calls continue. See [CallServer](/typescript-sdk/call-server#draining).
That is the one real difference from the Python plugin, which arms itself inside a LiveKit `AgentServer` and can therefore hook the worker's own drain. There is no worker lifecycle to hook here, because there is no LiveKit worker in this process: your agent is a separate worker that this one dispatches into a room.
## Writing the agent in Python instead
[`standin.plugins.livekit`](/python-sdk/plugins/livekit) does the same thing and additionally arms itself from inside your worker, so your agent file keeps the shape of every other LiveKit example and nothing starts except through `cli.run_app(server)`.
## Configuration
| Variable | Default | What it is |
| ------------------------ | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `STANDIN_SECRET` | *(required)* | Connection secret from the StandIn portal. |
| `STANDIN_PORT` | `9442` | Port the call listener binds. |
| `STANDIN_HOST` | `0.0.0.0` | Bind address. Use `127.0.0.1` when only a local tunnel should reach it. |
| `STANDIN_WS_PATH` | `/msteams/calling` | Path StandIn dials. |
| `LIVEKIT_URL` | *(required)* | Your project's `wss://` URL. |
| `LIVEKIT_API_KEY` | *(required)* | Project key. |
| `LIVEKIT_API_SECRET` | *(required)* | Project secret. Never logged, never sent to StandIn. |
| `LIVEKIT_AGENT_NAME` | *(unset)* | The name your worker registered with. Unset means automatic dispatch. |
| `LIVEKIT_ROOM_PREFIX` | `msteams-` | Prepended to the call id to name the room. |
| `LIVEKIT_DELETE_ROOM` | *(on)* | Set to `off` to keep the room at teardown instead of deleting it. |
| `LIVEKIT_TILE_VIDEO` | `auto` | Relay your agent's own video onto the bot's tile. See [The avatar tile](#the-avatar-tile). |
| `LIVEKIT_TILE_VIDEO_FPS` | `12` | Frames per second for that relay. Clamped to `MAX_TILE_FPS`, which is 20. A value that is not a whole number above zero is refused, naming the variable. |
A missing required variable is caught when the configuration is read, with one error naming the variable, rather than on the first real call with a caller already on the line.
## Next
The rest of the avatar surface: expression, visemes and your own frames on the tile.
What the wake phrase gate decides, and what stops playback.
# OpenAI Realtime plugin
Source: https://docs.komaa.com/typescript-sdk/plugins/openai
Answer Microsoft Teams calls with an OpenAI Realtime model, using the StandIn TypeScript SDK.
`@komaa/standin-sdk/openai` puts [OpenAI Realtime](https://platform.openai.com/docs/guides/realtime) on a real Microsoft Teams call. StandIn answers the call and dials your worker; this plugin answers that dial, opens one session per call, and relays the audio both ways.
## Install and run
There is nothing to install beyond the SDK. OpenAI Realtime is reached over an ordinary WebSocket, so this plugin adds no dependency.
```bash theme={null}
npm install @komaa/standin-sdk
```
```bash theme={null}
export STANDIN_SECRET=... # your StandIn connection secret
export OPENAI_API_KEY=...
npx standin-openai
```
Expose port `9442` at the `/msteams/calling` path and register the public `wss://` URL as your StandIn identity's agent voice URL. The full walkthrough is in the [Quickstart](/typescript-sdk/quickstart), and there is a runnable example at [examples/openai-msteams-connector](https://github.com/komaa-com/standin/tree/main/examples/openai-msteams-connector).
## The audio format
The Realtime API speaks PCM at 24 kHz and a Microsoft Teams call speaks 16 kHz. This plugin owns that conversion in both directions, using the same resampler the Python SDK uses, driven by the same shared conformance vectors. You never see it.
`OPENAI_VAD_TYPE` decides when the model thinks you have finished: `semantic_vad`, the default, listens for a finished thought, and `server_vad` listens for silence. Semantic interrupts less often mid-sentence. Any other value is refused when the configuration is read, naming the variable, rather than quietly falling back.
## What the agent can do about the call
Declared automatically when the call starts, so there is nothing to set up on the provider's side.
| Tool | Parameters | What it does |
| ------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `end_call` | none | Hang up. |
| `express` | `emotion` | Set the avatar's expression on the bot's tile. See [The avatar](/typescript-sdk/avatar#the-emotion-the-face-wears). |
| `show_image` | `url`, `caption?`, `display?` | Put a picture on the bot's video tile. |
| `look` | `source?`, `question?` | Look at the caller's screen share or camera. |
| `look_back` | `question?` | Look again at something the caller has already moved past. Recorded calls only. |
`show_image` fetches the URL through the SDK's guard: public hosts only, no private or link-local addresses, and the address is re-checked at connect time. `display` is `fullscreen` or `overlay`, and this plugin routes the call through `CallTools.dispatch`, so the model's choice is honoured. See [Vision and the avatar](/typescript-sdk/vision) for what each placement is for, and [Sending a picture](/typescript-sdk/sending-pictures) for putting a picture into chat instead.
The Realtime model hears but does not see, so `look` needs a vision model of your choosing. Set `STANDIN_VISION_API_URL` and `STANDIN_VISION_MODEL` to any OpenAI-compatible endpoint that accepts images, including one you run yourself. The frame is sent for inference and **not stored**, and only the description comes back. Without it the agent is told plainly that looking is unavailable, rather than being left with silence. The whole surface is on [Vision and the avatar](/typescript-sdk/vision).
Pass a `describer` in the handler options to point this plugin at a different vision model from the one the environment names, which is how one worker answers two identities with two vision budgets. Build it with `FrameDescriber.fromEnv()` or its constructor, both on [Giving a voice model eyes](/typescript-sdk/vision#giving-a-voice-model-eyes).
## Tools of your own
A custom tool runs in **your** worker, so it can reach whatever your worker can reach:
```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
import {
OpenAIHandler,
openAIConfigFromEnv,
type CustomTool,
} from "@komaa/standin-sdk/openai";
const tools: CustomTool[] = [
{
name: "open_ticket",
description: "Open a support ticket for the caller.",
parameters: {
type: "object",
properties: { summary: { type: "string" } },
required: ["summary"],
},
// ctx.call is the live call; ctx.recording says whether it is being recorded.
handler: async (params, ctx) => `opened ticket for ${String(params.summary)}`,
},
];
const config = openAIConfigFromEnv();
const server = new CallServer({
handlerFactory: () => new OpenAIHandler({ config, tools }),
});
await server.start();
```
The `description` is what the model reads to decide whether to call it, so write it for a model rather than for a developer. Keep the handler fast: the caller is waiting in silence while it runs.
A custom tool may not shadow a built-in one. That is refused when the handler is built rather than at the first call, because a shadowed `end_call` is an agent that has quietly lost the ability to hang up.
### A remote MCP server
An `mcpTools` entry is dialled by OpenAI itself, so nothing runs in your worker at call time:
```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
import { OpenAIHandler, mcpTool, openAIConfigFromEnv } from "@komaa/standin-sdk/openai";
const config = openAIConfigFromEnv();
const mcpTools = [
mcpTool({ server_label: "docs", server_url: "https://mcp.example.com" }),
];
const server = new CallServer({
handlerFactory: () => new OpenAIHandler({ config, mcpTools }),
});
await server.start();
```
`mcpTool` requires `server_label` and `server_url` and throws naming them if either is missing, so a typo stops the worker at startup rather than producing a session whose tool list is quietly one short.
Approval defaults to never, because a live voice call has no approval interface and a pending approval is a caller listening to silence. Pass `require_approval` in the entry yourself to override it: the defaults are spread first, so anything you set wins.
## Inside your own worker
The plugin is a `CallHandler` like any other:
```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
import { OpenAIHandler, openAIConfigFromEnv } from "@komaa/standin-sdk/openai";
const config = openAIConfigFromEnv();
const server = new CallServer({
handlerFactory: () => new OpenAIHandler({ config }),
});
await server.start();
```
Read the configuration **once**, then close over it. `handlerFactory` runs once per call, so
`new OpenAIHandler()` with nothing passed reads the environment again on every call, and a key
removed after startup would fail the next caller rather than failing you. `serve()`, which is what
`npx standin-openai` runs, does exactly this, then waits for `SIGINT` or `SIGTERM` before calling
`server.aclose()`. `server.start()` returns as soon as the listener is bound, so a script that
ends there exits before a single call arrives, and `aclose()` is what drains live calls and
releases the port.
## Configuration
| Variable | Default | What it is |
| ---------------- | ------------ | -------------------------------------------------- |
| `OPENAI_API_KEY` | *(required)* | Your API key. Never logged, never sent to StandIn. |
| Variable | Default | What it is |
| ---------------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `OPENAI_REALTIME_MODEL` | `gpt-realtime` | Which Realtime model answers. |
| `OPENAI_REALTIME_HOST` | `api.openai.com` | Must end `.openai.com` or `.azure.com`. The Azure allowance is what makes an Azure OpenAI deployment work here at all. |
| `OPENAI_VOICE` | *(the model's own)* | Override the voice. |
| `OPENAI_INSTRUCTIONS` | a short spoken-reply prompt | The agent's base prompt. Exported as `DEFAULT_INSTRUCTIONS` if you want to extend rather than replace it. |
| `OPENAI_VAD_TYPE` | `semantic_vad` | `server_vad` or `semantic_vad`. Anything else is refused when the configuration is read. |
| `OPENAI_TRANSCRIPTION_MODEL` | *(unset)* | Ask for transcripts alongside the audio. |
| `OPENAI_LOG_TRANSCRIPTS` | *(off)* | Exactly `true` to log turns. Gated a second time on the call being recorded. |
The configuration is read once when the worker starts, not per call, so a missing key stops the worker at startup rather than surprising the first caller. `OPENAI_REALTIME_HOST` is checked against those two suffixes, because your API key travels to it: a wrong host would be credential leakage rather than a failed call. The reasoning behind that shape is on [Configuration](/typescript-sdk/configuration).
## Next
The startup buffer, the echo guard and barge-in this plugin sits on.
What `look`, `show_image` and `express` reach.
# OpenClaw plugin
Source: https://docs.komaa.com/typescript-sdk/plugins/openclaw
Connect the realtime voice provider configured in OpenClaw to Microsoft Teams calls through StandIn.
`@komaa/standin-sdk/openclaw` connects the realtime voice provider configured in
[OpenClaw](https://openclaw.ai) to Microsoft Teams calls through StandIn.
**Scope.** This plugin covers the call path and the voice: a caller converses with the realtime
provider OpenClaw is configured with, with barge-in, an echo guard and a recording gate. OpenClaw's
agent consultation, tools, skills and memory are reached from chat today, not from the realtime
session this plugin drives. [Scope](#scope) has the whole boundary in one list.
## This is a plugin, not a worker you run
There is no `node app.js` here, and there is not supposed to be. The plugin loads **inside** the
OpenClaw gateway process, declared by its `openclaw.plugin.json`, because it consumes the host's
realtime speech-to-speech session, its provider registry and its logger, all of which are in-process
objects.
OpenClaw's external HTTP surface is text-only, so a worker that only called that HTTP surface could only use OpenClaw as a
text brain and bring its own STT and TTS, which is exactly the part this plugin exists to avoid.
The plugin registers one host-managed background service, so OpenClaw's own lifecycle starts the call
listener at boot and tears it down on shutdown or reload. There is nothing to supervise separately.
## Install
OpenClaw 2026.6.10 or newer. Install the SDK where your gateway can reach it:
```bash theme={null}
npm install @komaa/standin-sdk
```
`openclaw` is an **optional peer** dependency, never a bundled one: the plugin runs inside the
gateway and consumes the host's own realtime session, provider registry and logger.
OpenClaw does not load a plugin by package specifier. It is handed a **directory** through
`plugins.load.paths` and discovers what is inside it, so point it at the one inside the package:
```json theme={null}
{ "plugins": { "load": { "paths": [
"node_modules/@komaa/standin-sdk/dist/plugins/openclaw"
] } } }
```
## Configure and restart
Merge this into `~/.openclaw/openclaw.json`, keeping the `plugins.load.paths` entry above:
```json theme={null}
{
"plugins": {
"entries": {
"standin-msteams": {
"enabled": true,
"config": {
"secret": "sk_standin_REPLACE_ME",
"inboundPolicy": "allowlist",
"allowFrom": [
"00000000-0000-0000-0000-000000000000"
],
"inboundGreeting": "Hi, this is your assistant. How can I help?",
"callingPort": 9442,
"bindAddress": "127.0.0.1",
"path": "/msteams/calling",
"maxConcurrentCalls": 4,
"requireRecordingStatus": false,
"sessionScope": "per-call",
"meetingRecap": false,
"realtime": {
"provider": "openai",
"providers": {
"openai": {
"apiKey": "sk-REPLACE_ME",
"voice": "alloy"
}
},
"instructions": "You are on a Microsoft Teams call. Keep answers short and spoken."
}
}
}
}
}
}
```
Then restart the OpenClaw gateway, so the plugin's background service starts the call listener:
```bash theme={null}
openclaw gateway
```
Now expose port `9442` at the `/msteams/calling` path, and register the public `wss://` URL as your
StandIn identity's agent voice URL. [Expose your agent](/expose) carries the mount command and the
probes, and it is the only page that does: three spellings of one URL across three pages is what
caused a live incident here.
Call your StandIn number.
## Configuration reference
| Key | Meaning |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `enabled` | `false` turns the plugin off without uninstalling it. Anything else, including omitting it, leaves it on. |
| `secret` | The StandIn connection secret. Without it nothing starts. |
| `callingPort` / `bindAddress` / `path` | Where the call listener binds. Defaults `9442`, `0.0.0.0`, `/msteams/calling`. |
| `maxConcurrentCalls` | The operator's cap, default 4. Call 5 is refused with `busy`. |
| `inboundPolicy` / `allowFrom` | Who may call. **Defaults to `disabled`, which refuses everyone.** |
| `inboundGreeting` | Spoken on pickup. Omit it and the agent waits for the caller to speak first. |
| `requireRecordingStatus` | Hold caller media until Microsoft Teams reports recording active. |
| `sessionScope` | Memory of the agent session that writes the minutes: `per-call` (default), `per-thread`, or `per-aad`. The voice session is always per call. |
| `meetingRecap` | After the call ends, post minutes into the Microsoft Teams chat. Off unless exactly `true`. Needs the StandIn chat lane ([Tiers](/tiers#the-standin-ai-teammate-app)) and a summarization consult. Best-effort: hang-up does not await the post. |
| `realtime.provider` / `realtime.providers` | Provider selection and its credentials. |
| `realtime.instructions` | The system prompt for the call. |
| `realtime.suppressInputDuringPlayback` | Echo guard on or off. On by default. |
| `realtime.echoSuppressionWindowMs` | How long after our own audio to keep guarding. Default 600. |
| `realtime.echoBargeInRms` | How loud caller input must be to break through the guard. Default 0.04. |
The plugin fails closed on the secret. A non-string value coerces to empty rather than to
`"[object Object]"`, so an unresolved secret reference stops the listener starting instead of arming
it with a guessable value.
## Who may call
`inboundPolicy` takes `disabled`, `allowlist`, `pairing` or `open`, and an unset or unknown policy
**refuses**. Defaulting the other way would mean a config typo silently opens your agent to anyone who
can reach the number.
`allowFrom` takes the caller's **AAD object id**, not their email. Copy it from
`session.start.caller.aadId` on an inbound call (a refused call still logs it). Emails never match.
It matches on exact id, case-insensitive, or on digits-only for a phone number. The id match is what
lets a Microsoft Teams caller be allowlisted at all: their id is an AAD object id, and phone
normalization would reduce it to the empty string and match nothing.
An AAD object id is empty for guest and anonymous callers, so an anonymous caller can never be
allowlisted. That is the intended behaviour.
`pairing` is enforced as a plain allowlist on calls: there are no pairing codes, expirations or
approval prompts on this path, so the fix for a refused caller is the same as under `allowlist`.
## How a call flows
```text theme={null}
caller 16 kHz --> 24 kHz --> OpenClaw realtime session
model 24 kHz --> 16 kHz --> the caller
```
StandIn dials `wss:///msteams/calling/{callId}` once per call. The SDK answers that dial and
speaks the wire protocol; the plugin resamples both legs, guards against echo, and handles barge-in.
It carries the three things that make a call feel right rather than merely work:
`cancelPlayback()` runs **before** the response is cancelled upstream, so the caller stops hearing the
turn they interrupted rather than listening to the rest of it play out.
There are two triggers. The model truncating its own turn because it heard the caller, and a
deterministic verbal interrupt matched in code: "stop", "hold on", "never mind". The second exists
because the model is mid-generation when one arrives, and matching the phrase ourselves is what makes
the cut feel instant. It only fires while the agent is actually still speaking, because otherwise
"stop" is just a word in a sentence.
On a speakerphone the agent's own voice comes back up the caller leg loudly enough for a realtime
model's VAD to answer itself, which produces a call where the agent greets, interrupts itself and
re-greets while the caller has said nothing.
Caller input is dropped while our own audio is still playing out, plus a short window after it,
unless it is loud enough to be a real interruption. Tune with `echoSuppressionWindowMs` and
`echoBargeInRms`, or turn it off entirely with `suppressInputDuringPlayback: false`.
With `requireRecordingStatus` on, no caller media reaches the model until Microsoft Teams reports recording
active. This is a Microsoft Media Access API obligation rather than a preference: a bot that processes
call media before `updateRecordingStatus` goes active is out of policy.
It is off by default because the hosted service can be configured either way, and a gate nobody asked
for is a call of silence.
## Refusal reasons
The plugin refuses a call in four places, which map to three reason strings. Each one is handed to
StandIn rather than dropping the socket, so the refusal is visible on the call instead of appearing
as silence.
| Reason | Meaning |
| ---------------------- | -------------------------------------------------------------------- |
| `realtime-unavailable` | No realtime voice provider resolved, or the provider connect failed. |
| `not-allowed` | `inboundPolicy` rejected this caller. |
| `busy` | At `maxConcurrentCalls`. |
Policy and capacity are checked before the call slot is taken, so a refused caller cannot consume
capacity by dialling repeatedly.
## Checking it works
The gateway logs one line at boot when the listener is up:
```text theme={null}
standin-msteams: listening on 0.0.0.0:9442/msteams/calling (max 4 concurrent)
```
If instead it says nothing started, the `secret` is missing or unresolved. If it warns that no realtime
voice provider resolved, every call will be refused with `realtime-unavailable`, so set the provider's
API key. That warning names the configured provider id when there is one, which separates "the key is
missing" from "no provider is configured at all".
The provider is resolved **once, when the service starts**, not per call. Adding or fixing an API key
therefore needs the gateway restarted or the plugin reloaded before the next call sees it. The boot
line above is the confirmation that it did.
## Scope
What this plugin covers, stated once so you can design around it:
* **The call and the voice.** A realtime speech-to-speech session, barge-in, the echo guard, the
recording gate and the inbound policy. That is the whole of what a caller talks to.
* **OpenClaw agent consultation, tools, skills and memory** are reached from chat, not from the
realtime session this plugin drives. The prompt the caller talks to is `realtime.instructions`,
which is this plugin's own configuration rather than the agent's.
* **Chat and messages** belong to the SDK's own [chat lane](/typescript-sdk/chat), which is a
separate surface with its own page. When `meetingRecap` is on, this plugin opens that lane
listen-only so it can post minutes without answering chat.
* **Meeting recap** is this plugin's `meetingRecap` switch. Off by default. See
[Meeting recap](#meeting-recap) below, and the library it calls in
[Meeting recap](/typescript-sdk/minutes).
* **Vision, avatar expression and visemes** are on the StandIn call wire and in the SDK core, and
this plugin does not wire them into the OpenClaw realtime session. That is a boundary of this
plugin, not of the call. Reach for `VisionTools` and `CallTools` from a handler of your own:
[Vision and the avatar](/typescript-sdk/vision) and [The avatar](/typescript-sdk/avatar).
* **Providers** are the realtime speech-to-speech ones OpenClaw resolves, and the plugin consumes
the host's own registry rather than carrying a second list. A provider that offers only speech to
text and text to speech is not one this plugin can drive: the whole design rests on the model
hearing the caller directly, which is where the latency goes.
## Meeting recap
Off until you set `meetingRecap: true`. The plugin then:
1. Collects the call transcript.
2. Writes a restart-recoverable local spool before hang-up returns (`STANDIN_RECAP_DIR`, else `STANDIN_STATE_DIR/recap`, else `~/.standin/state/recap`). A process restart drains what was left if that directory survives the process. Treat recap as best-effort unless that directory is on storage that survives a process or container restart. The file stays until posted, expired (24 hours), or the retry limit (8).
3. Opens its own **listen-only** [chat lane](/typescript-sdk/chat) and posts minutes on it. That lane never answers chat, even if something else on the host already does.
The host must provide an outbound chat lane and a summarization consult. Hang-up does not wait for the consult. A recap that cannot open the chat lane is logged and not posted. Meeting recaps are best-effort. They may be lost if the worker exits during processing.
Delivery uses StandIn's outbound chat socket. That socket exists on the **StandIn AI Teammate app**
([Tiers](/tiers)). A bring-your-own-bot identity answers chat through
its Azure Bot messaging URL, not this socket, so this switch will not post there. Put
`STANDIN_RECAP_DIR` on persistent storage when recap reliability matters. A directory that dies with
the process loses unfinished minutes.
`sessionScope` keys the **consult that writes the minutes**, not the voice session. Voice is always
one session per call.
# Quickstart
Source: https://docs.komaa.com/typescript-sdk/quickstart
From one npm install to a Microsoft Teams call your Node.js worker answers.
This page gets a Node.js worker answering a real Microsoft Teams call. It uses the echo plugin
first, on purpose: if the echo answers, your secret, your tunnel and your StandIn identity are all
correct, and anything that breaks afterwards is your agent rather than the transport.
## Prerequisites
* Node.js 20 or newer
* A StandIn identity and its connection secret, from [standin.komaa.com](https://standin.komaa.com)
* A way to expose one port over HTTPS: a tunnel on your laptop, or an ingress in a cluster
```bash theme={null}
npm install @komaa/standin-sdk
```
One package. The echo plugin and the `standin-echo` command come with it.
```bash theme={null}
STANDIN_SECRET="your-StandIn-connection-secret" \
STANDIN_HOST="127.0.0.1" \
npx standin-echo
```
One line tells you the listener is up:
```text theme={null}
standin: answering Microsoft Teams calls on 127.0.0.1:9442/msteams/calling
```
`STANDIN_HOST=127.0.0.1` binds to loopback because a tunnel is the only thing that needs to reach it
on a laptop. In a container behind an ingress, leave it unset and the default `0.0.0.0` applies. The
upgrade is HMAC-authenticated either way.
StandIn dials `wss:///msteams/calling/{callId}`, so the public URL must terminate TLS and
forward WebSocket upgrades on that path. The path has to match, because that is where the per-call
`callId` segment is appended.
[Expose your agent](/expose#mount-the-call-path) has the mount command for Tailscale Funnel, ngrok,
cloudflared and devtunnel, and the probes that tell a registered mount from a working one. It is the
only page that carries those commands, because three divergent spellings of one URL across three
pages is what caused a live incident here.
Expose only that path. Anyone who reaches the port still needs a valid signature inside the freshness
window, so this is defence in depth rather than the authentication itself, but there is no reason to
publish more surface than one route.
In the StandIn portal, set the identity's agent voice URL to the public `wss://` URL including the
`/msteams/calling` path. Then call the identity from Microsoft Teams and talk. You should hear yourself.
## What just happened
```mermaid theme={null}
sequenceDiagram
participant T as Microsoft Teams
participant SI as StandIn
participant W as Your worker
T->>SI: caller dials the identity
SI->>W: WSS upgrade to /msteams/calling/{callId}, signed
W-->>SI: 101, handshake verified once
SI->>W: session.start
W->>W: onStart(session)
SI->>W: audio.frame (PCM16 16 kHz, continuously)
W->>SI: audio.frame (your reply)
SI->>W: session.end
W->>W: aclose(reason)
```
## Replace the echo with your agent
The echo plugin is about 70 lines, and the shape of it is the shape of every plugin. Copy
it and change `onCallerAudio`:
```ts theme={null}
import { CallServer, type CallSession } from "@komaa/standin-sdk";
class MyHandler {
#call: CallSession | undefined;
async onStart(session: CallSession) {
this.#call = session;
// Join a room, open a realtime socket, build an agent. Caller audio does not
// flow until this resolves, so a slow start delays the caller instead of
// dropping frames.
}
async onCallerAudio(pcm: Buffer) {
// PCM16, 16 kHz, mono, little-endian. Hand it to your agent.
const reply = await myFramework.respond(pcm);
await this.#call?.sendAudio(reply);
}
async onGoodbye(text: string) {
// StandIn is ending the call and wants this line spoken first.
}
async aclose(reason: string) {
// Always called exactly once, on every path.
}
}
const server = new CallServer({ handlerFactory: () => new MyHandler() });
await server.start();
```
If your agent speaks at 24 kHz, which every realtime speech-to-speech model does, do not write your
own resampler. Read [Audio](/typescript-sdk/audio) first: the SDK ships the resampler and the frame
aligner because getting the frame boundary wrong clips the end of every turn.
## Configuration
Everything reads from the environment, and `CallServerOptions` overrides any of it in code.
| Variable | Default | Meaning |
| --------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `STANDIN_SECRET` | none, required | The connection secret from the portal. Without it the server refuses to construct. |
| `STANDIN_PORT` | `9442` | Port the call listener binds. |
| `STANDIN_HOST` | `0.0.0.0` | Bind address. |
| `STANDIN_WS_PATH` | `/msteams/calling` | Path StandIn dials. |
| `STANDIN_CHAT_URL` | `wss://teams.standin.komaa.com/api/chat/channel` | The [chat lane](/typescript-sdk/chat) endpoint. |
| `STANDIN_CHAT_SECRET` | unset, then `STANDIN_SECRET` | The key the chat lane signs with, read **before** `STANDIN_SECRET`. Set it only when your deployment was issued a separate chat credential. |
That is the set a first worker needs. [Configuration](/typescript-sdk/configuration) lists every
variable the SDK reads and what reads it.
## When it does not answer
| Symptom | Cause |
| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The process exits at startup naming the secret | `STANDIN_SECRET` is unset. `CallServer` throws rather than start unauthenticated. |
| StandIn reports 401 | The secret does not byte-match the portal value, or the worker clock is more than 60 seconds off. |
| StandIn reports 404 | The public URL is missing the `/msteams/calling` path segment. |
| StandIn reports 503 | The worker is draining, or at `maxConnections`. |
| `401 handshake already used` | A correctly signed upgrade was replayed. The single-use guard is doing its job. |
| `409 call already has a live session` | A previous call for that `callId` has not finished tearing down, or a retry arrived while the first socket was still open. |
| The call connects and ends after 10 seconds | `session.start` never arrived. The pre-start watchdog closed it. |
| The call ends after two minutes with `no-agent-answered` | The handler never sent audio and never called `session.markAnswered()`. The usual cause is an agent dispatch that never landed: the socket, the signature and `onStart` were all fine, and nothing arrived to take the call. |
| The call ends mid-conversation with `caller-idle-timeout` | No caller audio for 45 seconds. A live call always sends frames, so the far side is gone. |
`GET /healthz` on the same port returns `{"ok": true, "calls": }`, which is the cheapest way to
confirm the listener is really bound. To go further without placing a call at all, see
[Checking the install](/typescript-sdk/checking-the-install).
## Next
The seven methods, the session object, and how to refuse a call.
The template, and how to turn it into your own plugin.
What to do when your agent speaks at 24 kHz.
Why a drifting clock turns every call into a 401.
# Reaching people
Source: https://docs.komaa.com/typescript-sdk/reaching-people
Speak into a call that is already up, otherwise ring somebody back and park the line until they answer, in the StandIn TypeScript SDK.
Every other lane in this SDK starts with somebody dialling your agent. This one runs the other way: something outside a call wants to reach a person. A scheduled job finished, a chat message asked to be called, a host handed off a task.
Which way that lands depends on one thing: whether a call to that person is up right now.
If one is, the line is spoken into it. Ringing somebody who is mid sentence with you is the rudest possible way to tell them something. If one is not, a call has to be placed and the line parked, so it is said the moment they answer.
`VoiceDelivery` makes that choice, and `LiveCalls` is what it consults.
```ts theme={null}
import {
LiveCalls,
OutboundCaller,
OutboundPolicy,
PendingMessages,
VoiceDelivery,
} from "@komaa/standin-sdk";
const live = new LiveCalls();
const delivery = new VoiceDelivery(live, {
caller: new OutboundCaller(),
policy: OutboundPolicy.fromEnv(),
pending: new PendingMessages(),
});
const result = await delivery.deliver("Your build finished.", userObjectId);
```
Four pieces, and each answers one question. `live` is which calls are up. `policy` is whether this agent may ring that person at all. `caller` places the call. `pending` is what to say when they pick up.
`new OutboundCaller()` reads `STANDIN_SECRET` and refuses to be built without it, because that secret is what signs the request. `STANDIN_WORKER_URL` is the StandIn control address it posts to, which StandIn gives you with your connection secret. Outbound calling is unavailable until it is set, and a URL that is not http or https, has no host, or carries credentials is refused when the caller is built rather than when somebody tries to ring.
Every option is optional, and the defaults are the safe ones rather than the convenient ones. With no `policy`, `VoiceDelivery` builds `new OutboundPolicy({})`, whose allowlist is empty, so every call-back is refused. Pass `OutboundPolicy.fromEnv()` or an explicit list.
## Registering a live call
A call registers itself, from the handler that owns it, for the life of that call.
```ts theme={null}
import {
CallServer,
LiveCalls,
decodeWav,
type CallSession,
type LiveSpeaker,
} from "@komaa/standin-sdk";
class MyHandler implements LiveSpeaker {
#live: LiveCalls;
#session: CallSession | undefined;
constructor(live: LiveCalls) {
this.#live = live;
}
async onStart(session: CallSession) {
this.#session = session;
this.#live.register(this, session.callId, session.start.threadId);
}
async say(text: string) {
const session = this.#session;
if (session === undefined) throw new Error("this call is no longer up");
// Whatever makes this plugin's provider speak.
await session.sendAudio(decodeWav(await tts.speak(text)));
}
async aclose(reason: string) {
const session = this.#session;
this.#session = undefined;
if (session) this.#live.unregister(this, session.callId, session.start.threadId);
}
}
const live = new LiveCalls();
const server = new CallServer({ handlerFactory: () => new MyHandler(live) });
```
`say(text: string): Promise` is the whole `LiveSpeaker` interface. It belongs to the plugin because only the plugin knows how to make its provider speak without stepping on whatever the agent was already saying.
Throw from `say` when the call has gone. A `say` that quietly does nothing resolves, and a resolved `say` is reported as `mode: "live-call"`, so the line is counted as spoken and nobody ever hears it.
### Two keys, one call
`register(speaker, callId, threadId = "")` files the speaker under the call id, and under the Microsoft Teams conversation when the call has one. Both, because a delivery addressed by conversation would otherwise miss a call that is only filed by call id, and the agent would place a second call to somebody already on the line with it.
Each key is trimmed, and an empty one is skipped. `size` counts keys rather than calls, so a call filed under both counts twice.
`deliver` looks the conversation up first and the target second. Whichever key names a live call wins, and `find(...keys)` returns `undefined` when none of them does.
### Unregister removes this speaker's keys, and only this speaker's
`unregister(speaker, callId, threadId = "")` deletes a key only when it still points at the speaker that is being removed.
That identity check is the point. A second call on the same conversation can start before the first one's teardown runs. A blind delete would then wipe the live call's entry, and every later delivery for that conversation would ring a fresh call. The person would hear a second ring instead of an answer.
### When the live call cannot speak
If `say` rejects, the delivery does not stop there. It logs, falls through to the call-back path, and comes back with `mode: "call-back"`.
A wedged provider socket would otherwise swallow the message with nobody told. The trade is deliberate: a half-spoken line can be repeated by the call-back, and a repeat beats silence.
## Who the agent may ring
`OutboundPolicy` is allow-by-explicit-listing. A directory id must be on the list before the agent can ring it, and an empty list means outbound calling is off.
| Option | Default | What it is |
| ------------ | ------- | ----------------------------------------------------------------------------------------------------------------- |
| `allowed` | empty | Directory ids the agent may ring. Empty means outbound is off. |
| `maxPerHour` | `6` | Calls placed in any rolling hour, across all targets. Zero means no cap, which is a choice rather than a default. |
`OutboundPolicy.fromEnv()` reads `STANDIN_OUTBOUND_ALLOW`, a comma-separated list of directory ids, and `STANDIN_OUTBOUND_MAX_PER_HOUR`. Unset means off.
Matching is case-folded on both sides. A directory id is not case-sensitive, and a case mismatch would read as "not allowed" with nothing to say why.
This list is separate from any inbound allowlist, and stricter. Allowing every inbound caller allows no outbound target.
The two answer different questions. Inbound asks "may this person talk to the agent?" and the person chose to dial. Outbound asks "may the agent ring this person?" and the agent was talked into it by whoever is on the call. An agent with an outbound tool and no allowlist is an agent that can be talked into cold-calling your directory.
Every refusal is one sentence, safe to read out loud:
| When | What comes back in `error` |
| ---------------- | ---------------------------------------------------------------------------------------------- |
| No allowlist | `outbound calling is off: set STANDIN_OUTBOUND_ALLOW to the directory ids this agent may ring` |
| Not on it | `that person is not on this agent's outbound allowlist` |
| No target at all | `an outbound call needs the person's directory id` |
| Over the cap | `this agent has already placed 6 calls in the last hour` |
## The tenant is operator configuration
Which organisation a call is placed into comes from the `tenantId` option on `VoiceDelivery`, falling back to `STANDIN_TENANT_ID`. That variable name is exported as `TENANT_ENV`.
It never comes from the message, the metadata, the model or the caller. A model is steered by whoever is talking to it, so a model that can choose the tenant is a model that can be talked into dialling a different organisation. With no tenant configured, `deliver` returns `no tenant is configured: set STANDIN_TENANT_ID to place calls` and rings nobody.
Who gets rung follows the same reasoning. `CHAT_CALLBACK_TOOL` and `CALL_BACK_TOOL` take a `message` and nothing else: there is no target parameter on them, and there never will be.
### Where the target comes from instead
`OutboundLane.rememberChatSender(message)` records who last wrote in a conversation, reading it off the `InboundMessage` itself, and `chatCallbackTarget(conversationId)` hands that back as a `ChatCallbackTarget`.
| Field | What it carries |
| ---------------- | ----------------------------------------------------------------------------- |
| `userObjectId` | The directory id to ring. |
| `tenantId` | The organisation that message came from. |
| `conversationId` | Where the request came from, so an unanswered call still has somewhere to go. |
| `displayName` | What to call them, when the message carried a name. |
The message is the only source. Never the message text, never a tool parameter: an agent that can be told who to ring is an agent that can be talked into ringing anybody, and the thing doing the telling is being steered by whoever is on the other end of it. A message with no `senderAadId` is recorded not at all, rather than recorded as an empty target.
When nobody has written, or the last message is more than ten minutes old, what comes back is a sentence rather than a target: `I do not know who to call for this conversation.` or `That was a while ago. Ask me again and I can call you.` A sentence rather than a thrown error, because whatever reads this is a tool result a model says out loud. Check which you got before you use it, and TypeScript makes you: the return type is `ChatCallbackTarget | string`.
```ts theme={null}
const target = lane.chatCallbackTarget(message.conversationId);
if (typeof target === "string") return target; // already safe to read out
const placed = await lane.place({
userObjectId: target.userObjectId,
text: "Your build finished.",
tenantId: target.tenantId,
threadId: target.conversationId,
});
```
That ten minute window is `CHAT_CALLBACK_WINDOW_MS`, and the Python twin holds the same ten minutes as `CHAT_CALLBACK_WINDOW_S`, in seconds. The number cannot be copied between the two SDKs, only the duration.
## The hourly budget counts calls that rang
`policy.record()` runs after the call is placed and a call id comes back, not when the attempt starts.
Counting attempts would let a broken worker burn the whole hour on calls that never rang anybody, and the next real delivery would be refused for an hour because of it. A worker that is down costs nothing against the budget. A phone that actually rang costs one.
## What is parked, and where
`PendingMessages` is on disk, and that is the part people get wrong.
The leg that answers is **a different call**. You ask for the call in one place, and StandIn dials your worker minutes later with `direction: "outbound"` and a fresh `callId`. A restart between the two is ordinary, and it may not even be the same process. Parked in memory, the message is lost silently: the callee picks up and hears nothing, with no error anywhere.
`stateDir()`, at the package barrel here and in `standin.outbound` on the Python side, is `STANDIN_STATE_DIR` when set, otherwise `~/.standin/state`, created owner-only. It is deliberately not a temp directory. A temp directory passes every test and loses every parked message on the next reboot.
`VoiceDelivery` parks the record before `deliver` returns, because the leg can be answered before a later park would have run.
| Field | What it carries |
| ------------- | -------------------------------------------------------------------------------------------------- |
| `callId` | The id the answering leg arrives with. |
| `text` | The line to speak on answer. |
| `threadId` | The conversation the request came from. |
| `tenantId` | Which tenant to post the fallback into. Never taken from a model. |
| `target` | Who was rung, so a second call to the same person can be refused while the first is still ringing. |
| `requestedBy` | Directory id of whoever asked for the call, for the audit trail. |
| `createdMs` | Stamped by `park` when the record has no time of its own. |
| `attempts` | Delivery attempts so far, so a chat that keeps failing stops. |
Popping is atomic. The rename is the lock, so two workers answering the same leg cannot both speak.
```ts theme={null}
const parked = pending.pop(session.callId);
if (parked !== undefined) await this.say(parked.text);
```
Nothing should be said while the leg is still ringing, and there is no "they picked up" message on the wire. `OutboundLane.attach(session, speak)` holds the record, treats recording going active as the answer, speaks then, and gives the record back if nobody ever picks up.
### The thread id is how an unanswered call still reaches them
A call is placed because somebody is owed something. If they do not pick up, they are still owed it.
`threadId` is carried on the record for exactly that: it is the conversation the request came from, so the line can be posted there instead of evaporating. `OutboundLane` owns that sweep, and it posts once even when both the ring timer and the call outcome fire for the same call.
`OutboundLane.place` parks no fallback at all for a call that has no real conversation, which is what `callThreadIsPostable` decides. A one-to-one call has no meeting conversation, and posting to what that field carries instead would either fail or reach the wrong place.
What does the posting is whatever you passed as `new OutboundLane({ chat })`, and `ChatSender` is the shape it has to have: one `send({ tenantId, conversationId, text, idempotencyKey })` returning a promise of a boolean. `ChatChannel` already satisfies it, so the ordinary case is handing over the channel you already have. The interface exists so that a plugin posting through something else is not forced to wrap one. It is structural in TypeScript and a `Protocol` in Python, and it takes one options object here where the Python twin takes keyword arguments.
It returns whether the message went, not whether it was attempted. A `false` puts the record back for another try, and after five attempts, or once the record is older than the lane's `ttlMs`, the lane gives up in the log rather than retrying for ever. The `idempotencyKey` is what makes the ring timer and the call outcome both firing for one call tell the person once.
## The leg that answers
Placing a call returns a `PlacedCall`, from `OutboundCaller.placeCall` and from `OutboundLane.place` alike.
| Field | What it is |
| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `callId` | The id the answering leg will arrive with. This is what a message is parked against. |
| `scenarioId` | StandIn's own correlation id when it sends one, otherwise an empty string. Log it and a call in your records matches a call in a support question. |
`OutboundLane.attach(session, speak)` turns that id back into an `OutboundLeg` on the call that answers, and returns `undefined` on an inbound one, so a plugin calls it unconditionally from `onStart` and forwards two things:
```ts theme={null}
class MyHandler {
#leg: OutboundLeg | undefined;
async onStart(session: CallSession) {
this.#leg = lane.attach(session, (message) => this.say(message.text));
}
async onContext(text: string) {
await this.#leg?.onContext();
}
async aclose(reason: string) {
await this.#leg?.aclose(reason);
}
}
```
The session only has to satisfy `OutboundSession`, which is the `callId`, the `direction` on `start`, `recordingActive` and an `end` to call, so a test double is four members rather than a whole call. The leg holds the parked line on `message` and speaks it through the `Speak` you gave `attach`, which owns the wording: only your plugin knows whether its provider takes an instruction or a literal line to say. A leg that finds nothing parked waits a few seconds before concluding there is nothing to say, because the leg can be answered before `place` has finished parking, and a race lost there is a caller who picks up to silence.
**Recording going active is what "they answered" means.** There is no "they picked up" message on the wire, and recording turning on is what happens when a Microsoft Teams call is actually connected, so a forwarded `onContext` is the signal. A plugin with a better one calls `answered()` directly. A pickup fast enough to beat the attach is covered too, because arming checks the recording once on the spot rather than only waiting for the next one.
**The leg runs its own watchdog**, ending a call nobody answered after the lane's `answerTimeoutMs`, which defaults to `DEFAULT_ANSWER_TIMEOUT_MS` and is two minutes. The idle watchdog cannot do this job: a ringing leg carries no caller audio by definition, so to that watchdog every outbound call looks dead from the moment it starts.
**Nothing unsaid is thrown away.** If speaking rejects, the line is not marked spoken, because it was not said. If the leg closes with the line still unsaid, `aclose` releases the record rather than deleting it, and the sweep posts it to chat instead. Somebody was owed something, and not picking up does not stop them being owed it.
Every outbound duration here is named in milliseconds and the Python twin names the same duration in seconds: `DEFAULT_ANSWER_TIMEOUT_MS` against `DEFAULT_ANSWER_TIMEOUT_S`, `answerTimeoutMs` against `answer_timeout_s`, `ttlMs` against `ttl_s`. Port the duration, never the number.
## The result
```ts theme={null}
const result = await delivery.deliver(text, userObjectId, conversationId);
if (!result.ok) return result.error ?? "";
```
| Field | What it means |
| -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ok` | Whether the line reached a lane that will say it. |
| `mode` | `"live-call"` when it was spoken into a call that was already up, or `"call-back"` when a call was placed and the line parked. Both strings are part of this API. |
| `callId` | The call that was placed, on the call-back path. Absent otherwise. |
| `error` | One sentence, safe to read out loud. Absent when `ok`. |
`deliver` never throws. Whatever reads the result is either a host that marks the whole platform failed on an exception, or a model that says it out loud, and neither can do anything useful with a stack trace.
Three refusals come from the placement rather than the policy:
| When | What comes back in `error` |
| ---------------------- | --------------------------------------------------------- |
| No `caller` was passed | `no outbound caller is configured on this deployment` |
| The request failed | `could not place the call: ...`, with the reason appended |
| No call id came back | `the call was not placed` |
An empty line is refused before the registry is even consulted, with `there was nothing to say`. An empty message must never place a real phone call to a real person.
# Realtime providers
Source: https://docs.komaa.com/typescript-sdk/realtime-providers
StartupBuffer and the echo guard for a speech-to-speech agent: what arrives before you are ready, and why the agent answers itself, in the StandIn TypeScript SDK.
A realtime speech-to-speech provider is handed the caller's audio and hands audio back, and the turn-taking is theirs. That is the whole reason to use one, and it is why [Turn-taking](/typescript-sdk/voice) is a page you can skip: no segmenter, no paced playback, no pipeline of three vendors to keep in step.
This page is the other half of that bargain. The provider owns the turn. It does not own the two things on either side of it:
* **what arrived before your provider was ready to take it**, which the provider cannot hold because it did not exist yet
* **whether the audio coming back is somebody else's voice or your own**, which the provider cannot know because it only sees one stream
Neither is a property of any framework, so neither lives in a plugin. Both are here.
```ts theme={null}
import {
ECHO_BARGE_IN_RMS,
ECHO_SUPPRESSION_WINDOW_MS,
MAX_PENDING_AUDIO,
MAX_PENDING_CONTEXT,
StartupBuffer,
pcm16Rms,
shouldSuppressEcho,
type EchoGuardOptions,
} from "@komaa/standin-sdk";
```
Every name on this page is exported from the package root. There is no deep import for these modules: the package publishes the root and the framework plugin subpaths and nothing else, so `@komaa/standin-sdk` is the import. `pcm16Rms` is also re-exported as `echoPcm16Rms`, which is the same function under an older name.
## What arrives before your agent is ready
The call starts the moment StandIn dials. Your agent starts later: a socket has to open, a room has to be joined, a session has to be configured, an agent job has to be dispatched and accepted.
`onStart` covers part of that gap for you, because inbound messages are queued while it runs. It does not cover the rest of it. A socket that is open is not a session that is configured, a room you have joined is not an agent that has joined it, and a handler that returns from `onStart` rather than holding the call open has handed itself the same gap deliberately, which is often the right call: `onStartTimeoutMs` is 15 seconds and a caller waiting on a slow provider hears dead air. See [onStart](/typescript-sdk/call-handler#the-seven-methods).
Two things land in that window and both matter.
**The caller's first words.** People start talking the instant the call connects, and the first thing they say is very often the reason they called. Drop it and the agent opens by asking a question that was already answered, which is the single most common way a demo call goes wrong.
**The first context.** The `There are 4 human participants on this call. Stay quiet unless directly addressed.` line and the `The Microsoft Teams call recording is now ACTIVE.` change both land in that same first moment. Drop those and a [group gate](/typescript-sdk/group-calls) never engages, so the agent answers every turn of a meeting, and a recording gate never opens, so an agent that waits to be told the call is recorded waits for the whole call. Neither failure looks like a dropped message. Both look like a feature that was never built.
Gate your own storage on `session.recordingActive`, which the SDK keeps current from `session.start` and every later status change, rather than on a sentence you happened to catch. What the buffer protects is the copy that reaches your **provider**, which has no other way to learn either fact.
```ts theme={null}
import { StartupBuffer, type CallSession } from "@komaa/standin-sdk";
class MyHandler {
readonly #pending = new StartupBuffer();
#agent: Provider | undefined;
async onStart(session: CallSession) {
this.#agent = await connectToProvider();
await this.#pending.release(
(pcm) => this.#agent!.sendAudio(pcm),
(line) => this.#agent!.sendContext(line),
);
const { audio, context } = this.#pending.dropped;
if (audio || context) {
console.info("standin: the caller outran the agent starting up");
}
}
async onCallerAudio(pcm: Buffer) {
if (this.#pending.holding) this.#pending.audio(pcm);
else this.#agent!.sendAudio(pcm);
}
async onContext(text: string) {
if (this.#pending.holding) this.#pending.context(text);
else this.#agent!.sendContext(text);
}
async aclose(reason: string) {
if (this.#pending.holding) this.#pending.discard();
}
}
```
| Member | What it does |
| ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `new StartupBuffer(maxAudio = MAX_PENDING_AUDIO, maxContext = MAX_PENDING_CONTEXT)` | One per call. Each bound is floored at 1. |
| `holding` | Whether the agent is still being set up. False after `release` or `discard`. |
| `audio(pcm)` | Hold one frame of the caller's voice. An empty buffer is ignored. |
| `context(text)` | Hold one line of call context. An empty string is ignored. |
| `await release(sendAudio?, sendContext?)` | Hand everything over, in order, and stop holding. Resolves to `{ audio, context }`, the counts released. |
| `dropped` | What was lost to the bounds, as `{ audio, context }`. |
| `discard()` | Throw it away, for a call that ended before the agent was ready. |
Order is preserved within each lane, and **audio is released before context**. The provider needs the caller's words in the order they were said; the context is a note about the call rather than part of the conversation, so it belongs after the words and not interleaved with them.
Both callbacks may be sync or async, and each is awaited. A provider's send is often fire-and-forget, and forcing a promise on one that is not is how a plugin ends up wrapping every send in a lambda. Either callback may be omitted, for a lane your provider has no route for: the buffer still drains and still stops holding, it just sends nothing.
`release` is safe to call twice: the second call releases nothing and resolves to `{ audio: 0, context: 0 }`. That matters on the failure paths, where `onStart` can end early in more than one place.
## Audio first, context second, oldest dropped first
Two caps, both defaults, both overridable per buffer:
| Constant | Value | What it is |
| --------------------- | ----- | --------------------------------------------------------------------------------------------- |
| `MAX_PENDING_AUDIO` | `200` | Frames of the caller's voice. At the wire's 20 ms frame that is **4 seconds** of speech. |
| `MAX_PENDING_CONTEXT` | `20` | Lines of call context. Context sentences are rare and each one is small, so this is generous. |
Four seconds is enough to hold an opening sentence and far short of enough to hide a socket that never opened. That is the whole calculation. A provider that is going to connect has connected by then; a provider that is not going to connect must not be allowed to accumulate the caller's entire call in memory while it fails to, on every concurrent call the worker is carrying.
When the cap is reached, the **oldest** entry goes and the newest is kept. If something has to be lost, lose the stale audio: the words the agent is about to answer are the ones the caller just said, and an agent that opens on a four-second-old fragment is worse than one that opens on the most recent sentence.
Log a non-zero `dropped`. It is the one signal that tells you the caller outran your provider's startup, and there is no other place it shows up: the call sounds normal, the agent answers, and the only trace is the sentence nobody accounted for. Treat it as a startup-latency alarm rather than an error.
`discard()` exists for the call that ends before the agent is ready. Use it in `aclose` when `holding` is still true, rather than releasing into a provider you are about to close.
## The agent answering itself
On a speakerphone, and on any laptop whose user has not got a headset on, the agent's own voice comes out of the speaker and goes straight back into the microphone. It arrives at your handler as caller audio. The provider's voice detection hears a turn, ends it, and replies to it. Then that reply loops back, and it replies to that. The caller is silent throughout and cannot get a word in, and it continues until somebody hangs up. It is not a subtle degradation, it is the call being taken over.
**The two SDKs are not the same shape here.** The Python SDK has an `EchoGuard` **class** that owns its own playout clock: you call `note_output`, `collapse` and `mark_caller_turn` on it, and ask `allow_input(rms)` per frame. This SDK has **no class**. It has one pure function, `shouldSuppressEcho`, and the clock is yours to keep. Anything you read about `EchoGuard` applies to the Python SDK only. The differences are tabulated at the end of this page.
`shouldSuppressEcho(pcm16k, playbackActiveUntil, opts?)` decides, per inbound frame, whether the caller's audio is withheld from the model.
```ts theme={null}
import {
FrameAligner,
REALTIME_SAMPLE_RATE_HZ,
SAMPLE_RATE_HZ,
frameDurationMs,
resamplePcm16,
shouldSuppressEcho,
type CallSession,
} from "@komaa/standin-sdk";
class MyHandler {
readonly #aligner = new FrameAligner();
#call: CallSession | undefined;
#provider: Provider | undefined;
/** Epoch ms at which the audio already sent finishes PLAYING. */
#playbackEndAt = 0;
/** False until the caller has genuinely spoken once. */
#callerTurnStarted = false;
async onCallerAudio(pcm: Buffer) {
if (shouldSuppressEcho(pcm, this.#playbackEndAt, { allowBargeIn: this.#callerTurnStarted })) {
return;
}
this.#provider!.appendInput(resamplePcm16(pcm, SAMPLE_RATE_HZ, REALTIME_SAMPLE_RATE_HZ));
}
async #onModelAudio(pcm24: Buffer) {
const pcm = resamplePcm16(pcm24, REALTIME_SAMPLE_RATE_HZ, SAMPLE_RATE_HZ);
for (const frame of this.#aligner.push(pcm)) {
await this.#call!.sendAudio(frame);
// Duration, not frame count: it is what the guard compares against the clock.
this.#playbackEndAt = Math.max(this.#playbackEndAt, Date.now()) + frameDurationMs(frame);
}
}
}
```
The polarity is "should I suppress this", not "may this through". `true` means **drop the frame**. Reading it the other way round mutes the caller for the whole call and passes the echo, which is the exact failure inverted.
The function is stateless and takes the raw frame rather than a number: it measures loudness itself with `pcm16Rms`, which returns root-mean-square amplitude normalised to `0.0` to `1.0` and is dependency-free because it runs on every inbound frame of every call. See [Audio](/typescript-sdk/audio).
`EchoGuardOptions` is four optional fields, and each one is read fresh on every call, so any of them can change during the call.
| Option | Default | What it does |
| ----------------------------- | ----------------------------------- | ---------------------------------------------------------------------------------- |
| `suppressInputDuringPlayback` | on | Exactly `false` turns the whole guard off. |
| `echoSuppressionWindowMs` | `ECHO_SUPPRESSION_WINDOW_MS`, `600` | How long after your audio finishes playing the guard stays armed. |
| `echoBargeInRms` | `ECHO_BARGE_IN_RMS`, `0.04` | Loudness an in-window frame must reach to be believed as a real interruption. |
| `allowBargeIn` | on | Exactly `false` drops the loudness exception and suppresses every in-window frame. |
What the function does, in full:
| State | Outcome |
| ---------------------------------------------------------- | ------------------------- |
| `suppressInputDuringPlayback: false` | Through. |
| Outside the playback window | Through. |
| In window, `allowBargeIn: false` | Dropped, at any loudness. |
| In window, barge-in allowed, RMS below the threshold | Dropped. |
| In window, barge-in allowed, RMS at or above the threshold | Through. |
`suppressInputDuringPlayback: false` is a real option, not a test hook. On a headset-only deployment there is no acoustic path from the speaker back into the microphone, so the guard has nothing to catch and every frame it filters is a barge-in the caller does not get. Note that only the literal `false` disables it: an `undefined` from an unset config key leaves the guard on, which is the safe way round.
## A playout clock, not a send clock
A realtime model streams its answer far faster than realtime. Five seconds of speech can be handed over in a fraction of a second, and a call consumes audio at one second per second, so the rest of it is still buffered and still being heard long after the send returned.
So wall-clock send time is useless here. A guard keyed on "did we send something recently" disarms a few hundred milliseconds into a sentence that is still coming out of the caller's speaker, which is the exact moment the echo it exists to catch starts arriving.
`playbackActiveUntil` is therefore the estimated **epoch milliseconds at which the audio you have already sent finishes playing**, and you accumulate it yourself, one duration at a time:
```ts theme={null}
this.#playbackEndAt = Math.max(this.#playbackEndAt, Date.now()) + frameDurationMs(frame);
```
The `Math.max(..., Date.now())` rather than a bare addition is the part worth reading twice. After a gap in speaking, the old horizon is already in the past, and adding to a stale horizon leaves the clock permanently behind real time, so the guard never arms again for the rest of the call and the failure looks like a guard that was never switched on.
Pass `frameDurationMs(frame)` rather than counting frames. Outbound chunk lengths are not fixed, and a frame count drifts against real time in exactly the direction that hurts.
It must be `Date.now()`, on the epoch-millisecond scale. `shouldSuppressEcho` compares against `Date.now()` internally, so a horizon built from `performance.now()` sits decades in the past and the guard never fires once. The [group gate](/typescript-sdk/group-calls) takes either clock because it only compares differences; this one does not.
## No barge-in until the caller's first real turn
Pass `allowBargeIn: false` until the caller has spoken once, and every in-window frame is dropped however loud it is.
That is not a conservative default, it is the specific case. The agent's opening greeting is the first thing on the call, it echoes back, and it is loud, because a greeting is spoken at full volume into a room nobody is talking over yet. Without this rule the loudest possible echo arrives at the exact moment there is no history to compare it against, the agent treats it as a barge-in, interrupts itself, and greets itself again. And again.
Flip the flag where you know a real turn happened: on a finished caller transcript, and on a barge-in your provider reported and you accepted. The method below is **your own**, wired to whatever your provider calls a transcript event, and it is what the SDK's own realtime plugin does on its provider's.
```ts theme={null}
#onProviderTranscript(role: "user" | "assistant", text: string) {
if (role === "user" && text.trim() !== "") this.#callerTurnStarted = true;
}
```
Because the flag is an argument rather than state inside a guard object, it latches in your own field and there is nothing to reset. Note that only the literal `false` withholds barge-in: leave the option out and the loudness exception is on, which is the wrong way round for the first few seconds of a call, so set it explicitly from the start.
## Collapse on an accepted barge-in
When you accept a barge-in you cancel the playback StandIn still holds. Your horizon does not know that. It counted that audio as if the caller would hear all of it, so it is still sitting several seconds in the future.
Left alone, the guard keeps filtering the caller's microphone for the length of the buffer you just threw away, which is precisely the words they interrupted you to say. The caller interrupts, the agent stops, the caller talks, and the agent hears nothing.
```ts theme={null}
async #onBargeIn() {
await this.#call!.cancelPlayback(); // first: un-send what StandIn still holds
this.#aligner.reset(); // the residual belongs to a dead turn
this.#playbackEndAt = Date.now(); // the horizon covered audio nobody will hear
this.#callerTurnStarted = true; // this was a real turn
this.#provider!.cancelResponse(); // then: stop the model generating more
}
```
Do this anywhere you call `cancelPlayback()`, which includes `onGoodbye`: a goodbye cancels playback too, and a horizon left standing after one spends the last seconds of the call filtering the caller's reply to it. See [Barge-in](/typescript-sdk/call-handler#cancelplayback-is-the-barge-in).
Assigning `Date.now()` is what the SDK's own realtime plugin does, and it leaves the guard armed for one more tail window, `600` ms by default, which is usually what you want: the last already-playing milliseconds are still echoing. The Python SDK's `collapse()` goes further and pulls the horizon to `now - tail_window_ms`, so its guard is disarmed immediately. Subtract `echoSuppressionWindowMs` yourself if you want that exact behaviour.
## The tail belongs to your provider
`echoSuppressionWindowMs` is an option and not a hard constant, and that is a deliberate line. It covers the network and the jitter buffer between you and the caller's speaker, and then however long the provider's own voice detector keeps hearing you after the audio stops. The first part is the wire and is the same for everybody. The second is a property of that provider, and providers differ.
`600` is the default and the value in `ECHO_SUPPRESSION_WINDOW_MS`. Move it with the failure you actually see:
* Too short and the tail of your own sentence gets through as a turn, and the agent answers its own last three words.
* Too long and a genuine interruption in the moments after you stop speaking is dropped, and the caller has to say it twice.
`echoBargeInRms` is the same kind of knob on the same kind of trade. `0.04` on the `0.0` to `1.0` scale `pcm16Rms` returns. Raise it in a loud room and you buy fewer false barge-ins with a real cost: the case it then gets wrong is a quiet caller trying to interrupt a loud agent, which is the caller who most needs the interruption to work.
Tune both per provider, keep them in your plugin's configuration rather than in its code, and treat the defaults as a starting point that was measured on a call rather than a constant that was derived from one. `EchoGuardOptions` is shaped to be spread straight from a config block, with your per-call `allowBargeIn` layered on top:
```ts theme={null}
shouldSuppressEcho(pcm, this.#playbackEndAt, {
...this.#config.echo,
allowBargeIn: this.#callerTurnStarted,
});
```
## What is not the same in the Python SDK
`StartupBuffer` is the same object in both, with the same caps and the same two lanes. One detail is language-shaped: `dropped` and the resolved value of `release` are objects here, `{ audio, context }`, and tuples there, `(audio, context)`.
The echo guard is genuinely different, and a port that assumes otherwise compiles and then fails on a real call.
| | Python SDK | TypeScript SDK |
| ------------------ | ------------------------------------------------------------- | ----------------------------------------------------------------- |
| Shape | `EchoGuard`, a class holding per-call state | `shouldSuppressEcho`, a pure function |
| Polarity | `allow_input(...)` is `True` to **pass** the frame | `shouldSuppressEcho(...)` is `true` to **drop** it |
| Input | a loudness float you computed yourself | the PCM16 `Buffer`, measured for you |
| The playout clock | inside the object, fed by `note_output(duration_ms)` | yours, passed as `playbackActiveUntil` |
| Clock source | monotonic milliseconds | epoch milliseconds, from `Date.now()` |
| Accepted barge-in | `collapse()`, disarms immediately | assign the horizon yourself, tail window still armed |
| First caller turn | `mark_caller_turn()`, a latch on the object | `allowBargeIn`, an option passed per frame |
| Tail and threshold | `tail_window_ms`, `barge_in_rms`, set once on the constructor | `echoSuppressionWindowMs`, `echoBargeInRms`, read fresh per frame |
| Disabling it | `EchoGuard(enabled=False)` | `suppressInputDuringPlayback: false` |
The constants are the same in both and carry the same names, `ECHO_SUPPRESSION_WINDOW_MS` at `600` and `ECHO_BARGE_IN_RMS` at `0.04`, so configuration and tests can name them rather than repeat them.
## Next
The gate that keeps the agent out of a meeting it was not addressed in.
A full realtime handler, with the resampling, the alignment and the barge-in.
The other half: an agent built from separate transcription, model and speech.
`frameDurationMs`, `pcm16Rms`, the resampler and the frame aligner.
# Security
Source: https://docs.komaa.com/typescript-sdk/security
The two StandIn HMAC lanes, their replay windows, who may call the agent, and how the TypeScript SDK fails closed.
StandIn authenticates with one shared secret and HMAC-SHA256. There are **two lanes**, they sign
different things, and they are not interchangeable.
| Lane | Signs | Window the SDK enforces | Header |
| ------------ | ---------------------------------------------------------------------- | ----------------------------- | ------------------------ |
| v1 handshake | `"{timestampMs}.{id}"`, where `id` is the channel name or the `callId` | 60 s, fixed | `X-StandIn-Signature` |
| v1 body | `"{timestampMs}."` followed by the exact body bytes | 300 s by default, overridable | `X-StandIn-Signature` |
| v2 request | `"{timestampMs}."` followed by `METHOD\npath\nsha256(body)` | 60 s | `X-StandIn-Signature-V2` |
The timestamp always travels in `X-StandIn-Timestamp`, in milliseconds. Signatures are lowercase hex.
There are three signers, not two. The body signer and the request signer carry different headers and
sign different things, and collapsing them loses the distinction that makes v2 worth having.
Their windows differ too, and the difference is the one people get wrong. The **body** lane allows
300 seconds because a relay retry is legitimately delayed. The **request** lane is verified inside
the 60 second handshake window, because a control request is sent and answered immediately. The only
v2 verifier in this SDK is the call-outcome route on `CallServer`, so a worker whose clock is more
than a minute out rejects call outcomes exactly as it rejects calls.
Both lanes are real and both are correct. The WebSocket lane signs the **channel or call name**; the
POST relay signs the **body**. Never present one as a fix for the other, and never "correct" a
handshake signer into a body signer. They protect different things.
## Signing helpers
```ts theme={null}
import {
REPLAY_WINDOW_MS, // 60000
CHAT_REPLAY_WINDOW_MS, // 300000
SIGNATURE_HEADER, // "x-standin-signature"
SIGNATURE_V2_HEADER, // "x-standin-signature-v2"
TIMESTAMP_HEADER, // "x-standin-timestamp"
nowMs,
} from "@komaa/standin-sdk";
```
Import a window rather than retyping `300000`. The header constants hold **lowercase** strings here
because Node lowercases incoming header names, while the Python SDK's constants of the same names
hold the canonical casing. HTTP header names are case-insensitive, so the wire is identical and only
the constant's spelling differs.
The four signer and verifier signatures:
```ts theme={null}
signHandshake(secret, timestampMs, handshakeId): string
verifyHandshake(secret, timestamp, handshakeId, signature, currentMs?): boolean
signBody(secret, timestampMs, rawBody): string
verifyBody(secret, timestamp, rawBody, signature, currentMs?, windowMs?): boolean
canonicalRequest(method, path, rawBody): string // "METHOD\npath\nsha256_hex(body)"
signRequest(secret, timestampMs, method, path, rawBody): string
```
`currentMs` exists so a test can pin the clock. `windowMs` on `verifyBody` defaults to
`CHAT_REPLAY_WINDOW_MS`.
## Lane 1: the WebSocket handshake
```ts theme={null}
import { signHandshake, verifyHandshake, nowMs } from "@komaa/standin-sdk";
// Inbound: StandIn dials your call listener. CallServer does this for you.
const ok = verifyHandshake(secret, timestampHeader, callId, signatureHeader);
// Outbound: your worker dials the chat channel. ChatChannel does this for you.
const signature = signHandshake(secret, nowMs(), "chat");
```
The payload is the literal string `` `${timestampMs}.${handshakeId}` ``. For an inbound call the
`handshakeId` is the `callId` from the URL path, which is why `session.callId` is trustworthy: it is
the value the signature covered, not something a caller supplied. A `session.start` body naming a
different `callId` is refused, because that is either a bug or an attempt to ride one call's signature
into another call's session.
For the outbound chat dial the `handshakeId` is the literal channel name, `"chat"`. That is the
inbound handshake scheme running in the opposite direction, and it is deliberately not the body
signer: the POST relay lane signs the body instead, and the two must not be "fixed" into each other.
The window is 60 seconds in both directions. Handshakes are dialed and answered immediately, so
anything older is a replay. A worker whose clock has drifted past a minute will see every call
rejected with 401, and that is usually what a sudden total 401 means.
## Lane 2: body and request signatures
```ts theme={null}
import { signBody, verifyBody, canonicalRequest, signRequest } from "@komaa/standin-sdk";
// v1 body signature: `{timestampMs}.{rawBody}`
const sig = signBody(secret, nowMs(), rawBody);
const ok = verifyBody(secret, timestampHeader, rawBody, signatureHeader);
// v2: binds the method and path as well as a hash of the body
canonicalRequest("POST", "/api/calls", rawBody); // "POST\n/api/calls\n"
const v2 = signRequest(secret, nowMs(), "POST", "/api/calls", rawBody);
```
Sign the **exact bytes you transmit**. Serialize once, sign that string or buffer, and send the same
one. Re-serializing between signing and sending changes key order or whitespace and the signature
stops verifying, which is the single most common plugin bug on this lane.
v2 is preferred for control requests: binding the method and path means a validly signed body cannot
be replayed against a different endpoint. The path is the request path only, without origin or query
string. `/api/calls` above is the real one, and
[`OutboundCaller`](/typescript-sdk/reaching-people) already signs it for you, so you only reach for
`signRequest` when you are writing a control client of your own.
## How both lanes fail closed
* Missing secret, timestamp or signature returns `false`. There is no permissive path.
* The timestamp must be ASCII decimal. `Number()` alone would accept empty strings, hexadecimal and
exponent notation.
* Comparison is constant time, with a length check first. `timingSafeEqual` throws on a length
mismatch, and this code is reachable by anyone who can open a socket, so an unguarded compare would
turn a short signature header into a 500 instead of a 401: both a crash path and an oracle that
tells "malformed" apart from "wrong".
That length check is this language's guard rather than a rule of the scheme; the Python SDK needs a
different one for `hmac.compare_digest`. Both SDKs fail closed, both restrict the timestamp to ASCII
decimal, and the shared conformance vectors assert that a signature produced by either verifies in
the other, byte for byte.
## The replay guard
A correctly signed upgrade replayed inside the freshness window must not open a second socket, so
`CallServer` keeps every accepted handshake fingerprint single-use.
Two details in that guard are worth knowing, because both were bugs waiting to happen:
* The fingerprint uses the **normalized** signature. Verification accepts case and whitespace variants,
so keying on the raw header would let the same capture replay once per casing.
* Entries are aged by the **signing** timestamp, never by arrival time. Verification accepts a
timestamp up to the window in the future, so an entry aged from arrival could be pruned while its
signature was still valid, reopening the exact replay the guard exists to close.
Pruning runs on a time throttle rather than a size watermark, and only correctly signed, not-yet-seen
handshakes ever reach the map, so it tracks StandIn's real call rate rather than attacker traffic.
## Handling the secret
* Keep it in the environment, or in your platform's secret store.
* `STANDIN_SECRET` is what `CallServer` and `OutboundCaller` read. The chat lane reads
`STANDIN_CHAT_SECRET` **first** and falls back to `STANDIN_SECRET`, so a deployment issued a
separate chat credential can run the two lanes on different keys without touching either
constructor. With one key for both, leave `STANDIN_CHAT_SECRET` unset. Every variable the SDK reads
is listed on [Configuration](/typescript-sdk/configuration).
* Never commit it, never log it, and never put it in a URL or query string.
* Rotate it in the StandIn portal. A rotated secret takes effect on the next handshake, so live calls
finish on the old one.
* If a config layer can hand you an unresolved reference, coerce a non-string to the empty string
rather than calling `String()` on it. `String({})` yields `"[object Object]"`, a non-empty and
guessable secret that a naive presence check would accept.
## Who may call the agent
The handshake authenticates the **socket**, not the human. It proves the call came from StandIn.
`session.start` then names a caller, and whether that caller may be answered is a different question
with a different answer: it is your deployment's policy.
The decision stays yours. The **matcher** is in the SDK, so every plugin makes it the same way:
```ts theme={null}
import {
describeInboundRejection,
isInboundCallAllowed,
type CallSession,
} from "@komaa/standin-sdk";
async onStart(session: CallSession) {
const from = session.start.caller.aadId;
if (!isInboundCallAllowed(config.inboundPolicy, config.allowFrom, from)) {
logger.warn(describeInboundRejection(config.inboundPolicy, from));
await session.end("not-allowed");
return;
}
}
```
| Function | What it does |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `isInboundCallAllowed(policy, allowFrom, from)` | `"open"` admits everyone; `"allowlist"` and `"pairing"` consult the list; anything else, including `undefined`, refuses. |
| `isAllowlistedCaller(from, allowFrom)` | True on an exact caller id match, case-insensitive, **or** a digits-only phone number match. |
| `normalizePhoneNumber(input)` | Digits only. |
| `describeInboundRejection(policy, from)` | The log line to print, naming the policy and the fix. |
Three things are worth carrying from the code:
**An unset or unknown policy refuses.** Defaulting the other way would mean a configuration typo
silently opens the agent to anyone who can reach the number.
**The id branch is what makes a Microsoft Teams caller allowlistable at all.** Their id is an AAD
object id, not a phone number, and phone normalization reduces it to the empty string, which matches
nothing. That is why the matcher tries both rather than normalizing everything.
**`caller.aadId` is empty for guest and anonymous callers**, so an anonymous caller can never match an
allowlist. That is the intended outcome, not a gap.
This module is TypeScript only, and it is a real difference between the SDKs rather than a
documentation gap. The Python SDK ships no policy module: a Python worker writes the same test in its
own handler, and the Python call-handler page shows it that way.
## The outbound fetch guard
An agent that can show the caller a picture will sooner or later be handed a URL by its own model,
and that model is steered by whoever is on the call. So the URL is untrusted input wearing a trusted
costume, and a plain `GET` of it is a server-side request forgery: `169.254.169.254` is cloud
credentials, `127.0.0.1` is whatever else you run, and `10.0.0.0/8` is the rest of your network.
```ts theme={null}
import { assertPublicHttpUrl, fetchPublicImage, isForbiddenIp } from "@komaa/standin-sdk";
```
| Name | What it does |
| --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `await assertPublicHttpUrl(url)` | Resolves to a `URL`, or throws naming the problem: a scheme that is not http or https, embedded credentials, no host, or a host that resolves into forbidden space. |
| `isForbiddenIp(address)` | The address test on its own, for a check of your own. Private, loopback, link-local, carrier-grade NAT, multicast and reserved ranges, and an IPv4 address wearing an IPv6 spelling is judged as the address it embeds. |
| `await fetchPublicImage(url, maxBytes)` | The whole guarded fetch, which is what the vision tools use. Bounded on total time, declared length, streamed length and redirects. |
The subtle half is the rebind. Validating a hostname resolves it once, and the resolution the HTTP
client does a moment later can answer differently, so the address the socket **actually** connects to
is re-checked against the same rules. One redirect hop is followed, because image CDNs habitually
redirect to the real asset, and the target goes through the whole guard again rather than being
trusted for having come from a host that passed.
Link-local is on the list by name, because that is where cloud metadata lives.
These three are top-level exports here. In the Python SDK they live in `standin.fetch` rather than on
the barrel. See [Vision and the avatar](/typescript-sdk/vision#fetching-an-image-a-model-chose) for
the image case.
## The generated wire builders
The frame builders and parsers are public, because the wire is public, and they are what the shared
conformance vectors assert across both languages: `audioFrame`, `sessionEnd`, `pong`,
`assistantCancel`, `parseMessage`, `parseSessionStart` and `decodePcm`. All of them are on the barrel
here; in the Python SDK only `assistant_cancel` is, and the other six are
`from standin.protocol import ...`.
`contextSentences` sits with them on the barrel and is the one that has no Python symbol at all: the
same three sentences exist there, word for word, but they are built inside the Python call server
rather than exposed as a helper. So a test that asserts on the exact wording can import it here and
has to retype it there.
A handler should still not reach for them. `CallServer` owns the sequence number and the outbound
timeline, and a hand-built frame is how those two stop agreeing.
## Transport and limits
Terminate TLS in front of the worker. The signature proves origin and freshness, it does not encrypt
anything, and the audio on the wire is unencrypted PCM inside the WebSocket frame.
A single inbound message is bounded at 2 MB on both the call and chat lanes, and a call-outcome POST
is bounded at 8 KB. A malformed URL, a bad percent escape, or an unparseable frame rejects that one
request and never escapes to kill the process.
## Next
Admission order, the replay guard in place, and the watchdogs.
Every `STANDIN_` variable, including both secrets.
The outbound dial, and the cross-tenant guard on every reply.
The outbound allowlist, which is stricter than any inbound one.
# Sending a picture
Source: https://docs.komaa.com/typescript-sdk/sending-pictures
Attach a checked image to a chat reply, and turn a MEDIA: marker into bytes worth sending, in the StandIn TypeScript SDK.
A picture your agent sends travels as **bytes**, never as a link. The reason is worth stating once,
because every shortcut here leads back to it: a link in a chat message is a beacon. An off-domain
image loads with no click, under your bot's name, it reports back who opened the conversation and
when, and whatever it serves can be swapped after everyone has looked at it. Bytes cannot do any of
that, and bytes can be checked before they leave.
So there is exactly one way to build a picture this SDK will send, and that one way does the checking.
This page is about pictures your agent **sends**. Putting a picture on the bot's video tile during a
call is a different lane with different limits: see [Vision and the avatar](/typescript-sdk/vision).
## `outboundImage`
```ts theme={null}
import { outboundImage } from "@komaa/standin-sdk";
const picture = outboundImage(pngBytes, "image/png", "q3-revenue.png");
```
It takes a `Buffer` or an already-base64 `string`, a declared content type, and an optional filename.
It returns an `OutboundImage`, which is the only shape the chat lane will carry:
```ts theme={null}
interface OutboundImage {
readonly contentType: string;
readonly contentBase64: string;
readonly name?: string;
}
```
It throws rather than returning something half-checked, and it runs its checks in this order:
1. **The declared type is on the allowlist.** `image/jpg` is folded to `image/jpeg` first, because that
spelling is common and is not a media type. Anything else throws naming the four that work.
2. **The bytes are within the cap.** Checked on the decoded length, so a base64 string that expands
past the limit is refused.
3. **The bytes carry the signature of the type they claim.** A declared type is a claim made by
whatever produced the bytes. The signature is what they are. Without this check an HTML document
or an SVG labelled `image/png` is posted into somebody's chat under your bot's name.
4. **The filename is sanitized**, which never throws: a name that cannot be made safe is simply dropped.
Step 3 is the one that matters, and it is strict in both directions: the sniffed type must **equal**
the declared type. A GIF labelled `image/png` is refused, not quietly relabelled. If you do not know
what you are holding, sniff first and pass the answer back in, which is exactly what `loadMedia` does
below.
That check is also the only thing standing behind a base64 `string`, because
`Buffer.from(data, "base64")` is lenient: Node discards characters that are not base64 rather than
objecting, so a truncated or scrambled string decodes quietly to fewer bytes and arrives at step 3 as
the wrong bytes. The Python SDK decodes strictly and raises on that same input, so do not port a test
that expects the decode itself to do the rejecting.
Every refusal on this page, from `outboundImage` and `loadMedia` alike, is a plain `Error` rather than
a `StandInError`, so an `instanceof StandInError` filter will not catch one. Catch `Error` and read
`err.message`: the messages are written to be handed straight back to whatever chose the bytes. The
Python twin raises `ValueError` on the same paths.
## What may be sent
| | Chat reply | On the call tile |
| -------- | ---------------------------------------------------- | ---------------------------------- |
| Constant | `OUTBOUND_IMAGE_CONTENT_TYPES` | `DISPLAY_IMAGE_MIME_TYPES` |
| Types | `image/png`, `image/jpeg`, `image/gif`, `image/webp` | `image/jpeg`, `image/png` |
| Cap | `OUTBOUND_IMAGE_MAX_BYTES`, `1024 * 1024` bytes | `MAX_IMAGE_BYTES`, `1400000` bytes |
Every constant in that table is importable from the barrel, so a check of your own can use the same
number the SDK does rather than a copy that drifts:
```ts theme={null}
import {
OUTBOUND_IMAGE_CONTENT_TYPES,
OUTBOUND_IMAGE_MAX_BYTES,
} from "@komaa/standin-sdk";
```
`image/svg+xml` is absent from the chat list deliberately, and it is not a gap to fill later. SVG is
scriptable XML, which is precisely what "an image" must not be when it renders inside somebody else's
client under your bot's identity.
## `sniffImageType`
```ts theme={null}
import { sniffImageType } from "@komaa/standin-sdk";
const actual = sniffImageType(bytes);
// string | undefined: "image/png", "image/jpeg", "image/gif" or "image/webp"
```
It reads the leading bytes and returns what they actually are, or `undefined` when they are none of
the four. WebP gets a second look: a `RIFF` header is only a WebP when bytes 8 to 11 say `WEBP`, and
a RIFF container that holds audio would otherwise sail through on its first four bytes.
Use it whenever the type you have came from somewhere you do not control: a `Content-Type` header, a
file extension, or a field a model filled in. All three are claims.
## `sanitizeImageName`
```ts theme={null}
import { sanitizeImageName } from "@komaa/standin-sdk";
sanitizeImageName("../../etc/passwd"); // "passwd"
sanitizeImageName("report:final?.png"); // "reportfinal.png"
sanitizeImageName(".."); // undefined
```
A filename reaches a chat as a **download**, and the model that chose it is being steered by whoever
is in the conversation. So the name is reduced to one path-free segment: backslashes are folded to
forward slashes and only the last segment survives, every character below a space is dropped along
with `<>:"|?*`, and what is left is trimmed. An empty result, `.` or `..` returns `undefined`, which
means the picture is sent with no name rather than with a dangerous one.
Names longer than 200 characters are truncated, keeping the extension so the file still opens with
the right application. `outboundImage` calls this for you, and it returns `undefined` rather than
throwing, because a bad filename is never a reason to drop an answer.
## Getting the picture into a reply
Two paths, and they differ in how many messages the person sees.
**One message carrying both.** `buildReply` takes the image as its fourth argument, after the kind:
```ts theme={null}
import { buildReply, outboundImage, parseInbound } from "@komaa/standin-sdk";
const message = parseInbound(rawBody);
const picture = outboundImage(pngBytes, "image/png", "q3-revenue.png");
const reply = buildReply(message, "Here is Q3.", "message", picture);
```
This is the path when you relay messages yourself rather than through `ChatChannel`. A `typing` reply
carries neither text nor image: it is a state, not a message.
**A second message from inside a handler.** `ChatChannel`'s own reply path posts the string your
`respond` returns, and that string carries no picture. `chat.send()` is how you post one:
```ts theme={null}
import { ChatChannel, outboundImage } from "@komaa/standin-sdk";
const chat: ChatChannel = new ChatChannel({
respond: async (message) => {
const { words, chart } = await agent.ask(message.text);
if (chart !== undefined) {
await chat.send({
tenantId: message.tenantId,
conversationId: message.conversationId,
text: "Here is the chart.",
image: outboundImage(chart, "image/png", "chart.png"),
bindingId: message.bindingId,
idempotencyKey: `${message.activityId}:chart`,
});
}
return words;
},
});
```
Your handler must return something more than whitespace. The channel reads a blank answer as a failed
turn and posts an error line, so you cannot post the picture yourself and then return `""` to suppress
the text. Because `send` runs inside your handler, the picture lands **before** the words: give the
picture its caption, as above, and return the words that follow it. If you need the caption and the
picture in one message, build the reply yourself with `buildReply`.
`send` is best-effort and returns `false` rather than throwing, since a failed post must never break a
live call. Pass an `idempotencyKey` whenever a retry is possible. `bindingId` says which StandIn
connection the message is from, and one organisation can have several, so echo the inbound value
rather than leaving it out.
## `MEDIA:` markers
Some agent frameworks let a reply name a file by writing a line like `MEDIA:/tmp/chart.png` and expect
the channel to attach it. That marker is an **instruction to the channel**, not something anyone
should see. A channel that does not understand the convention posts the line as prose, and the person
reads a temporary file path in their chat. On a call it is worse: text to speech reads the path out,
character by character.
`parseMedia` takes the markers out and hands back what they referred to. `loadMedia` turns one
reference into bytes, through the same guards everything else in this SDK uses.
### `parseMedia`
```ts theme={null}
import { parseMedia } from "@komaa/standin-sdk";
const { text, refs } = parseMedia(await agent.ask(question));
// text: "Here is the chart."
// refs: ["/tmp/chart.png"]
```
The returned `text` is what to post **and** what to say. Both, always: the whole point is that nobody
sees or hears the marker. `refs` holds what the markers pointed at, in the order they were written.
The shape is exported as `AgentMedia`.
Four things the parser does on purpose:
* **It is case-insensitive and one marker per line.** `MEDIA:`, `media:` and `Media: ` all match.
* **It is anchored to the end of the line**, so a path with spaces survives intact. The documented
form is backticked for exactly that reason, and the backticks are stripped: `` MEDIA:`/tmp/a b.png` ``
yields `/tmp/a b.png`. Splitting on whitespace would truncate that path at the first space.
* **A line that is not plausibly a reference is left alone.** A reference has to start with `http://`,
`https://`, `/`, `./`, `../`, `~/` or a Windows drive letter, or end in `.png`, `.jpg`, `.jpeg`,
`.gif` or `.webp`. Without that narrowing, a sentence like "MEDIA: we should talk to them about it"
vanishes out of the answer and whoever wrote it never learns why.
* **The blank line a removed marker leaves is collapsed.** Three or more newlines become two, which is
a paragraph break. Two are left alone, because they already are one.
Because the marker runs to the end of its line, nothing may follow the path on that line. Write
`MEDIA:/tmp/chart.png` alone; `MEDIA:/tmp/chart.png and here it is` makes the whole trailing sentence
part of the reference, and then `loadMedia` fails on a file that does not exist.
### `loadMedia`
```ts theme={null}
import { loadMedia, parseMedia } from "@komaa/standin-sdk";
const { text, refs } = parseMedia(reply);
for (const ref of refs) {
try {
const picture = await loadMedia(ref, { roots: ["/srv/agent-output"] });
// picture is an OutboundImage, ready for buildReply or chat.send
} catch (err) {
// Worth logging and worth saying: the agent chose this reference, and the
// reason is a sentence it can act on next turn.
console.warn(`could not attach ${ref}: ${String(err)}`);
}
}
```
Catch per reference, as above. One reference that will not load should cost that picture, never the
whole answer: `text` is still worth posting, and the reason is a sentence the agent can be told.
The SDK's own `logger` is not exported, and there is no `@komaa/standin-sdk/log` subpath to reach it
through, so a sample like this uses `console` or your own logger. What the barrel does export is
`setLogger`, which replaces the one the SDK writes its own lines with: pass an object with `debug`,
`info`, `warn` and `error` and the SDK's lines land wherever the rest of your worker's do. The Python
twin takes the other route and exposes `standin.log.logger` directly.
It resolves one reference and returns an `OutboundImage`. It throws with something worth reading,
because the caller is on a path where the alternative is a dropped answer with no explanation.
Options are `LoadMediaOptions`:
| Option | Default | Meaning |
| ---------- | ------------------------------- | --------------------------------------------------------- |
| `roots` | `STANDIN_MEDIA_ROOTS` | Directories a local reference may be read from. |
| `maxBytes` | `OUTBOUND_IMAGE_MAX_BYTES` | The size ceiling, applied to a fetch and to a file alike. |
| `name` | the basename of the path or URL | The filename the picture is sent with. |
What it does, and why each branch exists:
* **`http://` or `https://`** goes through the SDK's own fetch guard with a ten second budget, so a
reference aimed at a private, loopback, link-local or reserved address is refused rather than
fetched. The address the socket actually connects to is re-checked, which closes the window where a
hostname resolves publicly for the validation and privately for the fetch. See
[Fetching an image a model chose](/typescript-sdk/vision#fetching-an-image-a-model-chose).
* **Any other scheme is refused by name**, including `file://` and `data:`. Handing a `file://` URL to
a URL fetcher is the usual way around a path guard, so it is rejected before anything opens it.
* **Everything else is a local path**, read only from a directory an operator named, and only after a
`stat`: the size is checked **before** the read, because the point of a cap is not to load the file.
Whatever the branch, the bytes are sniffed at the end and the sniffed type is what gets sent. A local
file declares no type at all, so the sniff simply decides. A fetch does declare one, and a declared
type that disagrees with the bytes throws instead of being relabelled, with only
`application/octet-stream` accepted as "no opinion". A type from a response header is a claim; the
bytes are the fact.
A response with no `content-type` header at all is read as `image/jpeg`, so a PNG served bare is
refused with `that was served as image/jpeg but the bytes are image/png`. That is the fetcher's
default, not a sniff, and it is the one refusal on this list that usually means the server is wrong
rather than the picture. Fix the header, or fetch the bytes yourself and hand them to
`outboundImage`.
| What you passed | What comes back |
| ------------------------------------------ | ------------------------------------------------------------------------------- |
| `""` or whitespace | `there was nothing to send` |
| A local path with no roots configured | `sending a local file is off until a directory is named in STANDIN_MEDIA_ROOTS` |
| A path outside the roots | `that file is outside the directories this worker may read` |
| `file://`, `data:`, `s3://` | ` references are not allowed here` |
| A missing file | `no such file` |
| A file over the cap | `that file is N bytes, over the M limit` |
| A text file, or an SVG | `that file is not a picture this can send` |
| A URL served as one type, bytes of another | `that was served as X but the bytes are Y` |
### `mediaRoots` and `STANDIN_MEDIA_ROOTS`
```ts theme={null}
import { MEDIA_ROOTS_ENV, mediaRoots } from "@komaa/standin-sdk";
MEDIA_ROOTS_ENV; // "STANDIN_MEDIA_ROOTS"
mediaRoots(); // [] until an operator sets it
mediaRoots(["/srv/charts"]); // or pass them, bypassing the environment
```
`mediaRoots` returns the directories a local reference may be read from, resolved through symlinks.
With no argument it reads `STANDIN_MEDIA_ROOTS`, split on the platform path delimiter: `:` on Linux
and macOS, `;` on Windows. `~/` is expanded. A root that cannot be resolved, because nothing is there,
is dropped, since a directory that does not exist cannot contain anything.
**It is empty by default, and that is the whole design.** Sending a local file is off until somebody
opts in. An agent can be talked into writing `MEDIA:/etc/passwd`, and the answer to that has to be a
refusal rather than a file read followed by an upload into somebody's chat.
Two details in the containment check earn their keep:
* **Both sides are resolved through symlinks first.** A symlink inside a root that lands outside it is
judged by where it lands, so dropping a link named `innocent.png` into an allowed directory buys
nothing.
* **The comparison is separator-terminated.** Without that, `/tmp/rootevil` passes for `/tmp/root`,
and one adjacent directory becomes readable to a path a model chose.
Name the narrowest directory that works, and make it one your agent writes into rather than one it
merely happens to be able to read:
```bash theme={null}
export STANDIN_MEDIA_ROOTS=/srv/agent-output
```
Pass `roots` explicitly instead when a single worker serves more than one purpose. An explicit list
wins over the environment entirely, so the variable is not consulted at all on that call. It is still
resolved the same way: a root you pass that does not exist is dropped exactly as one from the
environment would be, so an explicit list is not a way around the check.
## Next
The lane a picture travels on, and what it echoes back.
Every variable the SDK reads, including the roots above.
# Vision and the avatar
Source: https://docs.komaa.com/typescript-sdk/vision
What the caller shows your agent, what your agent shows back, and the face the caller sees, in the StandIn TypeScript SDK.
A Microsoft Teams call carries more than voice. StandIn samples the caller's camera and their screen share and forwards single frames to your worker, and it will draw an image you send onto the bot's own tile.
Both halves are optional. A plugin that only wants to talk never touches any of this.
## Frames arrive sparsely
StandIn drops a frame rather than queueing it when the socket is busy. This is **not** a video stream and must not be treated as one.
The useful shape, and the one every provider plugin ends up with, is to keep the latest frame and look only when something asks:
```ts theme={null}
const frame = session.latestVideoFrame(); // screen share wins over camera
if (frame !== undefined) {
const answer = await describer.describe(frame, "What is on the slide?");
}
```
The server keeps the latest frame per source for you, so a plugin that only wants on-demand vision implements no callback at all. Pass `"camera"` or `"screenshare"` to be explicit; with no argument the screen share wins, because an agent asked to look is nearly always being asked about what is being *shown* rather than who is showing it.
Take every frame instead, which is what [ambient vision](#ambient-vision) needs, by implementing the optional sixth method:
```ts theme={null}
class MyHandler {
async onVideoFrame(frame: VideoFrame) {
// ...
}
}
```
`onVideoFrame` runs on the receive path of a live call. Do a slow model call off the frame loop, exactly as you would in `onCallerAudio`.
Every method on `CallHandler` is optional in TypeScript, so `onVideoFrame` is simply one more of them. The Python twin keeps it on a separate protocol, because its `CallHandler` is runtime-checkable and a sixth member there would break `isinstance` for every handler written before the vision lane existed.
## What a frame is
| Attribute | What it is |
| ---------------------------------- | ------------------------------------------------------------------------------------ |
| `source` | `"camera"` or `"screenshare"`. |
| `width`, `height` | Pixels, already downscaled by StandIn. |
| `mime` | `image/jpeg`. |
| `dataBase64` | The image exactly as it arrived. |
| `data` | The decoded bytes, for an API that uploads a file. |
| `dataUrl` | A `data:` URL, which is what most vision models take. |
| `participantId`, `participantName` | Who it belongs to. Best-effort, and **absent** for guest and anonymous participants. |
An unusable frame is dropped rather than raised. The call is healthy, the caller is still talking, and one malformed image is not worth ending a conversation over.
`dataBase64` is the field kept verbatim and the other two are derived from it, because base64 is the form most providers want back. That also makes the drop check exact: `parseVideoFrame` re-encodes what it decoded and compares, since `Buffer.from(value, "base64")` never throws on bad input and instead silently discards what it cannot read.
### Who it came from, and whether it changed
`VIDEO_SOURCES` is the pair `source` may hold, `["camera", "screenshare"]`, and the `VideoSource` type is derived from it. A message naming anything else never becomes a `VideoFrame` at all, so you never have to defend against a third lane appearing.
Four helpers turn a frame into something a model can be told about:
| Function | What it gives you |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `frameOwner(frame)` | The participant name the wire carried, or `undefined`. |
| `fallbackOwner(source)` | `"a participant"` for a screen share, `"the caller"` for a camera. |
| `frameCaption(owner)` | The sentence that goes beside the picture, built from that owner alone: `screen shared by a participant`, or `camera of Dana`. |
| `frameDigest(dataBase64)` | A 32 character fingerprint of the frame. |
Attribution **degrades rather than vanishing**. Guest and anonymous participants arrive with no name, and `fallbackOwner` gives those a generic label instead of nothing, because "a participant's screen" is worth more to a model than an unlabelled picture. Dropping the attribution instead would leave a model in a group call unable to say whose deck it is looking at. `FrameDescriber` composes the two for you, and so does ambient vision, both of them as `frameOwner(frame) ?? fallbackOwner(frame.source)`.
The digest answers one question, "is this the same screen as last time?", without keeping the picture to compare against. It hashes the base64 rather than the decoded bytes: two encodes of an unchanged screen are byte-identical, so it is exact, and it costs no decode on a path that sees every frame. The keyframe store uses it to avoid filling itself with one frozen slide, and ambient vision latches on it.
## Giving a voice model eyes
Most speech-to-speech providers hear but cannot see. `FrameDescriber` closes that gap: the frame goes to a vision model of your choosing and a sentence comes back.
```bash theme={null}
export STANDIN_VISION_API_URL=https://api.openai.com/v1/chat/completions
export STANDIN_VISION_MODEL=gpt-4o-mini
export STANDIN_VISION_API_KEY=...
```
```ts theme={null}
import { FrameDescriber } from "@komaa/standin-sdk";
const describer = FrameDescriber.fromEnv(); // undefined when not configured
```
Any OpenAI-compatible chat-completions endpoint that accepts images works, including one you run yourself. The frame is sent for inference and **not stored**, which is the difference between this and uploading it into a provider's own conversation history.
`describe` throws on anything that goes wrong, so a caller can hand the reason back to the agent that asked, and it puts `frameCaption` into the question it sends, so the model is told whose screen it is looking at before it is asked anything about it.
This URL is deliberately **not** put through the SDK's fetch guard. It is yours, set by you in the environment, and a vision model on localhost is a normal way to run one. That is the opposite of an image URL a model chose, which is untrusted and does go through the guard.
## Showing the caller something
```ts theme={null}
await session.displayImage(chartPng, {
mime: "image/png",
durationMs: 4000,
caption: "Q3 revenue",
});
```
The image is drawn on the bot's video tile for a few seconds, then the avatar returns. `mode` is `"fullscreen"` or `"overlay"` for a picture-in-picture inset.
A Buffer or an already-base64 string both work. An image that is too large, or a type the service will not draw, is refused here where the error names the problem, rather than letting the service close the socket in the middle of a call.
`DISPLAY_IMAGE_MIME_TYPES` is the whole list of what StandIn will draw, `["image/jpeg", "image/png"]`, and `MAX_IMAGE_BYTES` is `1400000`. That ceiling is not arbitrary: one wire message is bounded at 2 MB by both SDKs and base64 costs a third on top of the raw bytes, so 1.4 MB of image is what actually fits inside the envelope. A string you pass is measured **decoded**, because the decoded size is what the envelope bounds. Both names are at the barrel here; on the Python side they live in `standin.vision`.
`MAX_IMAGE_BYTES` is the **outbound** ceiling, on what you draw on the tile. Inbound chat attachments have a separate and larger one, and passing this constant where that limit belongs compiles cleanly and quietly caps them at 1.4 MB. See [Chat attachments](/typescript-sdk/chat-attachments#images).
### Fetching an image a model chose
An agent that can show a picture will sooner or later be handed a URL by its own model, and that model is steered by whoever is on the call:
```ts theme={null}
import { fetchPublicImage, MAX_IMAGE_BYTES } from "@komaa/standin-sdk";
const { bytes, mime } = await fetchPublicImage(url, MAX_IMAGE_BYTES);
```
http and https only, no embedded credentials, and no host that resolves into private, loopback, link-local or reserved space. The address the socket actually connects to is re-checked, closing the window where a hostname resolves publicly for the validation and privately for the fetch. One redirect hop is followed, because image CDNs habitually redirect to the real asset, and the target goes through the whole guard again.
## The tool surface
`VisionTools` is the layer a model reaches for, and it lives in the core because every provider wants the same things and none of them should write the guards again. Build one per call and route your provider's tools at it.
| Method | What it does |
| ------------- | ----------------------------------------------------------------- |
| `look` | Answer a question about what the caller is showing now. |
| `lookBack` | Answer about a frame the caller has already moved past. |
| `show` | Put an image you already have on the bot's tile. |
| `showUrl` | Fetch an image the model chose, through the guard, and show it. |
| `showMany` | Show several pictures in turn, paced, without blocking the model. |
| `walkthrough` | Say and show several of those in order, pausing for each. |
**None of them raise at a model.** Every one returns a sentence, because the caller is a tool result being read back to something that will say it out loud, and an exception there is a silent tool and a confused agent.
`VisionToolsOptions` is the whole constructor surface: `describer`, `budget`, `keyframes` and `defaultDisplayMode`. Pass none and you get a `VisionBudget` and a `KeyframeStore` of its own and no describer, and `look` says so in a sentence rather than failing. `reset()` stops anything running and forgets what was shown, which is what teardown calls.
Two guards travel with them.
**A budget**, because a model that can look can look in a loop and each look is a paid inference over somebody's screen. It is a sliding window, so a long call is not punished for having been long, and a failed look is refunded rather than charged. Spending returns a token and refunding takes that token back: two tool calls overlap, and refunding "the most recent charge" would refund the wrong one.
`new VisionBudget(6)` is the default ceiling per rolling minute. `tryConsume()` takes one look's worth or returns `undefined`, `refund(token)` gives one back and is idempotent, and `spent` is what has been charged in the current window. A `maxPerMinute` of zero means no cap at all, which is a deliberate choice rather than a default.
**A keyframe store**, so "what did that slide say?" can be answered about a slide that is already gone. It is bounded, and it only keeps anything **while the call is being recorded**: keeping a history of somebody's screen is a materially different promise from glancing at it once, and the recording is what told the caller their call is being kept.
`new KeyframeStore(16)` is the default capacity. `offer(frame, recording)` returns whether the frame was kept, `recent(source?)` reads the history oldest first, and `size` and `clear()` are the rest of it. It latches per source on `frameDigest`, so a screen nobody touched is kept once instead of filling the whole store with one picture.
### Fullscreen or beside your face
Every showing method takes a display mode, and the model chooses it per picture: `fullscreen` for something being read, `overlay` to keep the avatar beside it. `show_image` carries the choice as an enum in its schema, which a model obeys far more reliably than the same list written into a description.
Anything else a model says, and `pip` and `full` are both things it will say, falls back to the default you configured. When you configured none, the field is left off the wire entirely and the service decides, rather than this SDK deciding for it.
```ts theme={null}
const tools = new VisionTools(session, { describer, defaultDisplayMode: "overlay" });
await tools.showUrl(url, "Q3 revenue", "fullscreen");
```
`DISPLAY_MODES` is that pair, `["fullscreen", "overlay"]`, and `normalizeDisplayMode(value, fallback?)` is the single rule that turns whatever a model said into one of them or into the fallback. It is exported because every plugin has to apply the *same* rule: a provider that mapped `pip` to `overlay` on its own would give one model a placement the other providers refuse, and one prompt would then behave differently on two deployments.
What the caller can actually see is on `tools.lastShown`, recorded only after a send resolved. So "email me that" attaches what they saw rather than what was attempted. It is a `ShownImage`, carrying the `image`, its `mime`, the `name` shown beside it and the `atMs` it went up, with `asBase64()` for an API that wants the encoded form. One slot, replaced each time, because a list would be a growing copy of everything shown on the call.
The name comes from `displayImageName(pathOrUrl, mime)`. It takes the last path segment with any query and fragment cut off, and only when what is left already looks like a filename: a name, a dot, a short extension, no separators of any kind. Anything else becomes `image.png` or `image.jpg` built from the type. `../../etc/passwd` is not a filename, and this string was chosen by a model and is about to be drawn on the caller's screen.
A caption is trimmed to 200 characters before it reaches the screen, for the same reason the name is checked: a model's strings are as long as whoever is steering it wants them to be, and this one is drawn on somebody's call.
### Several pictures in a row
```ts theme={null}
await tools.showMany(
[
{ image: chartPng, mime: "image/png" },
{ image: "https://example.com/table.png" },
],
"Q3",
"fullscreen",
4000,
);
```
Each entry is a `ShowItem`: an `image` that is bytes, base64 or an https URL, an optional `mime` defaulting to `image/jpeg`, and an optional `name`. In TypeScript that is an interface, so an object literal is the whole construction; the Python twin is a dataclass you name.
The first picture is on the tile before the call resolves, so the model can say "here it is" and be right. The rest are paced from a detached chain. A model that waited out a ten-picture slideshow before speaking would leave the caller in silence for most of a minute.
`MAX_SLIDESHOW_IMAGES` pictures at a time, which is ten, and the sentence that comes back says when there were more: a model asked for "the slides" can mean forty of them. `holdMs` defaults to `SLIDESHOW_HOLD_MS`, four seconds, and whatever you or a model passes is clamped to between one and thirty seconds.
Each non-final picture is held past the pacing gap, by `SLIDESHOW_OVERLAP_MS` (half a second, at the barrel here and in `standin.vision_tools` on the Python side), so the next one arrives before the last has expired and the tile never blanks between two of them. The last carries no duration at all, so what stays on screen is the service's own default. Each picture is also loaded inside the pacing loop rather than up front, so a slideshow of ten URLs does not fetch all ten before the first appears, and one that fails skips that picture rather than the rest.
There is one tile, so starting a slideshow or a walkthrough stops whatever was already running on it. The one being replaced is woken rather than killed: a picture already being sent finishes being sent, and the new first picture goes up immediately instead of waiting out the old gap. `tools.slideshow` is the promise for the one now running, when you want to let it finish.
### Saying each one as it goes up
`walkthrough` is the narrated version, and a `WalkthroughStep` is one beat of it: a line to `say`, and optionally an `image` with its `mime` and `caption` to put up once that line has been said.
```ts theme={null}
import type { WalkthroughStep } from "@komaa/standin-sdk";
const steps: WalkthroughStep[] = [
{ say: "First, last quarter.", image: q3Png, mime: "image/png" },
{ say: "And here is the forecast.", image: forecastPng, mime: "image/png" },
];
await tools.walkthrough(steps, (text) => this.say(text), () => this.callerIsTalking);
```
`speak` is yours, a `Speaker` of `(text: string) => Promise`, because "the line has finished being said" is a thing only your provider knows, and a walkthrough that guesses talks over itself. `interrupted` is checked between beats, for the same reason: somebody who cuts in has stopped the tour, and only the plugin can tell that they did. A step whose line or picture fails stops the tour there and names the step, rather than narrating pictures nobody is seeing.
### Showing a web page
There is no browser in this SDK and there will not be one. `showPage` is the seam: you supply the renderer, it supplies the guard.
```ts theme={null}
import { CallTools, SHOW_PAGE_TOOL, VisionTools } from "@komaa/standin-sdk";
const vision = new VisionTools(session, { describer });
const tools = new CallTools(session, { vision });
tools.register(SHOW_PAGE_TOOL, (p) => vision.showPage(String(p.url), undefined, myRenderer));
```
`register` is on `CallTools`, not on `VisionTools`: one model sees one list of tools, so a tool of your own is added where the built-ins already are. The handler itself reaches straight into the vision tools.
Your renderer is a `PageRenderer`: it takes a URL and returns bytes plus a mime type, never a path. A path would make this SDK read a file chosen downstream of whoever is on the call, which is not a thing it should be able to do.
The URL goes through the same public-address guard as every other URL a model chose, and it goes through it **even when your renderer is a browser with private-network protection of its own**. That protection assumes whoever wrote the URL already has a shell on the machine. Here the URL was written by a model a stranger is steering, which is precisely the case the relaxation lets through.
`PAGE_RENDER_TIMEOUT_MS` is forty-five seconds and is `showPage`'s fourth argument when you want a different one. A renderer past it is not cancelled, because nothing here can cancel a host browser; the timeout is what stops the caller waiting in silence, and what comes back counts whole seconds, never `1 seconds` and never `0 seconds`, because it is read out loud to the person who is waiting. `PAGE_DISPLAY_MS` is fifteen seconds, longer than a chart gets, because a page is read rather than glanced at, and with no caption of your own the URL is used, cut to 80 characters.
The Python twin spells that first one `PAGE_RENDER_TIMEOUT_S` and holds `45.0`. The same forty-five seconds, a different units suffix, and a value that cannot be copied between the two SDKs.
`showPage` is deliberately not one of the built-in tools. The built-in list is declared unconditionally, so putting it there would tell every model on every deployment that it can show a web page, and then apologise to the caller on every call where no renderer exists. Register it when you have one.
Putting a **local document** on the tile is Python only today: `show_file` and its `STANDIN_SHOW_ROOTS` allowlist have no TypeScript twin. These tools cover looking, showing an image you have, showing one from a URL, and showing a page you render.
## Ambient vision
`look` answers when a model reaches for it, which is the right shape most of the time and has one blind spot: the model has to know there is something to look at. Somebody who shares a deck and says "what do you think?" has told it nothing it can act on.
`AmbientVision` closes that by pushing what changed on screen into the conversation between turns. It is **off unless a plugin turns it on**, because it spends money on every scene change and not every deployment wants that.
```ts theme={null}
import {
AmbientVision,
FrameDescriber,
VisionTools,
ambientImageDataUrl,
type AmbientImage,
type CallSession,
type VideoFrame,
} from "@komaa/standin-sdk";
class MyHandler {
#tools!: VisionTools;
#ambient!: AmbientVision;
async onStart(session: CallSession) {
this.#tools = new VisionTools(session, { describer: FrameDescriber.fromEnv() });
this.#ambient = new AmbientVision(session, (image) => this.#push(image), {
enabled: true,
budget: this.#tools.budget,
});
}
async onVideoFrame(frame: VideoFrame) {
this.#ambient.offer(frame);
}
async aclose(reason: string) {
await this.#ambient.close();
}
async #push(image: AmbientImage) {
// Into the provider's context, without asking it to answer.
await this.provider.addImage(ambientImageDataUrl(image), image.caption);
}
}
```
`offer` is synchronous, never blocks and never throws: it runs on the receive path of a live call, so all it does is keep the frame and wake the pass that does the work. That pass is *scheduled* rather than started inline, because calling an async function runs its body up to the first `await` synchronously, and a pass begun inside `offer` would do vision work on the frame loop and deliver frames out of order. `flush()` asks for a pass now and is safe to call from anywhere. `close()` stops it permanently and forgets what it was holding; the Python twin spells that one `aclose()`.
The session you hand it only has to satisfy `AmbientSession`, which is `recordingActive` and nothing else, so a fake in a test is one field. The options are `AmbientVisionOptions`. Neither type has a Python counterpart to import: there the options are keyword arguments, and the session is read for `recording_active` rather than typed at all.
### The sink is yours
`AmbientSink` is the one thing you supply, `(image: AmbientImage) => Promise`. What to DO with a picture stays in the plugin, because only the plugin knows how to hand one to its provider.
Two rules on it, and both are load bearing.
**It must not make the agent reply.** Ambient vision is context. An agent that answers every scene change talks over the person presenting, which is worse than not having looked.
**It must reject when delivery fails.** A sink that swallows its own error latches a frame that never arrived, and the model never sees that screen again.
`AmbientImage` is the frame with the decisions already made: `source`, `mime`, `dataBase64`, `width`, `height` and `ts` as they arrived, plus the `owner` and the `caption` that says whose screen this is. `ambientImageDataUrl(image)` builds the `data:` form for the providers that take one. It is a free function here because `AmbientImage` is an interface that a plain object satisfies; the Python twin is a dataclass and carries the same thing as an `image.data_url` property.
### What keeps it from being expensive or creepy
**The recording gate is checked before a frame is even stored.** Streaming somebody's screen to a model is a materially different promise from glancing at it once, and the recording is what told them their call is being kept. Storing the frame and gating only the send would mean that turning the recording on surfaces something from before the caller was told. Set `requireRecording: false` only where something else made that promise.
**The change latch is the last frame actually delivered**, per source, not the last one seen. A screen nobody touched costs nothing, and a delivery that failed leaves the latch alone and refunds its charge, so the same screen is retried rather than skipped. `changeKey` replaces how sameness is judged; by default it is `frameDigest`.
**The reserve stops the ambient lane spending the budget the caller's own request needs.** `VisionBudget.reserve` is a quarter of the window and never less than two, and the ambient lane is refused once the window is down to it, while an explicit `look` can still spend. A refused ambient push stops the whole pass rather than trying the next source, which would only spend the same exhausted budget. Hand `AmbientVision` the same `VisionBudget` your `VisionTools` holds, as the example above does, or that reserve protects nothing: given no budget it builds its own at `DEFAULT_AMBIENT_MAX_PER_MINUTE`, thirty a minute, and the two lanes are then capped independently instead of sharing one window. Handing it a budget with no ceiling at all, while enabled, logs a warning: every scene change is then charged.
### Pacing, and holding what cannot go yet
`AMBIENT_SOURCE_ORDER` is `["screenshare", "camera"]`. When both changed, the screen share goes first: somebody presenting is nearly always talking about the screen rather than about their face.
`AMBIENT_BACKSTOP_MS`, six seconds, is how often it looks again when no frame arrived at all. A screen share can go quiet without ending, and the last thing on it is still what is being discussed. The timer is armed by the first frame that gets past the gate, so a call with no video never starts one, and `backstopMs` of your own is floored at half a second.
`sinkReady` covers a provider whose socket comes up after the call does. While it answers no, images are charged, latched and held rather than dropped, because they are going to be sent and refunding would re-send the same screen once the sink came up. What was held goes out oldest first once it answers yes, so the model sees the screen change in the order it happened. The queue is bounded at `MAX_QUEUED_AMBIENT_IMAGES`, six: a sink that never comes up would otherwise hold the whole call's video, and the oldest is what goes, because what is on screen now is worth more than what was. A held image that fails on its way out is dropped rather than requeued, or a dead sink grows the queue for ever.
`onDelivered` is called with each image that actually landed, which is what a meeting recap hooks so the minutes can say what was on screen, and a throw inside it never fails a delivery that already happened. `delivered` and `queued` count those two things for a health check.
## The avatar
StandIn renders the avatar tile, and your worker steers it with two additive hints: the emotion the face wears, and the viseme timeline that makes the mouth match the words. Those, and putting your agent's own video on the tile in place of the rendered avatar, are on their own page: [The avatar](/typescript-sdk/avatar).
# Turn-taking
Source: https://docs.komaa.com/typescript-sdk/voice
VoiceLane, utterance segmentation, WAV handling and paced playback for an agent that is not a realtime model, in the StandIn TypeScript SDK.
A realtime speech-to-speech provider is handed the caller's audio and hands audio back, and the turn-taking is theirs. Everything else is not like that.
A transcription service wants one utterance at a time. A language model wants text. A text-to-speech engine hands back a whole buffer that somebody has to feed out at the rate a call consumes it. That shape is the same whichever three services you pick.
A realtime plugin needs none of this and pays nothing for it. `voice.ts` is a module you do not import.
## One utterance per phrase
A Microsoft Teams call delivers audio continuously: silence is still frames, fifty a second. `UtteranceSegmenter` is the gate between that and a service that wants a phrase.
```ts theme={null}
import { UtteranceSegmenter } from "@komaa/standin-sdk";
const segmenter = new UtteranceSegmenter();
async onCallerAudio(pcm: Buffer) {
const utterance = segmenter.feed(pcm);
if (utterance !== undefined) await transcribe(utterance);
}
```
Five bounds, and each exists because of a specific failure.
| Bound | Default | Why |
| ---------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `speechRms` | `0.02` | The loudness at which a frame counts as speech rather than room noise, on the 0.0 to 1.0 scale `pcm16Rms` returns. Nothing else opens the gate, so this is the first knob to reach for on a bad line. Raise it on a noisy room, where air conditioning or a fan holds the gate open and every utterance runs to the ceiling. Lower it for a quiet caller on a poor microphone, whose speech never trips it and who is never heard at all. |
| `silenceMs` | `800` | A pause inside a sentence is not the end of one. Too short and the agent interrupts a thinking caller; too long and it feels slow. |
| `prerollMs` | `240` | The syllable that trips the gate is part of the word. Without it every utterance begins mid-consonant and the transcript loses the first word of most sentences. |
| `minUtteranceMs` | `120` | A cough is not a turn. Sending one costs a request and returns nothing worth answering. Judged on the loud part alone, so the pre-roll and the trailing silence cannot carry a click over the floor. |
| `maxUtteranceMs` | `30000` | A stuck-open microphone or a television in the room never goes quiet, and without a cap one utterance grows for the whole call. |
Every default is an exported constant, so a test can assert against the value rather than retype it:
`DEFAULT_SPEECH_RMS`, `DEFAULT_SILENCE_MS`, `DEFAULT_PREROLL_MS`, `DEFAULT_MIN_UTTERANCE_MS`,
`DEFAULT_MAX_UTTERANCE_MS`.
`flush()` takes what is held mid-utterance, for teardown. `reset()` abandons it, for a barge-in.
`speaking` is true while the caller is mid-utterance right now.
This gate answers "did the caller stop talking?". It does not answer "was that the caller at all?".
On a speakerphone the agent's own voice comes back up the caller leg loudly enough to clear any
`speechRms` you would want to set, and the fix is a time window against the playout clock rather
than a louder gate. See
[Realtime providers](/typescript-sdk/realtime-providers#the-agent-answering-itself), which applies
here too whenever the caller is not on a headset.
## The WAV a speech engine hands back
It is rarely the one you wanted: 32-bit float, 44.1 kHz, stereo, or wrapped in `WAVE_FORMAT_EXTENSIBLE`, which ffmpeg emits even for plain PCM. Every one of those plays as noise on a call unconverted, and the header does not make the failure obvious.
```ts theme={null}
import { decodeWav, encodeWav } from "@komaa/standin-sdk";
const pcm = decodeWav(await tts.speak(text)); // PCM16 mono at the call's rate
```
Chunks are walked rather than assumed at a fixed offset, because a real encoder puts `LIST` and `fact` chunks before the data and a fixed offset reads them as samples. Stereo is averaged rather than halved, so whoever is on the right channel is not lost. Floats outside the range are clamped, because wrapping is the loudest possible click.
`encodeWav` goes the other way, for a transcription service that will not take raw PCM.
## Playing it back
A text-to-speech engine returns a whole utterance at once. A call takes 20 milliseconds every 20 milliseconds. Sending the buffer in one go hands the service seconds of audio it must queue, and that queue is what makes a barge-in arrive too late to matter: the caller interrupts, the model stops, and the bot keeps talking for the length of what was already sent.
```ts theme={null}
import { PacedPlayback } from "@komaa/standin-sdk";
const playback = new PacedPlayback((pcm) => session.sendAudio(pcm));
const result = await playback.say(pcm);
if (result.interrupted) { /* ... */ }
```
The pacing is on an **absolute clock**. Sleeping 20 ms per frame accumulates every scheduling delay, and a minute of speech ends seconds behind where it should be. This one sleeps until the next frame is due, so a late frame is followed by a short sleep rather than a full one.
The second constructor argument is the frame size in milliseconds, `FRAME_MS` by default, and the
byte size follows from it. `playing` is true while a buffer is going out.
`cancel()` stops what is playing, and is safe to call from a receive loop. Turns are serialised, so two cannot interleave into one stream the caller hears as both at once. The result carries `sentMs` against `totalMs`, which is the difference between "I told them" and "I started to".
## One object that runs the whole turn
The three pieces above are the parts. `VoiceLane` is the assembly: one per call, fed the caller's frames, responsible for everything between a frame arriving and an answer being heard.
```ts theme={null}
import { VoiceLane, type CallSession } from "@komaa/standin-sdk";
class MyHandler {
#lane!: VoiceLane;
async onStart(session: CallSession) {
this.#lane = new VoiceLane(session, myStt, myAgent, myTts);
}
async onCallerAudio(pcm: Buffer) {
await this.#lane.feed(pcm);
}
async aclose(reason: string) {
await this.#lane.aclose();
}
}
```
`feed` never blocks and never throws. It runs on the receive path of a live call, so the turn it starts is detached: awaiting a model there stops frames arriving, and a lane that has stopped hearing the caller cannot notice an interruption.
### The three callables
| You supply | It is given | It returns |
| ------------ | ------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `transcribe` | one whole utterance, PCM16, 16 kHz, mono `Buffer` | the words, as `Promise`. Empty or whitespace means nothing was said |
| `answer` | that transcript | the answer as `Promise`, or an `AsyncIterable` of pieces |
| `synthesize` | one piece of answer text | speech as PCM16, 16 kHz, mono: `Promise`, or an `AsyncIterable` of chunks |
Each half of that contract carries its weight.
`transcribe` is handed a phrase, not frames. Working out where the caller stopped is the segmenter's job above, and it is the one part of this a transcription service will not do for you.
Its empty return is a signal rather than a failure. It is how you say "there were no words in that", and the lane acts on it.
`answer` returning a plain string is the simple case. Returning an async iterable is what gets the first sentence out before the last one exists. The lane tells the two apart at runtime, so a callable is free to return either without a flag to configure.
`synthesize` returns PCM at the call's rate, because what comes back goes straight to paced playback. Nothing in the lane decodes a WAV for you: run `decodeWav` over it first if your engine returns one, and `resamplePcm16` if it speaks at 24 kHz.
### The awkward parts
**One turn at a time.** A new utterance retires the turn in flight before it starts its own. An agent asked two questions at once answers neither well, and both answers would be spoken over each other. JavaScript cannot cancel a promise, so the older turn is retired by generation: each turn carries a number, a new utterance bumps it, and the turn checks that number again before it speaks the answer. The retired turn runs to completion, and the answer it comes back with is never spoken. Whatever it had already started playing is dropped as the new turn begins. Nothing cancels your callables, so a request that is already in flight is still paid for, and whatever one of them holds open is released by its own `finally` rather than by the lane. The Python twin cancels the task instead, because asyncio can.
**Barge-in lands when the interruption starts, not when it ends.** The moment the caller's voice opens over the top of an answer, the buffered audio is dropped and StandIn is told to drop what it still holds. Waiting for that utterance to finish would spend the whole of it talking at somebody who has stopped listening, which is the difference between a call that feels alive and one that does not. `bargeIn()` is public, for an interruption the caller's audio does not show:
```ts theme={null}
async onGoodbye(text: string) {
await this.#lane.bargeIn();
await this.#lane.say(text);
}
```
**A silence is not a turn.** An utterance that transcribes to nothing ends there. The agent is never asked and nothing is spoken. The segmenter opens on any loud frame, so a cough, a door or a second of traffic reaches the transcriber, and waking the agent for every one of them is a bill and a caller being answered at random. Note the difference from the case below: an empty transcript is silent, a transcriber that fails says so out loud.
### Nothing throws into the call
A step that fails is logged, and the caller hears a sentence.
| When | What is spoken |
| ------------------ | --------------------------------------------------------------------- |
| `transcribe` fails | `TROUBLE_HEARING`, "Sorry, I did not catch that." |
| `answer` fails | `TROUBLE_ANSWERING`, "Sorry, I am having trouble answering just now." |
| `synthesize` fails | `TROUBLE_SPEAKING`, "Sorry, I am having trouble speaking just now." |
Silence is the one thing a caller cannot interpret. Somebody on a phone call cannot tell a broken transcriber from an agent that is thinking, so they wait, and then they hang up.
`TROUBLE_SPEAKING` goes back through your own `synthesize`, once. The retry is guarded, because an engine that is down cannot say the sentence about being down, and the lane goes quiet rather than looping on it.
All three are exported from `@komaa/standin-sdk`, so a test can assert on the sentence itself.
### A long answer, spoken as it is written
Return an async iterable from `answer` and each piece is synthesized and played as it arrives.
```ts theme={null}
async function* myAgent(heard: string) {
for await (const sentence of llm.stream(heard)) yield sentence;
}
```
The caller hears the beginning of a long answer while the rest is still being written, which is the whole point: a model that takes four seconds to finish a paragraph leaves four seconds of silence otherwise.
Yield whole sentences. Each piece is one `synthesize` call and one spoken buffer, so a token at a time is a synthesis request per word and speech chopped into syllables. Pieces that are blank are skipped. An interruption stops the loop, and what was never reached is never synthesized and never spoken. The generation is re-checked before each piece, so a stream that has been superseded goes quiet at the next piece rather than at the end of the answer. Leaving the loop closes the generator, so a `finally` inside it still runs.
### Saying a line nobody asked for
A greeting, a handover, something that arrived from outside the call.
```ts theme={null}
const turn = await lane.say("Thanks for taking the call.");
```
`say` skips `answer` and goes straight to synthesis and paced playback, so the line can be interrupted like any other answer, and it queues behind whatever is already speaking rather than overlapping it. It returns the `VoiceTurn`, which is how you find out whether it was heard to the end. It does not supersede a turn in flight, so call `bargeIn()` first when the line has to come before what is being said.
### Closing the lane
`aclose()` drops the buffered audio, resets the segmenter, retires the turn in flight and waits for it to settle. Nothing cancels a request that is already in flight, so that wait is as long as your slowest step has left to run. Call it from your handler's `aclose`, once.
After that, `feed` is a no-op rather than an error. Frames keep arriving for a moment after teardown begins, and the end of a call is the worst place to start throwing from the receive path.
### What each turn leaves behind
`VoiceTurn` is the record of one exchange: `heard`, `said`, `interrupted` and `error`. `interrupted` is not a failure. It is the most common way a real conversation goes.
Pass `onTurn` to receive one per caller turn the agent answered, and you have a transcript for free:
```ts theme={null}
import { VoiceLane, type VoiceTurn } from "@komaa/standin-sdk";
const remember = (turn: VoiceTurn) => {
transcript.add(callerName, turn.heard);
transcript.add("Assistant", turn.said, "assistant");
};
const lane = new VoiceLane(session, myStt, myAgent, myTts, { onTurn: remember });
```
It is a plain function, not an async one, and it is called inline at the end of a turn, so keep it short. An exception it throws is swallowed: a plugin's own bookkeeping must not end a call.
A turn the transcriber or the agent failed is spoken but not reported, so this is a record of the conversation rather than a count of what went wrong. An answer that came back empty is not reported either, because nothing was said. Nor is a line from `say`: it hands you its `VoiceTurn` as the return value instead, so a greeting is missing from a transcript built only from `onTurn`.
A synthesis failure is the one that does reach it, with the trouble sentence as `said` and the engine's message in `error`. `error` is `undefined` on a turn that went through cleanly, so check it before filing a turn as something the agent meant to say.
Three more things on the options object and the lane itself. `segmenter` takes an `UtteranceSegmenter` you tuned yourself, with the five bounds above. `speaking` is true while audio is going out, and `busy` covers the whole turn, including the model's own thinking. `turn` is the promise for the turn in flight, or `undefined`, for a test or a teardown that wants to let one finish.
## What this page does not cover
Two more shared lanes sit next to this one and are worth knowing exist before you write them
yourself.
* **The echo guard.** On a speakerphone the agent's own voice comes back in, the model's voice
detection hears it, and the agent answers itself in a loop. `shouldSuppressEcho` is what stops it,
on a playout clock rather than a wall clock. See
[Realtime providers](/typescript-sdk/realtime-providers#the-agent-answering-itself).
* **The meeting gate.** In a group call the prior question is whether to answer at all. `GroupGate`
decides, and is inert on a 1:1 call. See [Group calls](/typescript-sdk/group-calls).
## Next
`resamplePcm16`, `FrameAligner` and `pcm16Rms`, which this lane is built on.
Deciding which finished utterance was even meant for the agent.
The other shape: one provider owning the whole turn.
What to do with the `VoiceTurn` stream `onTurn` hands you.