OpenAI Realtime plugin

@komaa/standin-sdk/openai puts OpenAI Realtime on a real Microsoft Teams call. StandIn answers the call and dials your worker; this plugin answers that dial, opens one session per call, and relays the audio both ways.

Install and run

There is nothing to install beyond the SDK. OpenAI Realtime is reached over an ordinary WebSocket, so this plugin adds no dependency.

npm install @komaa/standin-sdk

export STANDIN_SECRET=...        # your StandIn connection secret
export OPENAI_API_KEY=...
npx standin-openai

Expose port 9442 at the /msteams/calling path and register the public wss:// URL as your StandIn identity’s agent voice URL. The full walkthrough is in the Quickstart, and there is a runnable example at examples/openai-msteams-connector.

The audio format

The Realtime API speaks PCM at 24 kHz and a Microsoft Teams call speaks 16 kHz. This plugin owns that conversion in both directions, using the same resampler the Python SDK uses, driven by the same shared conformance vectors. You never see it. OPENAI_VAD_TYPE decides when the model thinks you have finished: semantic_vad, the default, listens for a finished thought, and server_vad listens for silence. Semantic interrupts less often mid-sentence. Any other value is refused when the configuration is read, naming the variable, rather than quietly falling back.

What the agent can do about the call

Declared automatically when the call starts, so there is nothing to set up on the provider’s side.

Tool	Parameters	What it does
`end_call`	none	Hang up.
`express`	`emotion`	Set the avatar’s expression on the bot’s tile. See The avatar.
`show_image`	`url`, `caption?`, `display?`	Put a picture on the bot’s video tile.
`look`	`source?`, `question?`	Look at the caller’s screen share or camera.
`look_back`	`question?`	Look again at something the caller has already moved past. Recorded calls only.

show_image fetches the URL through the SDK’s guard: public hosts only, no private or link-local addresses, and the address is re-checked at connect time. display is fullscreen or overlay, and this plugin routes the call through CallTools.dispatch, so the model’s choice is honoured. See Vision and the avatar for what each placement is for, and Sending a picture for putting a picture into chat instead. The Realtime model hears but does not see, so look needs a vision model of your choosing. Set STANDIN_VISION_API_URL and STANDIN_VISION_MODEL to any OpenAI-compatible endpoint that accepts images, including one you run yourself. The frame is sent for inference and not stored, and only the description comes back. Without it the agent is told plainly that looking is unavailable, rather than being left with silence. The whole surface is on Vision and the avatar. Pass a describer in the handler options to point this plugin at a different vision model from the one the environment names, which is how one worker answers two identities with two vision budgets. Build it with FrameDescriber.fromEnv() or its constructor, both on Giving a voice model eyes.

Tools of your own

A custom tool runs in your worker, so it can reach whatever your worker can reach:

import { CallServer } from "@komaa/standin-sdk";
import {
  OpenAIHandler,
  openAIConfigFromEnv,
  type CustomTool,
} from "@komaa/standin-sdk/openai";

const tools: CustomTool[] = [
  {
    name: "open_ticket",
    description: "Open a support ticket for the caller.",
    parameters: {
      type: "object",
      properties: { summary: { type: "string" } },
      required: ["summary"],
    },
    // ctx.call is the live call; ctx.recording says whether it is being recorded.
    handler: async (params, ctx) => `opened ticket for ${String(params.summary)}`,
  },
];

const config = openAIConfigFromEnv();
const server = new CallServer({
  handlerFactory: () => new OpenAIHandler({ config, tools }),
});
await server.start();

The description is what the model reads to decide whether to call it, so write it for a model rather than for a developer. Keep the handler fast: the caller is waiting in silence while it runs. A custom tool may not shadow a built-in one. That is refused when the handler is built rather than at the first call, because a shadowed end_call is an agent that has quietly lost the ability to hang up.

A remote MCP server

An mcpTools entry is dialled by OpenAI itself, so nothing runs in your worker at call time:

import { CallServer } from "@komaa/standin-sdk";
import { OpenAIHandler, mcpTool, openAIConfigFromEnv } from "@komaa/standin-sdk/openai";

const config = openAIConfigFromEnv();
const mcpTools = [
  mcpTool({ server_label: "docs", server_url: "https://mcp.example.com" }),
];

const server = new CallServer({
  handlerFactory: () => new OpenAIHandler({ config, mcpTools }),
});
await server.start();

mcpTool requires server_label and server_url and throws naming them if either is missing, so a typo stops the worker at startup rather than producing a session whose tool list is quietly one short. Approval defaults to never, because a live voice call has no approval interface and a pending approval is a caller listening to silence. Pass require_approval in the entry yourself to override it: the defaults are spread first, so anything you set wins.

Inside your own worker

The plugin is a CallHandler like any other:

import { CallServer } from "@komaa/standin-sdk";
import { OpenAIHandler, openAIConfigFromEnv } from "@komaa/standin-sdk/openai";

const config = openAIConfigFromEnv();
const server = new CallServer({
  handlerFactory: () => new OpenAIHandler({ config }),
});
await server.start();

Read the configuration once, then close over it. handlerFactory runs once per call, so new OpenAIHandler() with nothing passed reads the environment again on every call, and a key removed after startup would fail the next caller rather than failing you. serve(), which is what npx standin-openai runs, does exactly this, then waits for SIGINT or SIGTERM before calling server.aclose(). server.start() returns as soon as the listener is bound, so a script that ends there exits before a single call arrives, and aclose() is what drains live calls and releases the port.

Configuration

Variable	Default	What it is
`OPENAI_API_KEY`	(required)	Your API key. Never logged, never sent to StandIn.

Optional settings

Variable	Default	What it is
`OPENAI_REALTIME_MODEL`	`gpt-realtime`	Which Realtime model answers.
`OPENAI_REALTIME_HOST`	`api.openai.com`	Must end `.openai.com` or `.azure.com`. The Azure allowance is what makes an Azure OpenAI deployment work here at all.
`OPENAI_VOICE`	(the model’s own)	Override the voice.
`OPENAI_INSTRUCTIONS`	a short spoken-reply prompt	The agent’s base prompt. Exported as `DEFAULT_INSTRUCTIONS` if you want to extend rather than replace it.
`OPENAI_VAD_TYPE`	`semantic_vad`	`server_vad` or `semantic_vad`. Anything else is refused when the configuration is read.
`OPENAI_TRANSCRIPTION_MODEL`	(unset)	Ask for transcripts alongside the audio.
`OPENAI_LOG_TRANSCRIPTS`	(off)	Exactly `true` to log turns. Gated a second time on the call being recorded.

The configuration is read once when the worker starts, not per call, so a missing key stops the worker at startup rather than surprising the first caller. OPENAI_REALTIME_HOST is checked against those two suffixes, because your API key travels to it: a wrong host would be credential leakage rather than a failed call. The reasoning behind that shape is on Configuration.

Realtime providers

The startup buffer, the echo guard and barge-in this plugin sits on.

Vision and the avatar

What look, show_image and express reach.

Get started

The SDK

Plugins

OpenAI Realtime plugin

Install and run

The audio format

What the agent can do about the call

Tools of your own

A remote MCP server

Inside your own worker

Configuration

Next

Realtime providers

Vision and the avatar

​Install and run

​The audio format

​What the agent can do about the call

​Tools of your own

​A remote MCP server

​Inside your own worker

​Configuration

​Next

Realtime providers

Vision and the avatar

Install and run

The audio format

What the agent can do about the call

Tools of your own

A remote MCP server

Inside your own worker

Configuration

Next