> ## Documentation Index
> Fetch the complete documentation index at: https://docs.komaa.com/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenAI Realtime plugin

> Answer Microsoft Teams calls with an OpenAI Realtime model, using the StandIn TypeScript SDK.

`@komaa/standin-sdk/openai` puts [OpenAI Realtime](https://platform.openai.com/docs/guides/realtime) on a real Microsoft Teams call. StandIn answers the call and dials your worker; this plugin answers that dial, opens one session per call, and relays the audio both ways.

## Install and run

There is nothing to install beyond the SDK. OpenAI Realtime is reached over an ordinary WebSocket, so this plugin adds no dependency.

```bash theme={null}
npm install @komaa/standin-sdk
```

```bash theme={null}
export STANDIN_SECRET=...        # your StandIn connection secret
export OPENAI_API_KEY=...
npx standin-openai
```

Expose port `9442` at the `/msteams/calling` path and register the public `wss://` URL as your StandIn identity's agent voice URL. The full walkthrough is in the [Quickstart](/typescript-sdk/quickstart), and there is a runnable example at [examples/openai-msteams-connector](https://github.com/komaa-com/standin/tree/main/examples/openai-msteams-connector).

## The audio format

The Realtime API speaks PCM at 24 kHz and a Microsoft Teams call speaks 16 kHz. This plugin owns that conversion in both directions, using the same resampler the Python SDK uses, driven by the same shared conformance vectors. You never see it.

`OPENAI_VAD_TYPE` decides when the model thinks you have finished: `semantic_vad`, the default, listens for a finished thought, and `server_vad` listens for silence. Semantic interrupts less often mid-sentence. Any other value is refused when the configuration is read, naming the variable, rather than quietly falling back.

## What the agent can do about the call

Declared automatically when the call starts, so there is nothing to set up on the provider's side.

| Tool         | Parameters                    | What it does                                                                                                        |
| ------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `end_call`   | none                          | Hang up.                                                                                                            |
| `express`    | `emotion`                     | Set the avatar's expression on the bot's tile. See [The avatar](/typescript-sdk/avatar#the-emotion-the-face-wears). |
| `show_image` | `url`, `caption?`, `display?` | Put a picture on the bot's video tile.                                                                              |
| `look`       | `source?`, `question?`        | Look at the caller's screen share or camera.                                                                        |
| `look_back`  | `question?`                   | Look again at something the caller has already moved past. Recorded calls only.                                     |

`show_image` fetches the URL through the SDK's guard: public hosts only, no private or link-local addresses, and the address is re-checked at connect time. `display` is `fullscreen` or `overlay`, and this plugin routes the call through `CallTools.dispatch`, so the model's choice is honoured. See [Vision and the avatar](/typescript-sdk/vision) for what each placement is for, and [Sending a picture](/typescript-sdk/sending-pictures) for putting a picture into chat instead.

The Realtime model hears but does not see, so `look` needs a vision model of your choosing. Set `STANDIN_VISION_API_URL` and `STANDIN_VISION_MODEL` to any OpenAI-compatible endpoint that accepts images, including one you run yourself. The frame is sent for inference and **not stored**, and only the description comes back. Without it the agent is told plainly that looking is unavailable, rather than being left with silence. The whole surface is on [Vision and the avatar](/typescript-sdk/vision).

Pass a `describer` in the handler options to point this plugin at a different vision model from the one the environment names, which is how one worker answers two identities with two vision budgets. Build it with `FrameDescriber.fromEnv()` or its constructor, both on [Giving a voice model eyes](/typescript-sdk/vision#giving-a-voice-model-eyes).

## Tools of your own

A custom tool runs in **your** worker, so it can reach whatever your worker can reach:

```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
import {
  OpenAIHandler,
  openAIConfigFromEnv,
  type CustomTool,
} from "@komaa/standin-sdk/openai";

const tools: CustomTool[] = [
  {
    name: "open_ticket",
    description: "Open a support ticket for the caller.",
    parameters: {
      type: "object",
      properties: { summary: { type: "string" } },
      required: ["summary"],
    },
    // ctx.call is the live call; ctx.recording says whether it is being recorded.
    handler: async (params, ctx) => `opened ticket for ${String(params.summary)}`,
  },
];

const config = openAIConfigFromEnv();
const server = new CallServer({
  handlerFactory: () => new OpenAIHandler({ config, tools }),
});
await server.start();
```

The `description` is what the model reads to decide whether to call it, so write it for a model rather than for a developer. Keep the handler fast: the caller is waiting in silence while it runs.

A custom tool may not shadow a built-in one. That is refused when the handler is built rather than at the first call, because a shadowed `end_call` is an agent that has quietly lost the ability to hang up.

### A remote MCP server

An `mcpTools` entry is dialled by OpenAI itself, so nothing runs in your worker at call time:

```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
import { OpenAIHandler, mcpTool, openAIConfigFromEnv } from "@komaa/standin-sdk/openai";

const config = openAIConfigFromEnv();
const mcpTools = [
  mcpTool({ server_label: "docs", server_url: "https://mcp.example.com" }),
];

const server = new CallServer({
  handlerFactory: () => new OpenAIHandler({ config, mcpTools }),
});
await server.start();
```

`mcpTool` requires `server_label` and `server_url` and throws naming them if either is missing, so a typo stops the worker at startup rather than producing a session whose tool list is quietly one short.

Approval defaults to never, because a live voice call has no approval interface and a pending approval is a caller listening to silence. Pass `require_approval` in the entry yourself to override it: the defaults are spread first, so anything you set wins.

## Inside your own worker

The plugin is a `CallHandler` like any other:

```ts theme={null}
import { CallServer } from "@komaa/standin-sdk";
import { OpenAIHandler, openAIConfigFromEnv } from "@komaa/standin-sdk/openai";

const config = openAIConfigFromEnv();
const server = new CallServer({
  handlerFactory: () => new OpenAIHandler({ config }),
});
await server.start();
```

Read the configuration **once**, then close over it. `handlerFactory` runs once per call, so
`new OpenAIHandler()` with nothing passed reads the environment again on every call, and a key
removed after startup would fail the next caller rather than failing you. `serve()`, which is what
`npx standin-openai` runs, does exactly this, then waits for `SIGINT` or `SIGTERM` before calling
`server.aclose()`. `server.start()` returns as soon as the listener is bound, so a script that
ends there exits before a single call arrives, and `aclose()` is what drains live calls and
releases the port.

## Configuration

| Variable         | Default      | What it is                                         |
| ---------------- | ------------ | -------------------------------------------------- |
| `OPENAI_API_KEY` | *(required)* | Your API key. Never logged, never sent to StandIn. |

<Accordion title="Optional settings">
  | Variable                     | Default                     | What it is                                                                                                             |
  | ---------------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
  | `OPENAI_REALTIME_MODEL`      | `gpt-realtime`              | Which Realtime model answers.                                                                                          |
  | `OPENAI_REALTIME_HOST`       | `api.openai.com`            | Must end `.openai.com` or `.azure.com`. The Azure allowance is what makes an Azure OpenAI deployment work here at all. |
  | `OPENAI_VOICE`               | *(the model's own)*         | Override the voice.                                                                                                    |
  | `OPENAI_INSTRUCTIONS`        | a short spoken-reply prompt | The agent's base prompt. Exported as `DEFAULT_INSTRUCTIONS` if you want to extend rather than replace it.              |
  | `OPENAI_VAD_TYPE`            | `semantic_vad`              | `server_vad` or `semantic_vad`. Anything else is refused when the configuration is read.                               |
  | `OPENAI_TRANSCRIPTION_MODEL` | *(unset)*                   | Ask for transcripts alongside the audio.                                                                               |
  | `OPENAI_LOG_TRANSCRIPTS`     | *(off)*                     | Exactly `true` to log turns. Gated a second time on the call being recorded.                                           |
</Accordion>

<Note>
  The configuration is read once when the worker starts, not per call, so a missing key stops the worker at startup rather than surprising the first caller. `OPENAI_REALTIME_HOST` is checked against those two suffixes, because your API key travels to it: a wrong host would be credential leakage rather than a failed call. The reasoning behind that shape is on [Configuration](/typescript-sdk/configuration).
</Note>

## Next

<CardGroup cols={2}>
  <Card title="Realtime providers" icon="bolt" href="/typescript-sdk/realtime-providers">
    The startup buffer, the echo guard and barge-in this plugin sits on.
  </Card>

  <Card title="Vision and the avatar" icon="eye" href="/typescript-sdk/vision">
    What `look`, `show_image` and `express` reach.
  </Card>
</CardGroup>