Attachments in chat

A Microsoft Teams message can carry a pasted screenshot, a file dragged in from disk, or a voice note. Without any of this, your handler gets InboundMessage.attachments as raw objects, so the best it can do is read a JSON blob to a model and the worst is answer a message about a picture as though nothing had been sent.

One call does the whole message

import { buildChatTurn, chatImageDataUrl } from "@komaa/standin-sdk";

const turn = await buildChatTurn(message, { transcribe: myStt });
const answer = await agent.ask(turn.query, turn.images.map(chatImageDataUrl));

buildChatTurn takes the InboundMessage your chat handler was given and returns a ChatTurn: the text to ask, the pictures to hand over, and a plain sentence naming anything that came with the message.

Field	Type	What it is
`query`	`string`	The text to put in front of the model, including every note below.
`images`	`ChatImage[]`	Pictures that were fetched and are ready for a vision model.
`voiceNote`	`string`	What the voice notes said, or empty.
`attachmentNote`	`string`	The “what was attached” sentence, on its own.

Everything but the message arrives in one ChatTurnOptions object, and every field of it is optional:

const turn = await buildChatTurn(message, {
  origin: undefined, // leave it off to derive it; see the origin pin below
  images: true, // false to skip image fetching entirely
  transcribe: undefined, // your speech-to-text function, or no transcription
});

images: false and no transcribe make it a pure text assembler that opens no sockets at all. A handler that fetches nothing pays nothing. There is a fourth option, fetchFn, which substitutes the fetch every request goes through: it exists so a test can reach every edge of this page without a socket.

This lives in attachments.ts, deliberately not in chat.ts. That module owns the socket, the duplicate check and the per-conversation ordering, and none of that changes here. Fetching is optional work that must never be able to wedge the transport, so it sits beside it rather than inside it. Everything on this page is re-exported at the package root, with exactly one exception, and that exception is under Images below.

The order is the order a person would say it in

query is assembled in a fixed order, and the order is not arbitrary: it is the order somebody would have said the same thing out loud.

What they typed. message.text, trimmed.
What they pressed. The submit payload of an Action.Submit on a card this agent sent, as cardActionNote. A card message arrives with empty text, so without this the agent is asked nothing at all and answers as though the person said nothing. The payload is serialized and truncated at CARD_PAYLOAD_MAX_CHARS, which is 4096 characters, and that bounds this agent’s own card template rather than a stranger’s message.
What they said out loud. The transcribed voice notes, under [They sent a voice message].
What they attached. The note from attachmentsNote, under [Attached to this message].

Each part is omitted when it is empty, and the parts are joined by blank lines. A message with nothing attached produces a query that is exactly the text the person typed.

It never throws

buildChatTurn does not throw. Anything that will not load is named in the note instead of failing the turn, because a failed picture is not a reason to leave a question unanswered. The note distinguishes the two outcomes per attachment:

[Attached to this message]
- plan.png [image] (attached)
- q3.xlsx [file] (unreadable)

That distinction is the point of the note. A model told a picture was attached and could not be opened says something useful. A model told nothing answers as if the message were empty. And a model told a picture was attached, when nothing was actually fetched, answers about a picture it has never seen, which is the worst of the three. The note is bounded at ATTACHMENT_NOTE_MAX_LINES entries, which is ten, followed by a count of the rest, so a message with forty attachments does not become the whole prompt. attachmentsNote(attachments, status) is callable on its own, where status is a Map<number, string> keyed by the attachment’s index in the message. Both that bound and CARD_PAYLOAD_MAX_CHARS above are exported from the package root here. The Python twin keeps its two out of its public surface entirely, so that page spells the numbers rather than naming a constant to import.

Images

import type { ChatImage } from "@komaa/standin-sdk";

Field	Type	What it is
`dataBase64`	`string`	The image as it arrived, base64.
`mime`	`string`	The real media type. The response’s own header wins; see below.
`name`	`string \| undefined`	The filename, when there was one.
`sizeBytes`	`number`	How big it was.

ChatImage is an interface, and an interface carries no accessors, so the two conveniences the Python twin exposes as properties are plain functions here:

import { chatImageData, chatImageDataUrl } from "@komaa/standin-sdk";

chatImageData(image); // Buffer, for an API that uploads a file
chatImageDataUrl(image); // "data:image/png;base64,..." for most vision APIs

fetchChatImages(attachments, { origin }) is the piece underneath, for a handler that wants the pictures without the rest of the turn. It is best-effort per attachment: one that will not load costs that attachment, never the answer. The caps are at the package root, with one missing on purpose:

import {
  IMAGE_FETCH_ATTEMPTS, // 8
  IMAGE_FETCH_TIMEOUT_MS, // 10_000
  MAX_IMAGES, // 4
} from "@komaa/standin-sdk";

MAX_IMAGES is the accept cap: four pictures are kept from one message, because each becomes a base64 blob in front of a model. IMAGE_FETCH_TIMEOUT_MS is how long one image has to arrive, in milliseconds. The Python twin spells the same two budgets in seconds, as IMAGE_FETCH_TIMEOUT_S and CLIP_FETCH_TIMEOUT_S, so a number carried across unchanged is out by a factor of a thousand. Override either cap on a fetchChatImages call with its maxImages and timeoutMs options.

IMAGE_FETCH_ATTEMPTS is a separate cap, and it is the one that keeps the worst case arithmetic. The accept cap counts only successes, so without it a message naming fifty attachments that all time out still costs fifty timeouts and blows the whole turn budget while nothing is ever accepted. The attempt cap counts requests whatever the outcome: eight tries at ten seconds, and that is the ceiling.

The one number you cannot import is how much one image may weigh. It is 4 MiB, it is the default for the maxBytes option, and it matches the per-attachment ceiling StandIn applies anyway, so a larger local number could never be reached.

MAX_IMAGE_BYTES is exported from the package root, and it is not this cap. The root one comes from vision.ts and is 1_400_000: the outbound limit on an image you draw on the bot’s video tile. The inbound attachment cap is a different number in a different direction of travel, and it keeps its own MAX_IMAGE_BYTES inside attachments.ts rather than colliding with the other at the root.Two consequences. Passing the root MAX_IMAGE_BYTES as fetchChatImages’s maxBytes compiles cleanly, is a perfectly real number, and quietly caps inbound attachments at 1.4 MB instead of 4 MiB. And there is no deep import that reaches the other one: the package’s exports map lists the root and the plugin subpaths only, so @komaa/standin-sdk/attachments does not resolve. Leave maxBytes unset to take the 4 MiB default, or pass a number of your own. buildChatTurn never takes one at all, so a turn assembled through it always uses the default. See Vision for the outbound one.

Two more things the fetch does, both about what arrives rather than what was claimed. The media type is judged before a byte is read, and the response’s own header wins, because an error page would otherwise be base64’d in front of a model as though it were a picture. What the message declared is the fallback, and only when it is itself a media type: a dragged-in file declares a bare extension, and taking that literally would fail every image/ and audio/ check. And the size cap holds while the body is read rather than after it, because a content-length that lies, or is simply absent, otherwise gets to allocate whatever it likes before a later check objects.

A dragged-in picture is a file, not an image

A pasted screenshot arrives with kind of image. The same file dragged in from disk arrives with kind of file and a contentType that is a bare extension like png rather than a media type. So gating on the declared kind alone is the trap here, and it is a quiet one: the picture is never fetched, but the note still says one was attached, and the model answers about a picture nobody gave it. An attachment is therefore worth a request when its kind matches or when its kind is exactly file and its filename’s extension, or a declared type that is itself a bare extension, is one of png, jpg, jpeg, gif, webp, bmp, heic, heif. The widening is file only: it exists to rescue the dragged-in case, not to override a kind the message stated outright. What the file then turns out to be is decided by the response, not by the extension that got it tried. An attachment StandIn marked as not relayable is skipped without a request.

Voice notes

import type { ChatAudio, Transcriber } from "@komaa/standin-sdk";

ChatAudio is one clip before anything has transcribed it: data as a Buffer, mime, and name. fetchChatAudio returns them. video/ types are accepted as well as audio/, because some clients label a voice note with a container type that speech-to-text reads perfectly well, and the same extension widening applies: a voice note relays as a file in practice, so a plugin gating on kind === "audio" transcribes nothing, ever. Transcriber is the function you supply:

type Transcriber = (data: Buffer, mime: string) => Promise<string>;

The core ships none and reads no provider key. Deciding which speech vendor somebody’s voice is sent to is not a decision an SDK gets to make on their behalf, and a default would make it silently.

import { transcribeVoiceMessages } from "@komaa/standin-sdk";

const said = await transcribeVoiceMessages(attachments, {
  origin,
  transcribe: myStt,
});

It returns the clips as one block of text, and empty when there are none, when no transcriber was supplied, or when every one failed. With no transcriber it fetches nothing at all rather than downloading audio that has nowhere to go. A transcriber that throws costs that clip and nothing else. The clip budgets are deliberately larger than the image ones, and all four are at the package root:

import {
  CLIP_FETCH_ATTEMPTS, // 4
  CLIP_FETCH_TIMEOUT_MS, // 20_000
  MAX_CLIPS, // 2
  MAX_CLIP_BYTES, // 16 MiB
} from "@komaa/standin-sdk";

A voice note is minutes of audio where a picture is one screen, so it needs four times the bytes and twice the wall clock to arrive. The attempt cap is halved to pay for that: four tries at twenty seconds is the same eighty second worst case as eight tries at ten. And two clips rather than four, because each accepted clip is a transcription call on top of the download.

An engine that only takes a path

Plenty of transcription engines will not take bytes. spoolClip writes the clip to a temporary file, named with the extension its media type implies, and hands your callback the path:

import { spoolClip, type Transcriber } from "@komaa/standin-sdk";

const myStt: Transcriber = async (data, mime) =>
  spoolClip({ data, mime, name: "" }, (path) => engine.transcribeFile(path));

It resolves to whatever your callback resolved to, and it rethrows whatever your callback threw. The file is removed on the way out, on every path, including the one where the engine threw. That is not tidiness. A transcription engine is being handed somebody’s voice, and a clip left behind in a temporary directory is a copy nobody decided to keep, sitting on disk long after the conversation it came from ended.

This is a callback rather than a context manager, which is the shape TypeScript has for “hold this open for exactly this long”. The other difference from the Python twin is that this one always spools into the system temporary directory: there is no directory argument here.

The origin pin

Every fetch on this page is pinned to the one origin the messages themselves arrived from:

import { chatAttachmentOrigin } from "@komaa/standin-sdk";

chatAttachmentOrigin(); // from STANDIN_CHAT_URL, else the default
chatAttachmentOrigin("wss://example.com/api/chat/channel"); // "https://example.com"
chatAttachmentOrigin("ws://127.0.0.1:9444/x"); // "http://127.0.0.1:9444"

It is derived from the chat channel’s own URL rather than configured separately, so it is right by construction for a self-hosted or local setup and there is no second setting to get wrong. ws maps to http and wss to https, and a default port is not spelled out, so wss://host:443 and wss://host compare equal.

The name is chatAttachmentOrigin here and attachment_origin in Python. That is the one name on this page that is not a straight casing translation of its twin, so a search for attachmentOrigin finds nothing.

It fails closed. A URL it cannot resolve, and anything that is not http, https, ws or wss, returns undefined, and undefined means fetch nothing at all.That direction is the whole design. An attachment URL is signed, but not by you and not for you: it arrives inside a message somebody else wrote, and the pin is the only thing bounding where this worker can be told to go. An unset origin read as “anywhere” would turn a configuration typo into a fetcher that a stranger’s message can point at any address it likes, an internal service or a cloud metadata endpoint included.

Two consequences worth knowing. An attachment whose URL is not on that origin is refused without a request, so the log line is the only cost. And no redirect is followed: the URL is same-origin and signed, so a redirect off it is already anomalous, and following one would reopen the door the pin just closed, since the pin is checked on the URL in the message and not on wherever a 302 points. Pass origin explicitly when your handler already knows it. Otherwise buildChatTurn derives it for you.

Chat

The channel these messages arrive on, and what goes back out.

Security

The signing lanes, and the rest of the fetch posture.

Get started

The SDK

Plugins

Attachments in chat

One call does the whole message

The order is the order a person would say it in

It never throws

Images

A dragged-in picture is a file, not an image

Voice notes

An engine that only takes a path

The origin pin

Next

Chat

Security

​One call does the whole message

​The order is the order a person would say it in

​It never throws

​Images

​A dragged-in picture is a file, not an image

​Voice notes

​An engine that only takes a path

​The origin pin

​Next

Chat

Security

One call does the whole message

The order is the order a person would say it in

It never throws

Images

A dragged-in picture is a file, not an image

Voice notes

An engine that only takes a path

The origin pin

Next