Attachments & Images

The AI chat does not embed file bytes inside the chat request body. Files are uploaded to the attachments API first, then the chat dispatcher receives a list of attachmentIds, hydrates each one into provider-native content, and appends it to the last user message before calling the Vercel AI SDK.

This page documents the contract end-to-end so that custom integrations — your own React composer, a CLI client, a curl test — produce something the model actually sees.

End-to-end flow

┌──────────────┐   1. POST /api/attachments (multipart)   ┌──────────────────┐
│ Browser /    │ ───────────────────────────────────────▶ │ /api/attachments │
│ API client   │ ◀─── { item: { id, fileName, ... } } ─── │ (core module)    │
└──────────────┘                                          └──────────────────┘
       │
       │ 2. POST /api/ai_assistant/ai/chat?agent=<id>
       │    { messages, attachmentIds: [id1, id2], ... }
       ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ Chat dispatcher                                                          │
│   ↳ resolveAttachmentPartsForAgent(attachmentIds, authContext)           │
│       — loads each attachment row, scopes to caller tenant/org           │
│       — classifies image / pdf / file                                    │
│       — reads bytes (≤ 4MB inline) or extracted text                     │
│   ↳ attachAttachmentsToMessages(messages, parts)                         │
│       — appends FileUIPart entries to the last user message              │
│   ↳ summarizeAttachmentPartsForPrompt(parts)                             │
│       — appends `[ATTACHMENTS]` block to the system prompt               │
│   ↳ convertToModelMessages(...) → streamText({ messages, ... })          │
└──────────────────────────────────────────────────────────────────────────┘

Client-side: `<AiChat>` composer

The shipped <AiChat> composer at packages/ui/src/ai/AiChat.tsx already wires the full path:

The user picks a file via the paperclip icon. Selected files land in local state.
useAiChatUpload POSTs each file to /api/attachments as multipart/form-data with the required entityId ('ai-chat-draft') and recordId (a per-batch UUID).
The returned attachmentId is stored on the pending entry.
When the user sends the message, the chat hook puts the attachmentId list under attachmentIds in the JSON body.
For images, an in-chat preview is generated by FileReader.readAsDataURL (capped at 2 MB for the browser fallback cache). The data URL is attached to the rendered local message only; server-side conversation storage persists attachment ids and safe file metadata, not preview data URLs. Durable backend preview URLs are intentionally avoided because the LLM provider can never reach a localhost origin.

The pieces you need if you write your own composer:

import { uploadAttachmentsForChat } from '@saasframe/ui/ai/upload-adapter'

const { items, failed } = await uploadAttachmentsForChat([file], {
  entityType: 'ai-chat-draft',     // optional — this is the default
  // recordId: '<uuid>',           // optional — minted per batch otherwise
  // perFileTimeoutMs: 60_000,     // optional — per-file abort. Pass 0 to disable
})
const attachmentIds = items.map(({ attachmentId }) => attachmentId)

uploadAttachmentsForChat is framework-agnostic (no React, no Next.js); the React hook useAiChatUpload is a thin wrapper that adds per-file progress/error state.

Pairing results back to the composer's chips

Every UploadedAttachment and UploadFailure carries two pairing fields the chat composer relies on:

Field	Why it matters
`originalFileName`	Exactly the `File.name` you passed in — survives server-side filename sanitization (whitespace, unicode, dangerous chars)
`inputIndex`	Position in the input array. Lets you pair results back to chips even when two files in the same batch share a name

The shipped <AiChat> keys upload outcomes by inputIndex rather than by filename. If you build your own composer, do the same — the legacy "match by fileName" pattern silently strands chips on the spinner when the server sanitizes a name.

Per-file timeout

The adapter aborts each upload after 60 seconds by default (configurable via perFileTimeoutMs). The aborted upload lands in failed with reason: 'aborted' and a clear message, instead of leaving the chip spinning forever. The timer is wired through a child AbortController so the parent batch's abort still works.

Send button gating

<AiChat> disables the Send button while any attachment is mid-upload. Without that gate, hitting Enter before the upload completes would ship attachmentIds: [] to the dispatcher (the chip is visible but the server hasn't returned an id yet) — the model never sees the file. The composer hint flips to "Uploading attachments… Send is disabled until they finish." while the upload is in flight.

warning

Calling POST /api/attachments without entityId and recordId returns 400 entityId, recordId and file are required. Always go through the adapter or include both fields explicitly — otherwise the upload silently fails and the chat sends an empty attachmentIds, which is exactly the "the AI doesn't see the image" symptom.

Server-side: chat dispatcher request shape

POST /api/ai_assistant/ai/chat?agent=<module>.<agent>

{
  "messages": [
    { "role": "user", "content": "Tell me what's in this photo." }
  ],
  "attachmentIds": ["att_abc123", "att_def456"],
  "pageContext": { "entityType": "catalog.product", "recordId": "<uuid>" },
  "conversationId": "<stable-uuid>",
  "debug": false
}

Field	Required	Notes
`messages`	yes	Plain `{ role, content }` items. Do NOT inline image bytes here — the dispatcher does that for you on the server.
`attachmentIds`	no	Array of attachment row IDs the caller previously uploaded. The dispatcher resolves them inside the caller's tenant/org scope.
`pageContext`	no	Forwarded to `agent.resolvePageContext` for system-prompt hydration.
`conversationId`	no	Stable per-chat id used by the mutation-approval idempotency hash.
`debug`	no	When `true`, the server emits extra debug parts in the SSE stream.

The agent's acceptedMediaTypes (image / pdf / file) gates which classes flow through. If the agent declares acceptedMediaTypes: ['image'] and the caller uploads a PDF, the PDF is dropped with a console.warn.

How each media type is encoded for the model

The hydration step turns each attachment into a model-ready part. The mapping is:

Class	Source	Resulting part	What the LLM receives
`image/*` (≤ 4 MB)	bytes from disk	`FileUIPart` with `url: 'data:<mediaType>;base64,<bytes>'`	Inline base64 image content block (Anthropic / OpenAI / Google all accept this).
`image/*` (> 4 MB)	falls back to metadata-only — no inline bytes	system-prompt `[ATTACHMENTS]` line only	The model only learns the file exists.
`application/pdf` (≤ 4 MB)	bytes from disk	`FileUIPart` with `url: 'data:application/pdf;base64,...'`	Anthropic native PDF document block. The provider does the OCR.
Text-like (`text/*`, JSON, YAML, CSV, XML)	the attachment row's pre-extracted `content` column	injected into system prompt as a `[ATTACHMENTS]` block	Plain text up to 64 KB (truncated with a marker).
Anything else	none	system-prompt summary line only	Same as above — model is aware the file exists.

The 4 MB inline ceiling

Most provider APIs (Anthropic, OpenAI, Google) accept inline base64 image / PDF payloads under ~5 MB. The runtime caps at 4 MB to stay well under that ceiling. Larger files are not silently truncated — they degrade to metadata-only. If your tenant needs to ship larger files inline, register an attachmentSigner in the DI container that returns a short-lived signed URL; the runtime then emits a FileUIPart with the URL instead of the data URL.

// packages/<your-module>/src/modules/<module>/di.ts
container.register({
  attachmentSigner: asValue({
    async sign({ attachmentId, mediaType }) {
      return `https://cdn.example.com/attachments/${attachmentId}?token=...`
    },
  }),
})

The signed URL must be reachable by the LLM provider's outbound HTTP — local-only origins (e.g. http://localhost:3000/...) won't work because Anthropic / OpenAI fetch from their own infrastructure.

What the LLM actually sees

After the dispatcher runs convertToModelMessages, the conversation handed to streamText looks like:

[
  { role: 'system', content: '...agent prompt...\n\n[ATTACHMENTS]\n- invoice.pdf (application/pdf, source=bytes)' },
  {
    role: 'user',
    content: [
      { type: 'text', text: 'Tell me what is in this photo.' },
      { type: 'file', mediaType: 'image/png', filename: 'photo.png', data: 'data:image/png;base64,iVBORw0...' },
    ],
  },
]

The Vercel AI SDK then normalizes the data URL: it parses the data: scheme, extracts the raw base64 portion, and hands { type: 'file', mediaType, data: '<base64-only>' } to the provider. Anthropic translates that into:

{
  "type": "image",
  "source": { "type": "base64", "media_type": "image/png", "data": "iVBORw0..." }
}

That's the shape Claude expects. Never construct your own data: 'data:image/png;base64,...' in user code and pass it directly to streamText — the SDK does the parsing for you, but only via the FileUIPart → convertToModelMessages path. Hand-rolled language-model file parts skip that step.

Direct API example (curl)

# 1. Upload the image. Save the `id` from the response.
curl -X POST https://your-host/api/attachments \
  -H "Cookie: <your session cookie>" \
  -F 'entityId=ai-chat-draft' \
  -F 'recordId=00000000-0000-0000-0000-000000000001' \
  -F 'file=@./photo.png'
# {"item":{"id":"att_abc123","fileName":"photo.png", ... }}

# 2. Send a chat turn referencing the attachment id.
curl -X POST 'https://your-host/api/ai_assistant/ai/chat?agent=catalog.merchandising_assistant' \
  -H "Cookie: <your session cookie>" \
  -H "Content-Type: application/json" \
  --data '{
    "messages": [{ "role": "user", "content": "Describe this product photo." }],
    "attachmentIds": ["att_abc123"]
  }'

The response is a Vercel AI SDK UI message stream (text/event-stream).

Common mistakes

Forgetting entityId / recordId when uploading — you get 400 and the chat sends an empty attachmentIds. Use useAiChatUpload or uploadAttachmentsForChat to avoid this.
Inlining base64 inside messages[].content — the dispatcher accepts only plain text in chat messages. Image bytes must travel via attachmentIds. Inlined data URLs in content will be sent to the model as text and ignored.
Uploading > 4 MB images without registering an attachmentSigner — the bytes never reach the model. Either compress before upload, raise the ceiling deliberately, or ship a signer.
Cross-tenant leakage — attachmentIds are scoped to the caller's tenant/org. Out-of-scope IDs are silently dropped with a console.warn. There is no leak, but the caller may wonder why the model doesn't acknowledge the file.
Public URLs to a localhost origin in a custom signer — LLM providers fetch URL-based content from their own infrastructure. http://localhost:* URLs always fail. Use base64 inline for dev, signed CDN URLs for prod.

packages/ai-assistant/src/modules/ai_assistant/lib/attachment-parts.ts — server-side hydration & encoding.
packages/ai-assistant/src/modules/ai_assistant/lib/agent-runtime.ts — attachAttachmentsToMessages, the function that appends FileUIPart entries to the last user message.
packages/ui/src/ai/upload-adapter.ts — the framework-agnostic upload helper.
packages/ui/src/ai/useAiChatUpload.ts — the React hook used by <AiChat>.
Agents — agent-level acceptedMediaTypes field.

End-to-end flow​

Client-side: <AiChat> composer​

Pairing results back to the composer's chips​

Per-file timeout​

Send button gating​

Server-side: chat dispatcher request shape​

How each media type is encoded for the model​

The 4 MB inline ceiling​

What the LLM actually sees​

Direct API example (curl)​

Common mistakes​

Related​