# stdapi.ai
> Self-hosted OpenAI, Anthropic and Cohere compatible API gateway for Amazon Bedrock. Point your applications at it and reach 100+ models across 20+ providers including Claude, OpenAI GPT, xAI Grok and Amazon Nova, for chat, images, video, speech, transcription, embeddings and moderation — in your own AWS account, at AWS rates with no markup.
---
## Home
Source: https://stdapi.ai/md/index.md
# Your OpenAI & Anthropic apps on AWS. Not just chat.
An AI gateway you run in your own AWS account. Point Claude Code, Open WebUI, n8n, OpenClaw — or your own code — at it, and they reach [100+ models](models.md) including Claude, OpenAI GPT, DeepSeek and Nova, at AWS Bedrock rates with zero markup. Adoption is quick: they point at your gateway instead of the vendor's, and the model they name is now drawn from all of them, not one vendor's list.
AWS Qualified Software$0.10/container-hour0% markup on model usage<1 ms gateway overheadOpen-source Community Edition
app.py — pointing the SDK at your gateway
- client = OpenAI()+ client = OpenAI(base_url="https://ai.yourco.com/v1")
response = client.chat.completions.create(
model="gpt-5.6-sol",
messages=messages
)
# 100+ more models, one catalogue# same for the Anthropic SDK — point it at /anthropic
FOR PRODUCTION WORKLOADS
## Bedrock throttling you? Add a region, add its quota.
Every AWS region has its own independent Bedrock quota. stdapi.ai routes requests across the regions you enable and retries eligible failures elsewhere — on throttling, a temporary regional outage, or a retired model. Routing happens in the gateway, so you never touch application code.
stdapi.ai routing + failover
us-east-1 · enabledown quota
us-west-2 · enabledown quota
eu-west-1 · enabledown quota
throttle on one region → eligible requests retry on the next
- +1 quota per region — every region you enable brings its own
- auto retry in another enabled region on eligible throttling or outage
- 0 code changes — routing happens in the gateway, not your app
- 24/7 multi-AZ ECS Fargate deployment via the validated Terraform module
Streaming responses can only retry before the stream opens, and asynchronous jobs stay in the region that accepted them.
{ .panel-stats__note }
[:octicons-arrow-right-24: Resilience & failover documentation](operations_resilience.md)
FOR EVERY MODALITY
## One gateway, every modality you're already calling.
Most gateways stop at chat completions. stdapi.ai covers text, retrieval, embeddings, images, video, speech, live voice, batch inference, moderation, reranking, and file storage across the OpenAI, Anthropic, and Cohere protocols — with conversations kept server-side and continued by id instead of resent.
the parameters your SDK sends are honoured wherever AWS supports them — not just the common subset
- 3 API protocols — OpenAI, Anthropic, and Cohere — from one deployment
- 80+ endpoints — text, retrieval, images, video, audio, batch, moderation, files
- WS live speech-to-speech on the OpenAI Realtime API — speech in, speech and a transcript out
[:octicons-arrow-right-24: API overview](api_overview.md)
FOR RAG & KNOWLEDGE
## Your documents answer, and the model cites them.
Attach a file and it is chunked, embedded and indexed in a vector bucket in your own account, then searched by meaning. Point the same endpoints at an Amazon Bedrock knowledge base you already run and it answers as a vector store too. Then hand the stores to any chat model on the Responses API: it runs the searches the turn needs and cites the files it drew on.
POST /v1/responses200 · file_search
modelany chat model on /v1/responses
searches run by the model"vacation days" · "PTO accrual"
passages kept4 · scored, attribute-filtered
grounded answer2 file citations
managed store · Amazon S3 Vectorsyour Amazon Bedrock knowledge base
- 0 extra infrastructure — no chunker, no embedding pipeline and no vector database to run beside the gateway
- 2 kinds of store behind one API — files you attach here, or a knowledge base you already run
- 1 citation per file the answer drew on, with the passages returned on request
A knowledge base is addressed under an allowlist and is never created or deleted through this API.
{ .panel-stats__note }
[:octicons-arrow-right-24: Vector stores & file search](features.md#retrieval-vector-stores)
FOR AI AGENTS
## Every endpoint is an agent tool.
Agents need no HTTP glue code. stdapi.ai publishes its whole API surface over the Model Context Protocol — chat, images, audio, files, model search — so Claude Code, OpenCode, OpenClaw, LangGraph or any MCP client calls it directly. Underneath, tool calling supports the full OpenAI and Anthropic schemas and tool-choice modes.
- 80+ endpoints exposed as named MCP tools, each with generated documentation
- 2 transports — Streamable HTTP at /mcp, SSE for older clients
- 0 HTTP client code — agents call every endpoint directly
- auto discovery — agents find every tool through the server card and API catalog
This exposes the gateway's own AI and media APIs over MCP; it is not an aggregator for third-party MCP servers.
{ .panel-stats__note }
[:octicons-arrow-right-24: MCP & agent capabilities](features.md#mcp-model-context-protocol)
FOR MODEL CHOICE
## 100+ models — including OpenAI GPT and Anthropic Claude.
Bedrock, Bedrock Mantle, Polly, Transcribe and Comprehend all surface as models in one catalog, detected automatically at startup. On a shared endpoint they interchange by name — swapping a Polly voice for a Bedrock speech model is a one-word change, and nothing needs writing or maintaining as AWS adds and retires models. All four text APIs work with every discovered model, and retired model IDs can redirect to their supported successor instead of failing.
+ many more
standard model names resolve automatically — no ARNs, no ID mapping
- 0 configuration — one catalog spanning Bedrock, Mantle, Polly, Transcribe and Comprehend
- 100+ models across 20+ providers in a typical multi-region catalog
- 4 text APIs on every model — passthrough or converted automatically
[:octicons-arrow-right-24: Browse every model, price and score](models.md)
FOR AWS-NATIVE TEAMS
## Deep AWS features, zero custom code.
Built for AWS, not around it — Bedrock-native capabilities are exposed through standard OpenAI and Anthropic parameters, with AWS AI services and S3 woven into the same API.
all via standard OpenAI & Anthropic parameters — no AWS SDK in your app
- 5 AWS AI services unified — Bedrock, Polly, Transcribe, Translate, Comprehend
- s3:// direct S3 inputs in chat, images, and embeddings — generated media lands back in your bucket
- IAM least-privilege reference policies documented per feature
[:octicons-arrow-right-24: All features](features.md)
FOR BUDGET OWNERS
## Pay AWS rates. See which user spent them.
No subscriptions, no minimums, no markup on model usage. Optional cost tracking prices each call from AWS's own Price List — serving region, service tier, cached tokens, and long-context rates included. And each end user's model calls can run under their own short-lived role session, so AWS reports their spend separately in Cost Explorer and the Cost and Usage Report — from the invoice itself, not from an estimate.
POST /v1/chat/completions200 · 1.9s
modelclaude-fable-5
region · tiereu-west-1 · priority
tokensin 12,410 (9,800 cached) · out 642
end userbilled under their own role session
estimated cost$0.048231 USD
- 0% markup on model usage — Bedrock billed by AWS directly
- live rates from the AWS Price List catalog — fetched from AWS, not hand-maintained
- per user spend on the AWS bill itself — grouped in Cost Explorer and the CUR
- batch price on asynchronous request sets — submit a corpus, pay Bedrock's discounted batch rate
Per-request cost figures are estimated from published AWS prices, not read back from your invoice; per-user attribution is off by default and needs a role you create.
{ .panel-stats__note }
[:octicons-arrow-right-24: Cost management documentation](operations_cost_management.md) · [compare model prices](models.md)
FOR DEVOPS TEAMS
## Production on AWS in two Terraform commands.
The validated Terraform module ships the whole stack — ECS Fargate, HTTPS, auto-scaling, and optional WAF and monitoring. It works as-is with secure defaults, and exposes advanced options for power users: VPC integration, multi-region, cost-optimized setups.
- 2 commands from AWS Marketplace subscription to a production endpoint
- FSBP aligned defaults out of the box — private subnets, least privilege, encryption at rest
- 100+ optional variables for power users — bring your own VPC, go multi-region, or cost-optimize
[:octicons-arrow-right-24: Deploy on AWS guide](operations_getting_started.md)
Prefer hands-off? A [managed deployment service](https://aws.amazon.com/marketplace/pp/prodview-xknxzjgl7zi5s) sets it up in your account — no Terraform required.
{ .panel-stats__note }
Amazon AI
Amazon Bedrock
Claude
DeepSeek
Amazon Polly
Meta Llama
Nvidia
Qwen
OpenAI GPT
xAI Grok
Moonshot AI
Amazon Translate
Mistral AI
Cohere
Stability AI
Minimax
Amazon Transcribe
AI21 Labs
Anthropic
Z.ai
Amazon Nova
Google Gemma
Luma AI
Twelve Labs
Amazon Comprehend
Writer
FOR REGULATED WORKLOADS
## No third party sits between your users and your models.
Unlike SaaS gateways, stdapi.ai is infrastructure you run. There is no vendor endpoint in the request path — your traffic goes from your application to your own deployment to AWS.
- :material-shield-lock: __Runs in your account__
Inference stays on the AWS services you enable. Bedrock does not share your prompts with model providers or use them for training.
- :material-earth: __Region allow-lists__
Pin workloads to approved regions, disable global routing, or use geography-pinned inference profiles where supported.
- :material-key: __Customer-managed encryption__
Bring your own KMS key for data at rest, with prompt and response bodies unlogged unless you enable it.
- :material-shield-star: __Security Hub aligned__
Terraform module built against AWS FSBP controls; GuardDuty and DNS Firewall opt-ins close the gaps.
AWS compliance certifications apply to the AWS services and regions you choose — they are not inherited by stdapi.ai or by your application. [:octicons-arrow-right-24: Data sovereignty & compliance guide](operations_compliance.md)
{ .band__note }
AWS Qualified Software Verified by AWS against its technical and security requirements for AWS Marketplace.
## How it compares
All four expose an OpenAI-compatible API in front of Amazon Bedrock — the coverage differs. stdapi.ai is AWS-only, and therefore AWS-deep: if you need multi-cloud routing or per-key spend budgets, LiteLLM is the better fit. Competitor capabilities verified against official sources on 5 August 2026.
| | stdapi.ai | LiteLLM | Access Gateway | Bedrock Mantle |
| ------------------------------------------------------------------------------------------- | ------------------------ | --------------------------- | --------------------------- | --------------------------- |
| Full multi-modal API — images, video, audio, files | ✓full | ◐partial | —not available | —not available |
| OpenAI + Anthropic + Cohere protocols | ✓full | ◐partial | —not available | ◐partial |
| Multi-region capacity — combine independent regional quotas | ✓full | ◐partial | —not available | —not available |
| Regional retry — throttling, region outages, retired models | ✓full | ◐partial | —not available | —not available |
| Zero-config model discovery — every region, Bedrock + Mantle | ✓full | ◐partial | ◐partial | ◐partial |
| AWS AI services & advanced Bedrock features — Polly, Transcribe, guardrails, service tiers | ✓full | ◐partial | ◐partial | ◐partial |
| Own AI & media APIs exposed as MCP tools | ✓full | ◐partial | —not available | —not available |
| Multi-provider routing beyond AWS | —not available | ✓full | —not available | —not available |
| Spend limits enforced at request time | —not available | ✓full | —not available | —not available |
| Per-request cost tracking & observability | ✓full | ✓full | —not available | ◐partial |
| Production AWS deployment — Terraform, auto-scaling, optional WAF | ✓full | ◐partial | ◐partial | ✓full |
MCP is not a like-for-like row: stdapi.ai exposes its own AI and media endpoints as tools, while LiteLLM gateways external MCP servers — related capabilities that solve different problems.
{ .compare__legend }
✓ full ◐ partial / manual setup — not available · [Full comparison](features.md#how-stdapiai-compares)
[See pricing](#transparent-pricing){ .md-button }
## Verified against the tools teams already use
Every integration is the same four steps: deploy, copy your endpoint URL, paste it into the tool's settings, then name a model the deployment serves — picked from the whole catalogue, not one vendor's list. There is no step five. The tools in **bold** are driven end to end by an automated suite against a real deployment — not just documented.
PRIVATE CHATGPT
Enterprise chat
ChatGPT-style assistant for your organization — chat, voice, images, and RAG, with every conversation staying in your account.
Open WebUI · wyoming-openai · LobeHub · LibreChat
[Open WebUI guide](use_cases_openwebui.md)
CODING AGENTS
AI-assisted development
Frontier coding models in your IDE and terminal — without sending your codebase to a third-party AI cloud.
Claude Code · Codex · Qwen Code · pi · OpenCode · Zed
Add AI steps to business processes with visual workflow builders — classification, summarization, content generation.
n8n · Haystack · Dify · Langflow · Flowise
[n8n guide](use_cases_n8n.md)
AUTONOMOUS AGENTS
Agents you control
Self-directed agents on infrastructure you own — with the built-in MCP server exposing every endpoint as an agent tool, and Cognito tokens giving each caller its own identity.
Speech-to-speech agents over a single WebSocket, transcription streamed phrase by phrase, and subtitles — on Amazon Bedrock, Polly and Transcribe, without a second AI vendor.
wyoming-openai · Pipecat · LiveKit Agents · Home Assistant
Media generation, knowledge management and team chatbots are covered too. [:octicons-arrow-right-24: All use cases & integration guides](use_cases.md)
{ .usecases__more }
PUBLIC ENGINEERING EVIDENCE
## Compatibility you can inspect
“Compatible” should mean more than one successful chat request. The test suite is public, and the same test bodies also run against the real OpenAI, Anthropic, and Cohere endpoints — so compatibility is measured against the originals, not asserted.
- :material-test-tube: __6,000+ test cases__
Run against real AWS services rather than mocks.
- :material-account-check: __20 client & framework suites__
Real CLIs, apps, and libraries driven end to end against a live deployment.
- :material-brain: __100+ model-probe records__
Committed observations of what each model actually accepts and rejects.
- :material-robot: __80+ MCP API tools__
Every exposed tool called end to end through the official MCP client.
- :material-shield-check: __95%+ branch coverage__
Measured across the full suite, with every test tier enabled.
[:octicons-arrow-right-24: Inspect the public test suite](https://github.com/stdapi-ai/stdapi.ai/tree/main/tests) · [what each client suite exercises](https://github.com/stdapi-ai/stdapi.ai/blob/main/tests/agentic/README.md)
{ .band__note }
## Transparent pricing
Start local, graduate to AWS — same API, same SDKs. And zero lock-in: leaving is the same client-side change that got you in.
Community
$0
AGPL-3.0 · Docker image · full API · local dev & open-source projects
Per running container — the Terraform module defaults to one per Availability Zone. No markup on model usage: pay Bedrock rates directly. Hardened container, Terraform module, commercial support (1 business day), no AGPL obligations. Billed through AWS Marketplace onto your existing AWS invoice — no new vendor onboarding.
Custom terms and duration, committed usage, and a preferential rate — procured through your existing AWS relationship, so there's no new vendor to onboard. Want to try first? Use the free trial, then accept your offer.
---
## Features
Source: https://stdapi.ai/md/features.md
# :material-check-all: Features — AI Gateway for Amazon Bedrock
stdapi.ai is an **AI gateway purpose-built for AWS**. It brings full OpenAI, Anthropic, and Cohere API compatibility to Amazon Bedrock and AWS AI services — so the tools, SDKs, and applications your team already uses run against your own AWS account, from the moment they point at a new base URL.
---
## :material-sitemap: How It Works
stdapi.ai translates OpenAI, Anthropic and Cohere API calls into native AWS requests. A tool or SDK that speaks one of the three protocols connects on the base URL alone — no plugins, no custom integrations.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
openwebui[" Open WebUI"] --> stdapi[" stdapi.ai"]
n8n[" n8n"] --> stdapi
ide[" IDE + AI Assistant"] --> stdapi
openai_app[" Any OpenAI App"] --> stdapi
anthropic_app[" Any Anthropic App"] --> stdapi
stdapi --> bedrock[" Amazon Bedrock"]
bedrock --> claude[" Claude"]
bedrock --> qwen[" Qwen"]
bedrock --> mistral[" Mistral"]
bedrock --> stability[" Stability AI"]
bedrock --> more["✨ and more..."]
stdapi --> transcribe[" Amazon Transcribe"]
stdapi --> polly[" Amazon Polly"]
stdapi --> s3[" Amazon S3"]
```
---
## :material-api: API Compatibility
### 80+ endpoints, all three protocols, every one on an AWS service
Not a chat proxy with a few extras: the OpenAI, Anthropic and Cohere surfaces are served in full, each endpoint backed by an AWS service running in your account.
| What your application calls it for | AWS service behind it |
|-----------------------------------------------------------------------------------------|------------------------------------------------------------------|
| Chat completions, Responses, Messages, legacy completions, token counting | Amazon Bedrock Converse API · Bedrock Mantle |
| [Server-side conversations](api_openai_conversations.md) and stored responses | Amazon Bedrock Sessions · Bedrock Mantle |
| [Embeddings](api_cohere_embed.md) and [reranking](api_cohere_rerank.md) | Amazon Bedrock embedding and rerank models |
| [Vector stores and file search](api_openai_vector_stores.md) | Amazon S3 Vectors · Amazon Bedrock Knowledge Bases |
| [Batch inference](api_openai_batches.md) | Amazon Bedrock batch inference |
| Image generation, editing and variations | Amazon Bedrock image models |
| [Video generation](api_openai_videos.md) | Amazon Bedrock video models |
| Text-to-speech | Amazon Polly |
| Transcription and speech translation | Amazon Transcribe · Amazon Translate · Amazon Nova Sonic |
| [Live speech-to-speech](api_openai_realtime.md) | Amazon Bedrock |
| [Content moderation](api_openai_moderations.md) | Amazon Bedrock Guardrails · Amazon Comprehend |
| Files and multipart uploads | Amazon S3 |
| [Model discovery](api_search_models.md) and [pricing](api_model_pricing.md) | Amazon Bedrock · AWS Price List |
Anthropic and Cohere routes live under `/anthropic` and `/cohere`, so all three protocols are served side by side without colliding on `/v1` — and either prefix can become the path your clients already send ([Anthropic](operations_configuration.md#anthropic-routes-prefix), [Cohere](operations_configuration.md#cohere-routes-prefix)).
[:octicons-arrow-right-24: Every endpoint, with its parameters](api_overview.md)
### Parameter Coverage
stdapi.ai maps as many parameters as possible to Bedrock equivalents — across all routes, not just chat:
- **Generation controls** — `temperature`, `max_tokens`, `top_p`, `top_k`, `stop`, `seed`, `frequency_penalty`, `presence_penalty`, `logit_bias`, `top_logprobs`, streaming via SSE, token usage reporting
- **Reasoning** — `reasoning_effort` (none/minimal/low/medium/high/xhigh), `enable_thinking`, `thinking_budget`
- **Tool / function calling** — Full OpenAI and Anthropic schemas, parallel tool calls, tool choice modes
- **All content types** — System, developer, user, assistant, and tool roles; text, image, audio, video, and document content
- **Response formats** — JSON object, JSON schema, streaming chunks, `reasoning_content`, `annotations`
- **Model-specific extras** — Any parameter beyond the standard API via `extra_body` or top-level request fields
!!! note "Bedrock & model differences"
Not every parameter maps identically across all models. Check the [API documentation](api_openai_chat_completions.md) for details.
---
## :material-brain: 100+ Models Across 20+ Providers
Access every model available on Amazon Bedrock through a single, consistent API — including OpenAI GPT, xAI Grok, and [other frontier models](#bedrock-mantle-models). [Browse the full list](models.md).
- { style="height: 1.2em; vertical-align: text-bottom;" } **Anthropic Claude**
Claude Fable/Mythos, Claude Opus, Claude Sonnet, Claude Haiku — including reasoning models. Use official Anthropic model names (e.g., `claude-fable-5`) — they resolve automatically.
- { style="height: 1.2em; vertical-align: text-bottom;" } **OpenAI GPT**
GPT frontier models plus open-weight gpt-oss, under OpenAI's own model names. The enrollment-gated Daybreak variants are served and priced with the rest of the family where your account is enrolled with OpenAI's Daybreak programme.
- { style="height: 1.2em; vertical-align: text-bottom;" } **Google Gemma**
Gemma 4 and other Gemma open-weight variants.
- { style="height: 1.2em; vertical-align: text-bottom;" } **Amazon Nova**
Nova — including reasoning-capable variants. Canvas for images. Multimodal embeddings. Built-in web grounding and code interpreter.
- { style="height: 1.2em; vertical-align: text-bottom;" } **Meta Llama**
Llama Scout, Maverick, and earlier Llama variants.
- { style="height: 1.2em; vertical-align: text-bottom;" } **Alibaba Qwen**
Qwen and Qwen3 Coder — including thinking mode.
- { style="height: 1.2em; vertical-align: text-bottom;" } **DeepSeek**
Latest DeepSeek V3 models with automatic reasoning content surfacing.
- { style="height: 1.2em; vertical-align: text-bottom;" } **Moonshot Kimi**
Kimi with optional thinking mode.
- { style="height: 1.2em; vertical-align: text-bottom;" } **Mistral AI**
Mistral, Mixtral, and Mistral Large variants.
- { style="height: 1.2em; vertical-align: text-bottom;" } **Cohere**
Command models for chat; Embed v4 for multimodal embeddings.
- { style="height: 1.2em; vertical-align: text-bottom;" } **Stability AI**
Stable Diffusion 3.5, SD3 Ultra, and specialty models (upscale, style, search).
- { style="height: 1.2em; vertical-align: text-bottom;" } **MiniMax & more**
MiniMax, xAI Grok, Writer Palmyra, AI21 Jamba, TwelveLabs Marengo video embeddings, and others.
This is a hand-picked sample, not the full roster — the [Models](models.md) page lists every model this gateway actually serves, generated from the live catalogue.
### Model Management
- **Automatic model discovery** — Configured regions are scanned at startup, so there is no model list to maintain by hand and nothing to keep in step with Bedrock's catalogue
- **Aliases** — A model is published under whichever names you choose, with Claude and OpenAI names resolving on their own; an alias can carry [its own service tier, guardrail, metadata and parameters](operations_configuration.md#model-aliases-configuration), so one model serves several policies under several names
- **Deprecation handled for you** — Models AWS has retired drop out of the list your users pick from, so nobody builds on a model about to be withdrawn, and requests naming one are redirected to its replacement; a workload that still depends on one can [keep it listed](operations_configuration.md#bedrock-legacy)
- **Capability discovery** — The [catalogue](api_search_models.md) advertises what each model can actually do — modality, route, streaming, speech to speech, transcription and translation, the search surfaces, and Batch API support — filterable over HTTP and through the same tool an agent reads before it calls anything
- **Published, not just discoverable** — The [Models](models.md) page lists every model this gateway serves on AWS, with modalities, regional availability, AWS prices and independent leaderboard scores
---
## :material-image-multiple: Multi-Modal Capabilities
### :material-chat: Text & Conversational AI
- [Server-side conversations](api_openai_conversations.md) — a thread is kept server-side and continued by id instead of resending the history, its items listed and managed, and a response attached to it through the Responses API `conversation` parameter; a long thread can be [compacted](api_openai_responses.md#conversation-compaction) into a reusable summary item rather than replayed in full
- Token counting before a call, so a prompt can be sized against a model's window without paying to generate
- Streaming over Server-Sent Events with tokens delivered as they arrive; reasoning content blocks on the models that produce them, and web search results as context
- Image, document, audio and video attachments on multimodal models — see [Attachment Size](#attachment-size) for how large ones are carried
### :material-image: Images
- **Generation** — Text-to-image in PNG, JPEG or WebP, at the size, aspect ratio, quality and compression asked for, with model-specific style presets and partial previews streamed while it renders
- **Editing** — Mask-based inpainting over a region you define, image-to-image transformation on style or structure, background removal, object search-and-replace, object recolor, and creative or conservative upscaling
- **Variations** — Alternative versions of an existing image
- **Nothing re-uploaded** — An input image is referenced by Files API `file_id` or by URL rather than sent again with every call
### :material-microphone: Audio
**Text-to-speech (Amazon Polly)**
- 60+ voices across 30+ languages, on the Standard, Neural, Long-Form and Generative engines, with the language detected automatically
- SSML control over pronunciation, emphasis, pauses and prosody, at 0.2× to 2.0× speed; MP3, PCM, Opus, AAC, FLAC and OGG Vorbis output
- Long input — up to 100,000 characters per request, 24× OpenAI's limit ([20,000 with a generative voice, which speaks it as the audio is delivered](api_openai_audio_speech.md#long-input))
**Speech-to-text (Amazon Transcribe)**
- 100+ languages, detected automatically when the request does not name one
- Speaker diarization, word-level and segment-level timestamps, and SRT/VTT subtitle export
- Vocabulary customization and custom language models, per language — so a request identifying between several applies the right resources to each one
- [Streamed results](api_openai_audio_transcriptions.md#streaming) — each phrase comes back as it is recognized instead of after the whole recording, whenever the request names the language to expect
- [Transcripts encrypted with your own KMS key](operations_configuration.md#aws-transcribe-output-encryption-key-arn), under a key policy scoped to this workload rather than to a whole bucket
**Speech translation** — Transcribe audio and translate to English in a single request; a language pair that cannot be served is refused as a request problem instead of failing after the audio has been transcribed.
**Speech-to-text (Amazon Nova Sonic)** — An alternative backend on both audio routes, at the lowest transcription cost available here — about $0.006 per minute of audio at current Amazon Bedrock rates. Punctuated transcripts in the language spoken, with automatic language detection, and translation to English produced by the model itself in one request. `json` and `text` output only, up to 10 minutes of audio per request — no timestamps, subtitles, diarization or detected-language reporting.
**Live speech-to-speech ([Realtime API](api_openai_realtime.md))**
- Bidirectional audio over a single WebSocket, OpenAI Realtime API compatible
- 24 kHz PCM, or G.711 at 8 kHz for telephony; server-side voice activity detection or manual turn control, with barge-in on the item the caller spoke over
- Ephemeral, browser-safe client secrets, minted by one instance behind a load balancer and verified by any other
- A configured guardrail is applied per turn — a written item is checked before it reaches the model, a spoken answer once it is complete ([guardrail coverage](api_openai_realtime.md#guardrail-coverage))
- **Its limits, up front** — a session lasts at most 8 minutes and calls no tools, and WebSocket is the only transport: upstream's WebRTC and SIP call route is not served. For WebRTC or telephony, LiveKit Agents and Pipecat terminate the media themselves and reach this API like any other client — see [transports](api_openai_realtime.md#transports) and the [feature compatibility table](api_openai_realtime.md#feature-compatibility)
### :material-file-document: Documents & Files
- PDF input with optional citations — the answer points back at the exact source passage
- Plain text and structured content blocks as context; a large PDF or document is carried by reference on a model that reads it from storage — see [Attachment Size](#attachment-size)
- Upload once and reference by ID across requests, with expiry anywhere from 1 hour to 30 days
- Multipart uploads for large files, backed by S3's own multipart
### :material-video: Video
- Text-to-video and image-to-video generation (Amazon Nova Reel, Luma Ray 2) through the OpenAI Videos API, as an asynchronous job — create, list, poll, download, delete
- Video as chat input on the models that read it (Amazon Nova among them); long clips follow the [Attachment Size](#attachment-size) policy
- `s3://` URLs as direct video input for multimodal embeddings
### :material-paperclip: Attachment Size
On chat completions, messages and responses served by Amazon Bedrock, every attachment — an image, document, audio or video sent as base64, a data URI, an HTTPS URL, an `s3://` URI or a Files API ID — is measured before the request is built, and carried the way that model accepts:
| Attachment size | How it is sent |
|------------------------------------------------------------|--------------------------------------------------------------------------|
| Within the model's inline capacity | Embedded in the request |
| Above it, on a model that reads that kind from storage | Staged in [a bucket of yours](operations_configuration.md#aws-s3-regional-buckets) in the region serving the request, and referenced |
| Above it, on a model that reads it inline only | Refused with `413`, stating the size that model accepts |
Models differ in their per-attachment and per-request limits and in which kinds they read from storage — the Amazon Nova families read images, documents and video that way and TwelveLabs Pegasus reads video, while the rest read attachments from inside the request — so the same file can be inline for one model and staged for another, with nothing in the request changing either way. An attachment already in S3 is referenced as it stands whatever its size, on a model that reads that kind from storage. Bedrock Mantle-served models, image editing and variations, transcription and the embeddings routes keep their own input handling.
### :material-vector-polyline: Embeddings
- Text embeddings, single or batched, and multimodal embeddings over images, audio, video and PDF documents
- Dimension reduction where the model offers it, in float or Base64 encoding
- `s3://` input for large files, with oversized base64 payloads staged to S3 for you
### :material-database-search: Retrieval & Vector Stores
There is no embedding pipeline, chunker or vector database to run alongside the gateway. The [Vector Stores API](api_openai_vector_stores.md) takes an attached file, chunks it, embeds it and indexes it in the background, reports the indexing as it progresses, then searches it by meaning.
- **Search by meaning, with the passages** — Each result carries its file, its text, its score and the attributes stored with it
- **Attribute filters** — Tag a file with up to 16 attributes and restrict a search to the ones that match
- **File batches and expiration policies** — Attach many files under a single identifier; expire a store after a number of days without a search
- **Held in your own account** — Documents, passages and vectors live in an [Amazon S3 vector bucket](operations_configuration.md#aws-s3-vectors-bucket) in your account, not in an index somebody else runs
#### Point it at the knowledge base you already run
A store can equally be [an Amazon Bedrock knowledge base you already run](api_openai_vector_stores.md#knowledge-base-stores), addressed through the same endpoints — searched, its documents attached, listed and read. Bedrock-managed and customer-managed knowledge bases over documents are both served, whichever way you built yours; one built over a structured data store or an Amazon Kendra index is not, since it answers with database rows or search hits rather than passages.
It stays yours. Knowledge bases are **allowlisted one by one**, never created or deleted here, and anything that would reshape one — renaming, expiry, chunking strategy, rewriting a file's attributes — is refused, naming why. A request naming a knowledge base that is not on the allowlist answers exactly as one that does not exist, so the allowlist cannot be probed for what a deployment holds.
#### The model does the retrieving
[`file_search` on `/v1/responses`](api_openai_responses.md#file-search) gives **any chat model served on that route** the stores you name, whether managed or knowledge-base backed. The model decides when to search and with which query; each search is reported as a `file_search_call` item carrying the queries it used (and the passages themselves on request, plain or streamed), and the grounded answer carries a `file_citation` annotation for every file it drew on. A filter operator the serving store cannot apply, or a score threshold against a store whose scores have no defined scale, is refused with a `400` rather than quietly dropped — so an answer does not come back as though it had honoured a restriction it ignored.
---
## :material-aws: Purpose-Built for AWS
### Multi-Region Routing & Quota Headroom
A deployment spanning several AWS regions draws on more than one Bedrock quota and keeps serving when one region is degraded. How traffic spreads across them is yours to [choose](operations_configuration.md#bedrock-region-routing), and the choice trades throughput against prompt-cache hit rate:
| Routing across regions | What it gets you | Prompt Caching |
|------------------------|---------------------------------------------------------|----------------|
| In order | Deterministic placement; blocked regions skipped | ✓ Compatible |
| Lowest latency | The fastest measured region for each call | ✓ Compatible |
| Round robin | Load spread evenly, at the cost of cache locality | —not compatible |
| Single region | Every call to a model served from one place | ✓ Compatible |
- **Each region adds its own quota** — Bedrock tokens-per-minute and requests-per-minute limits are per region, so a multi-region deployment draws on several independent quotas rather than one. How much of that headroom a workload reaches depends on the quota granted per model in each region and on the routing strategy
- **Eligible failures retry elsewhere** — A throttle, quota or service error switches region transparently, under a backoff that widens while a region keeps failing. Streaming responses can only retry before the stream opens, and asynchronous jobs stay in the region that accepted them
- **Health is tracked per model** — A region that failed for one model is set aside for that model alone and brought back once it recovers, rather than taking the whole catalogue down with it
[:octicons-arrow-right-24: Resilience & Failover](operations_resilience.md)
### Advanced Bedrock Features
| Feature | Description |
|------------------------------------|------------------------------------------------------------------------------------------------------------------------|
| **Prompt Caching** | Cache system prompts, messages and tools section by section, at the TTL you choose — a long system prompt is billed at the cache-read rate on later turns instead of in full, with cache metrics in the response |
| **Reasoning Modes** | Extended thinking on Claude and Nova, driven by effort level or by a token budget |
| **Bedrock Guardrails** | Content filtering and safety policies applied to traffic from every client, with the trace detail you choose |
| **Service Tiers** | Priority, default, flex and reserved tiers, per request or as a default per model — a latency-sensitive workload and a cheap bulk one share the deployment |
| **Application Inference Profiles** | Isolate a workload and see it separately on the AWS bill |
| **Prompt Routers** | Bedrock prompt routers for intelligent model selection |
| **Cross-Region Inference** | Geography-pinned (US, EU, APAC) and global profiles, so inference stays inside the geography your data residency requires |
| **Web Search / Grounding** | Built-in web search with source citations, billed per query: Amazon Nova grounding (Chat Completions, Responses, and Messages) and OpenAI GPT built-in search (`/v1/responses` only). A deployment can [keep grounding off the open internet](operations_configuration.md#bedrock-external-web-access) entirely, and a request asking for what it forbids is refused rather than silently rewritten |
| **Server-side tools** | Amazon Nova's code interpreter, and Claude's bash, text editor, computer use and memory tools on the generations that carry them |
### :material-tray-full: Asynchronous Batch Inference
Large request sets run asynchronously at Amazon Bedrock's discounted batch price, on both dialects — [`/v1/batches`](api_openai_batches.md) and the Anthropic [`/v1/messages/batches`](api_anthropic_batches.md). Submit, poll, collect, cancel.
- **Chat or a whole corpus** — On the OpenAI surface a batch is a JSONL file of chat completion *or* [embeddings](api_openai_embeddings.md) requests, each result carrying its own `custom_id`
- **A model per request** — An Anthropic batch may name a different model for each request and is still submitted, tracked and collected as a single batch, whatever it fans out to
- **Priced as batch** — Usage is recorded and priced at the tier that actually served the call, so a batched request is reported at the batch rate rather than the on-demand one
- **Discoverable before you submit** — The [model catalogue](api_search_models.md) reports and filters on Batch API support. It is best effort: a model without the flag is still submitted, since the absence may only mean no price is published for it yet
- **Result files expire on your terms** — rather than being kept until deleted
!!! note "Batches run under a role of yours"
Amazon Bedrock reads the requests and writes the results itself, under an IAM role and a bucket of yours — the batch runs on your account's terms, not the gateway's. See [Batch inference IAM](operations_iam_permissions.md#batch-inference).
### :material-layers-triple: Bedrock Mantle Models { #bedrock-mantle-models }
stdapi.ai serves models from the **Amazon Bedrock Mantle** endpoint alongside the classic Bedrock catalog: OpenAI GPT, xAI Grok, Google Gemma, Qwen, GLM, DeepSeek, MiniMax, Kimi, Nemotron, and more — the available catalog varies per region and grows over time.
- **Every text API, every model** — All four text APIs (chat completions, responses, messages, legacy completions) work with every Mantle model: served natively when the model supports the API upstream, converted automatically otherwise
- **Predictable routing** — A model available on both endpoints is served by the classic one, so nothing silently changes backend under an existing workload; Mantle serves the models only it has. A dual-homed model can be [pointed at Mantle](operations_configuration.md#bedrock-mantle-preferred-models) for the whole deployment or for a single request, tapping Mantle's separate throughput quotas on top of your Bedrock ones
- **The same operational behaviour** — Region failover, quota backoff, usage recording and pricing work as they do on classic Bedrock; requests chained via `previous_response_id` stay pinned to their origin region; and access runs on the same AWS credential chain, with no separate API key to issue, store or rotate
- **Native stored conversations** — `/v1/responses` with `store` and `previous_response_id` uses Mantle's own server-side storage: 30-day retention, region-local, project-scoped
- **Built-in web search** — The OpenAI GPT-5.x family grounds answers in current web content with source citations on `/v1/responses`, inside the AWS boundary by default
Mantle models appear in the same catalogue as the rest, under the same `/v1/models` call — nothing in a client distinguishes them. A deployment whose IAM policy does not reach Mantle simply does not list them.
!!! note "Limitations & conversion details"
Bedrock Guardrails and cross-region inference profiles do not apply to Mantle-served requests, and the built-in [`web_search` tool](api_openai_responses.md#openai-gpt-web-search) is served on `/v1/responses` only. API-shape conversion preserves the core request semantics (messages, tools, sampling, streaming, usage); parameters with no equivalent in the serving API are dropped or adapted. The exact parameter tables, response-ID specifics, and per-route limitations are on the API pages: [chat completions](api_openai_chat_completions.md#bedrock-mantle), [responses](api_openai_responses.md#model-support), [messages](api_anthropic_messages.md#bedrock-mantle), and [legacy completions](api_openai_completions.md#feature-compatibility).
[:octicons-arrow-right-24: Bedrock Mantle Configuration](operations_configuration.md#bedrock-mantle-enabled)
### Amazon S3 as the file layer
S3 backs the whole API surface, not just file storage, which buys three things a file API bolted onto a database cannot:
- **No artificial size ceiling** — Files go up to S3's own limit of roughly 5 TB, uploaded in native multipart parts and streamed rather than buffered. One file ID works on both the OpenAI and the Anthropic endpoints
- **`s3://` is a first-class input** — An object already in your buckets is named directly in chat completions, Messages, embeddings and image operations, read under the gateway's IAM role: no pre-signed URLs, no download-and-re-upload round trip
- **Region-local by construction** — Anything the gateway stages sits in a bucket in the region serving the request, so payloads do not cross a region on the way to the model; a generated image can be handed back over [S3 Transfer Acceleration](operations_configuration.md#aws-s3-accelerate), downloaded from a CloudFront edge instead of the bucket's region
---
## :material-shield-lock: Security & Compliance
### Authentication
| Method | How | Best For |
|---------------------------|--------------------------------------------------------------------------------------------------------------------|------------------------------|
| **API Key** | `Authorization: Bearer` or `X-API-Key` header; stored in SSM Parameter Store or Secrets Manager (never plain text) | Direct clients, SDKs |
| **Cognito user pool JWT** | `Authorization: Bearer` with an Amazon Cognito access token, validated per request | Per-user access, agents |
| **OIDC / Cognito** | Delegate to AWS Application Load Balancer or API Gateway | Web apps, SSO |
| **AWS IAM (SigV4)** | Via API Gateway with IAM authorization | Internal AWS services |
| **No authentication** | Open access | Private VPC deployments |
- **Per-caller identity** — [Amazon Cognito user pool tokens](operations_configuration.md#cognito-authentication) are accepted instead of, or alongside, the API key, so each caller reaches the API with their own credential — validated in-process against the pool's published keys, with no AWS call on the request path, and it is that verified identity [per-user cost attribution](#per-user-cost-attribution) bills against
- **The posture is asserted, not inferred** — [Name the method you intend to run](operations_configuration.md#authentication-mode): the server refuses to start when the method you named is not actually in force, or when one you configured would be silently ignored, so a deployment cannot drift into answering unauthenticated traffic
- **Agents authenticate without being configured** — Every unauthorized response points at the [document](operations_configuration.md#oauth-discovery) naming the authorization server and the scope this deployment expects
[:octicons-arrow-right-24: Authentication & Security](operations_authentication_security.md)
### Security Features
- **A URL in a request stays outside your network** — Loopback, link-local and private addresses are refused, along with DNS rebinding; hostname allowlisting, CORS policy and CSRF protection govern what may call the service and from where
- **Malformed requests stop at the edge** — Out-of-spec requests are rejected before they reach an AWS call, under a [strict mode](operations_configuration.md#strict-input-validation) that also refuses unknown fields instead of ignoring them
- **API keys are not stored in the clear** — Held in SSM Parameter Store or Secrets Manager, kept in memory only as a salted hash, and compared in constant time so a key cannot be recovered by timing
- **Encrypted in transit** — TLS 1.2+ on every AWS service call; the Terraform module terminates client traffic on TLS 1.3 with post-quantum hybrid key exchange, and forwarded headers from ALB and CloudFront are processed safely
- **A hardened supply chain** — A validated container image built without exposure to a public package registry at run time
!!! success "Commercial: hardened image, Security Hub validated :material-arrow-right: [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-su2dajk5zawpo)"
The commercial image is security-validated by AWS Marketplace: **read-only root filesystem**, **dropped Linux capabilities**, minimal installed packages, no shell. The Terraform module is built against the **AWS Security Hub Foundational Security Best Practices** standard, passes a large share of applicable controls out of the box, and configures a **Customer Managed KMS key** with auto-rotation for all data at rest. Optional variables add native **GuardDuty Runtime Monitoring** and **Route 53 Resolver DNS Firewall** on the module's dedicated VPC.
[:octicons-arrow-right-24: AWS Security Hub, GuardDuty & DNS Firewall Integration](operations_authentication_security.md#aws-security-hub-guardduty-dns-firewall-integration)
### Compliance & Data Sovereignty
The gateway runs on infrastructure you own, so no third party sits between your users and your models, and Amazon Bedrock does not store your prompts or use them to train models. AWS service calls are restricted to the regions you configure. The AWS services used by stdapi.ai (Bedrock, S3, Polly, Transcribe, and more) are in scope for **GDPR**, **ISO 27001/27017/27018**, **SOC 1/2/3**, **HIPAA**, **FedRAMP**, **PCI-DSS**, and **CSA STAR Level 2** — these certifications apply to the AWS services and regions you choose, and are not inherited by stdapi.ai or by your application. The commercial Terraform module adds VPC endpoints (no internet egress), Customer Managed KMS keys, and region-pinned cross-region profiles for strict data residency.
[:octicons-arrow-right-24: Data Sovereignty & Compliance](operations_compliance.md)
---
## :material-puzzle: Works with Your Existing Tools
stdapi.ai speaks the APIs hundreds of applications and tools already speak, so adoption is quick: an application points at your deployment instead of the vendor's, with the key your gateway issues. Model names carry over — Claude and OpenAI names resolve on their own — and the name is now drawn from every provider in the catalogue rather than one vendor's list. A name the catalogue does not hold returns `404` instead of a lookalike, and a served model can be [published under whichever name your application already sends](operations_configuration.md#model-aliases), so an application whose model name is not yours to change keeps working untouched.
- :material-chat: **Chat Interfaces**
Open WebUI, LobeHub, AnythingLLM, LibreChat — private ChatGPT-style experiences on AWS
- :material-code-braces: **AI Coding Assistants**
Claude Code, Cline, OpenCode, Pi Agent, Zed — backed by Claude, Kimi, Qwen3 Coder
- :material-graph-outline: **Workflow Automation**
n8n, Langflow, Dify, Flowise — connect AI to your business processes
- :material-robot: **Agent Frameworks**
OpenClaw, Hermes Agent, LangChain, LangGraph, CrewAI, OpenAI Agents SDK, Pydantic AI, Agno, Strands Agents — multi-agent systems on Bedrock
- :material-microphone-message: **Voice & Audio**
Pipecat, LiveKit Agents, TEN Framework, Home Assistant — voice agents on live speech-to-speech, transcription, and translation
- :material-magnify: **RAG & Semantic Search**
LlamaIndex, Haystack, RAGFlow, Docling, LightRAG — built-in vector stores, embeddings and Cohere-compatible reranking
Team chatbots in Slack, Discord or Microsoft Teams and knowledge tools such as Obsidian Copilot, Khoj and SiYuan connect the same way.
[:octicons-arrow-right-24: See all use cases](use_cases.md)
---
## :material-robot: AI Agents
### Agent Discovery
An agent that has only the base URL can work out the rest for itself, through standards rather than a hand-written integration: RFC 8288 `Link` headers on `/` and an RFC 9727 catalog at `/.well-known/api-catalog` point at the [OpenAPI schema](operations_configuration.md#enable-openapi-json), the documentation and the SEP-1649 MCP server card, which advertises the transports on offer. It can work out how to *authenticate* itself the same way: an RFC 9728 [protected resource metadata document](operations_authentication_security.md#authentication-discovery-for-agents) names the authorization servers issuing tokens for this deployment and the scopes they need, and every `401` carries its address in the `WWW-Authenticate` challenge — so an MCP client reaches a secured deployment it was never configured for.
### MCP (Model Context Protocol)
stdapi.ai exposes its full API surface as MCP tools, letting AI agents and orchestrators call any endpoint directly through the Model Context Protocol — no HTTP client code required.
- **80+ tools, no integration code** — Every API operation (chat, images, audio, embeddings, files, model search) is a named MCP tool with generated documentation, over Streamable HTTP at `/mcp` or SSE at `/sse` for older clients
- **A tool surface you choose** — [Tools are included or excluded by name](operations_configuration.md#mcp-exclude-tools), so an agent is handed exactly the capabilities it should have and no more — a read-only deployment, or one without file deletion, is a list away
- **Written for an agent's context window, not a human's** — Schemas hide parameters an MCP client cannot use and results come back as compact JSON, so each call costs the calling agent fewer tokens
- **Media-aware results** — An endpoint answering with bytes returns an image or audio result the agent can use directly, falling back to a download reference for what the protocol cannot carry, such as a generated video
[:octicons-arrow-right-24: MCP Configuration](operations_configuration.md#mcp-model-context-protocol)
---
## :material-chart-line: Observability & Operations
### Logging & Tracing
- **JSON to stdout, ingested by CloudWatch as it stands** — Every request logs its method, path, status, model, the region or regions that served it and how long it took, so a slow model or a region that started failing is one query away
- **Prompts stay out of your logs unless you ask for them** — Full request and response payloads and the client IP are [available when you are debugging](operations_configuration.md#log-request-params) and are not written otherwise
- **Traces and metrics into what you already run** — OpenTelemetry export to AWS X-Ray, Datadog, Jaeger or any OTLP backend, one root span per request with correlation IDs, sampled at a rate you set
### Cost Tracking
- **Usage counts read back from AWS** — Token, character, second and image counts come from the AWS responses themselves rather than from client-side counting; recorded per request across chat, embeddings, images, audio and built-in tools, and reported to the caller in the same `usage` shape on every endpoint — input, output, reasoning and cached tokens included
- **Priced on the dimensions AWS bills on** — From AWS's own price list, refreshed automatically with no list to maintain by hand: serving region, service tier (standard, flex, priority or batch — the tier that *actually served* the call), prompt-cache TTLs, cross-region and latency-optimized routing, long-context rates, image resolution and quality. Operator overrides cover any gap
- **Beyond tokens** — Built-in web searches are counted per query, and a search against a knowledge base the backend manages at its published rate per call. What cannot be accounted for is [stated rather than approximated](operations_cost_management.md#vector-stores)
- **Currency-safe figures** — Cost appears in the request log in your AWS partition's own currency (USD, EUR, CNY), as exact decimal amounts that are never summed across two
- **Model Pricing API** — The loaded catalog is queryable at [`GET /model_pricing`](api_model_pricing.md), over HTTP or as an MCP tool, for cost-aware model selection; spend can also be published to CloudWatch as EMF metrics
!!! note "An estimate, not a bill"
Costs are estimated from AWS's published prices, not read back from your invoice — a best-effort figure for visibility and alerting. See [Cost Tracking](operations_cost_management.md#cost-tracking-real-time-aws-pricing) for its accuracy and known limitations.
### Per-User Cost Attribution
- **Each end user on the AWS bill** — Model calls run under a short-lived role session opened for the user behind the request, so AWS reports their spend separately in Cost Explorer and CUR 2.0 — from the invoice, not from an estimate
- **The identity the gateway verified** — The authenticated caller where authentication is enabled, otherwise the identifier the request declares (`safety_identifier`/`user`, or `metadata.user_id` on the Anthropic Messages API). It travels as a session tag: a cost allocation dimension in Cost Explorer, and an access boundary testable in IAM as `aws:PrincipalTag`
- **Fail-closed** — A session that cannot be opened fails the request rather than quietly billing the gateway, and requests identifying no user can be rejected outright
!!! note "It covers model invocations"
Each user's model calls are attributed; the rest of the gateway's AWS usage stays on its own identity — see [Per-User Attribution](operations_cost_management.md#per-user-attribution) for the role those calls run under.
### Day-to-Day Operation
- **An API reference on the deployment itself** — Swagger UI at `/docs` to try an endpoint in a browser, ReDoc at `/redoc` to read it, and the OpenAPI schema at `/openapi.json` to generate a client or import into Postman
- **Proxy-aware outbound connections** — `HTTPS_PROXY`, `HTTP_PROXY` and `NO_PROXY` are honoured by the connections the server makes to AWS and to model endpoints, not by the AWS SDK alone ([proxied deployments](operations_deploy_advanced.md#proxied-deployments))
- **A missing permission reads as one** — An unconfigured resource or a denied AWS call answers `503` `feature_unavailable`, with the server log naming the operation, the model and the permission AWS refused, instead of reaching clients as their own key being rejected
---
## :material-speedometer: Performance
A gateway earns its place by adding as little as possible on top of the model call. Measured gateway CPU on the production serving stack, single worker, over the complete request path:
| Request shape | Gateway CPU per request |
|---|---|
| Typical chat request (2.5 KB) | **0.8 ms** |
| Large context (1 MB body) | **4.6 ms** |
| Large context, streamed (~100 events) | **8.6 ms** |
Independent work fans out concurrently; JSON, the AWS wire format and the HTTP serving stack all run compiled; and a streamed response is passed through as it arrives rather than buffered — which is why those figures hold precisely where load does, on large contexts and streaming.
!!! note "Negligible next to the model call"
Even at its most expensive — a 1 MB request — the gateway's processing adds a few milliseconds to an invocation the model itself takes seconds to answer: well under 1% of end-to-end latency. Measured live, a typical chat completion spends about a millisecond in the gateway out of a several-hundred-millisecond round trip — a share that holds even with the server capped to 0.25 vCPU, the smallest Fargate task size.
---
## :material-clipboard-check: Quality Assurance
"OpenAI-compatible" is easy to claim and hard to keep. The gateway sits between your application and a set of backends that change underneath it — new model generations, new parameters, silently altered behaviour — so its test suite is built to catch that drift before you do.
- :material-test-tube: __6,000+ automated tests__
Run against real AWS services rather than mocks.
- :material-shield-check: __95%+ branch coverage__
Measured with every test tier enabled — the full suite, not the offline CI gate that runs on every push.
- :material-account-check: __Validated against the vendors__
The same tests run against the real OpenAI, Anthropic and Cohere APIs.
- :material-robot-happy: __Driven by real client software__
Twenty third-party clients — Claude Code, n8n, Open WebUI and more — driven end to end against a live gateway.
### Compatibility is proven against the real APIs, not against our own docs
A gateway tested only against itself proves only that it is self-consistent. The suite runs **the same test bodies against the genuine OpenAI, Anthropic and Cohere endpoints**, changing nothing but the URL, and a test that passes here and fails there is a compatibility bug in this product rather than an acceptable difference. The expected behaviour is captured from the vendor's own API before the feature exists, so the implementation is written against a contract that was verified, not assumed.
### Real applications, not just API calls
Passing an HTTP test says nothing about whether a real coding agent, chatbot or automation platform actually works. So the suite runs them: **complete third-party client software, unmodified, against a live gateway** — coding agents (Claude Code, Codex, pi, OpenClaw, Hermes, Qwen Code), the n8n workflow runner, a Haystack RAG pipeline, the Open WebUI chat interface, Home Assistant's voice bridge, the LiteLLM proxy, Docling Serve's vision pipeline, LiveKit Agents and Pipecat on the [documented WebRTC and telephony configurations](api_openai_realtime.md#transports), and the LangChain, LlamaIndex, Pydantic AI, Agno, inspect-ai and OpenAI Agents SDK libraries — driving multi-turn tool-calling sessions, document retrieval and reranking, server-side conversations, vector-store retrieval loops and speech pipelines across dozens of models and all three API dialects. A regression that only appears in a real client is caught here, not by you.
### Every model is measured, not assumed
Model documentation describes what a model is supposed to accept. What it actually accepts is something else, and it changes between generations of the same model family. Each supported model is **probed empirically** for the parameters and features it genuinely honours — including the parameters it accepts and then quietly ignores, which are the ones that silently produce a wrong answer. The results are recorded and the behaviour is pinned by tests. [See what each model reports](models.md).
### Beyond automated tests
- **Every release is validated twice** — the complete suite locally, then again against a real deployment of the exact build being released
- **Static analysis and strict typing** on every line of source and test code
- **Documented behaviour is tested behaviour** — the compatibility tables on these pages are derived from what the tests assert, so a change in behaviour that the docs do not reflect fails the build
- **Continuous review** — periodic audits of the codebase and of the test suite itself, because a passing test that asserts nothing is worse than no test
- **Security scanning** of dependencies and container images, with the commercial image validated against AWS Security Hub's Foundational Security Best Practices
!!! tip "Found something anyway?"
No suite catches everything. If you hit a compatibility difference between stdapi.ai and the API it mirrors, open an issue on [GitHub](https://github.com/stdapi-ai/stdapi.ai/issues) — or [email us](contact.md) if you would rather not report it publicly. That class of bug is treated as a defect, not as expected behaviour.
---
## :material-rocket-launch: Deployment
### Community vs Commercial
| | :material-docker: **Community** | :material-aws: **Commercial** |
|-------------------------------|---------------------------------------------------------|---------------------------------------------------------------------------------------------------|
| **Price** | Free | $0.10/container-hour - With 14-day free trial |
| **License** | AGPL-3.0 | AWS Marketplace SCMP |
| **API compatibility** | ✓ Full | ✓ Full |
| **Container image** | ◐ Community (GHCR) | ✓ Hardened, AWS Marketplace validated |
| **Deployment** | ◐ Docker / self-managed | ✓ Terraform module (ECS Fargate) - AWS Marketplace container image |
| **Production infrastructure** | —not available | ✓ Fully featured - AWS Well-Architected - Hardened |
| **Security posture** | Manual (self-managed) | ✓ Security Hub FSBP validated by default; GuardDuty & DNS Firewall integrations |
| **Commercial support** | —not available | ✓ 1 business day |
---
## :material-scale-balance: How stdapi.ai Compares
All four solutions below expose an OpenAI-compatible API in front of Amazon Bedrock. The comparison focuses on the AWS deployment context — LiteLLM is evaluated with AWS services as the backend provider (Bedrock, Polly, Transcribe), not as a multi-cloud proxy. Bedrock Access Gateway is the official AWS-maintained open-source sample. Bedrock Mantle is AWS's own managed OpenAI-compatible endpoint, requiring no self-hosting — and stdapi.ai can also front it as an additional backend, serving Mantle-only models through the gateway (see [Bedrock Mantle Models](#bedrock-mantle-models)).
Competitor capabilities were verified against official sources on 5 August 2026; the batch inference, vector store, file search and per-end-user billing rows were verified on 21 August 2026; the image, audio and video rows were re-verified on 24 August 2026. stdapi.ai is AWS-only: if you need multi-cloud routing or spend limits enforced at request time, LiteLLM is the better fit.
!!! note "Cost attribution is not a spend limit"
Bedrock's native attribution — IAM principal, application inference profiles, projects and workspaces — reports [aggregated billed cost to Cost Explorer and CUR 2.0 at per-usage-type-per-day granularity](https://docs.aws.amazon.com/bedrock/latest/userguide/cost-management.html), not as a per-request row. That is reporting, not enforcement: it cannot block a request that would exceed a budget. stdapi.ai's per-request cost figures are likewise an estimate for visibility. Enforcing a hard limit requires a gateway layer in front of inference — the approach AWS itself takes in its [Generative AI Gateway Solution](https://aws.amazon.com/solutions/), which uses LiteLLM.
| Capability | stdapi.ai | LiteLLM (on AWS) | Bedrock Access Gateway | Bedrock Mantle |
|---------------------------------------------|:-----------------------------------:|:---------------------------------------:|:---------------------------------------:|:---------------------------------------:|
| **OpenAI Chat completions** | ✓full | ✓full | ✓full | ◐partial [^2] |
| **OpenAI Completions** | ✓full | ✓full | —not available | —not available |
| **OpenAI Embeddings** | ✓full | ✓full | ✓full | —not available |
| **Anthropic Messages API** | ✓full | ✓full | —not available | ◐partial [^14] |
| **OpenAI Responses API** | ✓full | ◐partial [^18] | —not available | ◐partial [^2] |
| **OpenAI Image generation** | ✓full | ✓full | —not available | —not available |
| **OpenAI Image editing** | ✓full | ◐partial [^28] | —not available | —not available |
| **OpenAI Image variations** | ✓full | —not available | —not available | —not available |
| **OpenAI Videos API** | ✓full | —not available | —not available | —not available |
| **OpenAI Moderations API** | ✓full | —not available | —not available | —not available |
| **OpenAI TTS (speech)** | ✓full | ◐partial [^12] | —not available | —not available |
| **OpenAI STT (transcription)** | ✓full | —not available | —not available | —not available |
| **OpenAI Files & Uploads API** | ✓full | —not available | —not available | —not available |
| **Asynchronous batch inference** | ✓full | ✓full [^21] | —not available | —not available [^22] |
| **Managed vector stores** (create, attach, index, search) | ✓full | ◐partial [^23] | —not available | —not available |
| **Model-run file search, with citations** | ✓full | ◐partial [^24] | —not available | —not available |
| **Server-side stored conversations** | ✓full | ◐partial [^17] | —not available | ✓full |
| **OpenAI Realtime API** | ✓full | ✓full | —not available | —not available |
| **Cohere Rerank API** | ✓full | ✓full | —not available | —not available |
| **Cohere Embed API** | ✓full | —not available | —not available | —not available |
| **Own AI & media APIs as MCP tools** | ✓full | ◐partial [^20] | —not available | —not available |
| **Bedrock Full model catalog** | ✓full | ◐partial [^1] | ◐partial [^10] | ◐partial [^2] |
| **Unified Bedrock + Mantle catalog** | ✓full | ◐partial [^15] | —not available | —not available |
| **Every model on every text API** | ✓full | ◐partial [^16] | —not available | —not available |
| **Multimodal inputs** | text · image · audio · video · docs | text · image · docs | text · image | text · image |
| **Multi-region capacity combining** | ✓full | ◐partial [^8] | —not available | —not available |
| **Bedrock Cross-region inference profiles** | ✓full | ◐partial [^13] | ◐partial [^13] | —not available |
| **Bedrock system tools** | ✓full | —not available | —not available | —not available |
| **Bedrock Guardrails** | ✓full | ✓full | —not available | —not available |
| **Bedrock Service tiers** | ✓full | ✓full | —not available | ✓full |
| **Bedrock Application inference profiles** | ✓full | ✓full | ✓full | —not available [^19] |
| **Bedrock prompt routers** | ✓full | —not available | —not available | —not available |
| **Bedrock Prompt caching & reasoning** | ✓full | ◐partial [^6] | ✓full | ◐partial [^9] |
| **Runs in your AWS account** | ✓full | ✓full | ✓full | —not available |
| **Model auto-discovery** | ✓full | ◐partial [^1] | ◐partial [^7] | ✓full |
| **Deprecated model failover** | ✓full | —not available | —not available | —not available |
| **Cost & usage tracking** | ✓full | ✓full | —not available | —not available |
| **Per-end-user cost on the AWS bill** | ✓full | —not available [^25] | ◐partial [^26] | ◐partial [^27] |
| **Ready-to-use deployment** | ✓full | —not available | ◐partial [^3] | ✓full |
| **Commercial support** | ✓full | ✓full | —not available | ✓full [^4] |
| **Self-hosted** | ✓full | ✓full | ✓full | —not available |
| **AWS-native focus** | ✓full | ◐partial [^5] | ✓full | ✓full |
| **Multi-provider support** | —not available | ✓full [^11] | —not available | —not available |
| **Source license** | AGPL-3.0 (community) · commercial | MIT | MIT-0 | AWS service |
| **Distribution & supply chain** | AWS Marketplace · GHCR | pip/PyPI | GitHub (MIT-0) | AWS-managed |
✓ full ◐ partial / manual setup — not available
!!! info "About the alternatives"
- **LiteLLM** — widely adopted multi-cloud proxy with a large open-source community. Ideal when you need a single entry point across OpenAI, Azure, AWS, and others. AWS deployment and security features (WAF, VPC endpoints) require manual setup. Also offers a commercial Enterprise tier.
- **Bedrock Access Gateway** — official open-source AWS sample (MIT-0). Covers chat completions and embeddings only. No WAF, auto-scaling, monitoring, or commercial support included.
- **Bedrock Mantle** — AWS's own native OpenAI-compatible endpoint backed by AWS's full compliance and SLA. No self-hosting required. Supports the Chat Completions, Responses, and Anthropic Messages APIs — but each model is restricted to specific APIs (frontier GPT = Responses only, Claude = Messages only, open-weight = Chat Completions), and it covers only a subset of Bedrock models (see the [^2] markers in the comparison table above for exact coverage). Guardrails, prompt routers, and cross-region inference profiles are not available on Mantle. Routes through an AWS-managed endpoint, not your private VPC. See [model availability](https://docs.aws.amazon.com/bedrock/latest/userguide/models-endpoint-availability.html). Rather than an either/or choice, stdapi.ai can serve the Mantle catalog itself: enable [Bedrock Mantle Models](#bedrock-mantle-models) to access OpenAI GPT, xAI Grok, and other Mantle-only models through the gateway, with all four text APIs available for every model.
---
## Ready to Get Started?
- :material-aws: [**Start 14-Day Free Trial**](https://aws.amazon.com/marketplace/pp/prodview-su2dajk5zawpo) — Production-ready Terraform deployment on AWS Marketplace
- :material-rocket-launch: [**Getting Started Guide**](operations_getting_started.md) — Deploy to AWS with Terraform
- :material-docker: [**Run Locally**](operations_getting_started_local.md) — Free Docker image for development
- :material-book-open-variant: [**API Reference**](api_overview.md) — Full endpoint documentation and examples
[^1]: Full Bedrock catalog supported; each model must be declared in config (applies to auto-discovery)
[^2]: Subset of Bedrock models — mostly newer open-weight models (DeepSeek, Gemma, Qwen, GLM, MiniMax, Kimi, newer Mistral) plus OpenAI GPT/gpt-oss and xAI Grok; newer Anthropic Claude generations are available via the Messages API only. Amazon Nova, Meta Llama, AI21, Cohere, and Stability AI (images) are not available — see [AWS endpoint availability](https://docs.aws.amazon.com/bedrock/latest/userguide/models-endpoint-availability.html)
[^3]: CloudFormation reference sample (API Gateway + Lambda, or ALB + Fargate) — no WAF, auto-scaling, monitoring, or commercial support
[^4]: Covered through your existing AWS Support plan
[^5]: Generalist multi-cloud proxy covering 100+ providers; AWS-specific Bedrock features and security integrations may lag behind dedicated solutions
[^6]: Prompt caching and reasoning supported on standard routes; coverage varies by model — not all Bedrock models support prompt caching or extended thinking
[^7]: Auto-discovery limited to the single deployed region — some models are only available in specific AWS regions
[^8]: Achievable via the LiteLLM router, but requires manually declaring each model per region with explicit TPM/RPM limits — no automatic quota distribution
[^9]: Prompt caching on Mantle is supported but model-dependent (per each model card); reasoning is served via frontier models (GPT-5.x, gpt-oss) and select open-weight models (Qwen3 thinking, etc.)
[^10]: Single-region deployment — some models are only available in specific AWS regions; no cross-region catalog aggregation
[^11]: 100+ providers: OpenAI, Azure OpenAI, GCP Vertex, Anthropic direct, and more — ideal when you need a single gateway across multiple clouds
[^12]: Requires connecting Amazon Polly as the TTS backend — not included by default in a LiteLLM on AWS deployment
[^13]: Supported by specifying the cross-region inference profile ARN as the model ID — no automatic profile selection
[^14]: Anthropic Messages API supported on Mantle for newer Claude generations only — Claude 3.x and some 4.x are not available; structured outputs (`output_config.format`) are rejected with a 400 on Mantle — see [AWS endpoint availability](https://docs.aws.amazon.com/bedrock/latest/userguide/models-endpoint-availability.html)
[^15]: LiteLLM ships a [first-class `bedrock_mantle/` provider](https://docs.litellm.ai/docs/providers/bedrock_mantle) covering all Mantle models, so reaching Mantle needs no custom endpoint. Bedrock and Mantle remain two separate providers, each model declared in `model_list`, rather than one auto-discovered catalog
[^16]: LiteLLM translates between API formats per provider adapter; coverage is not a guaranteed all-text-APIs-on-every-model surface
[^17]: Conversation state kept in LiteLLM's own datastore rather than native provider server-side storage
[^18]: Served via LiteLLM's `/responses` endpoint, auto-bridged to `/chat/completions`; no native Bedrock Responses passthrough
[^19]: Mantle uses Projects/Workspaces (tag-based cost allocation via Cost Explorer) for workload isolation instead of application inference profiles — see [Amazon Bedrock Projects](https://docs.aws.amazon.com/bedrock/latest/userguide/projects.html)
[^20]: LiteLLM provides an [MCP gateway](https://docs.litellm.ai/docs/mcp/) that proxies configured third-party MCP servers and converts OpenAPI specs to tools — a related capability, but it does not expose LiteLLM's own AI and media endpoints as tools
[^21]: Amazon Bedrock is a [first-class batch provider](https://docs.litellm.ai/docs/providers/bedrock_batches) for LiteLLM's Batches API — on the proxy only, not the SDK, and backed by [Managed Files](https://docs.litellm.ai/docs/proxy/managed_batches) with a Postgres database plus an operator-configured S3 bucket and Bedrock batch role. LiteLLM's own cost tracking for batches is an Enterprise feature
[^22]: The Bedrock Mantle endpoint [serves](https://docs.aws.amazon.com/bedrock/latest/userguide/endpoints.html) the Responses, Chat Completions and Anthropic Messages APIs only. Amazon Bedrock does offer an [OpenAI-compatible Batch API](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-openai-batch.html) on its `bedrock-runtime` endpoint — for OpenAI models only, reading its input from an S3 URI you upload yourself rather than through a Files API
[^23]: With Amazon Bedrock it is a [bridge to a knowledge base you registered yourself](https://docs.litellm.ai/docs/providers/bedrock_vector_store): search, retrieve, list, update and delete are forwarded, while OpenAI-shaped store creation and [file attachment](https://docs.litellm.ai/docs/vector_store_files) accept the `openai` provider only. LiteLLM's own [`/v1/rag/ingest`](https://docs.litellm.ai/docs/rag_ingest) route does chunk, embed and index into a Bedrock knowledge base, but it is not part of the OpenAI API surface an unmodified client speaks
[^24]: [Emulated](https://docs.litellm.ai/docs/tutorials/file_search_responses_api): the store is presented to the model as a function tool and the search is executed by the proxy rather than by the model's own server. LiteLLM documents that `file_citation` annotations can come back empty when the vector backend does not return the file metadata they need
[^25]: Spend is attributed per key and per user in [LiteLLM's own database](https://docs.litellm.ai/docs/proxy/cost_tracking), not in AWS's billing data. Bedrock `requestMetadata` can be forwarded, which AWS surfaces in [invocation logs rather than Cost Explorer or CUR 2.0](https://docs.aws.amazon.com/bedrock/latest/userguide/cost-mgmt-request-metadata.html)
[^26]: Application inference profiles put billed dollars in [Cost Explorer and CUR 2.0](https://docs.aws.amazon.com/bedrock/latest/userguide/cost-management.html), but per application rather than per end user — the gateway does not map a caller to a profile
[^27]: [IAM principal attribution, Projects and Workspaces](https://docs.aws.amazon.com/bedrock/latest/userguide/cost-management.html) put billed dollars in Cost Explorer and CUR 2.0 — per identity or per project rather than per end user, and behind a shared gateway every caller arrives as the same identity
[^28]: [`/images/edits` lists AWS Bedrock (Stability) among its providers](https://docs.litellm.ai/docs/image_edits), so Stability's editing operations are reachable on Bedrock. It does not extend to the other Bedrock image models: the [Bedrock provider page](https://docs.litellm.ai/docs/providers/bedrock) lists `/images/generations` but not `/images/edits` among its supported OpenAI endpoints, and Amazon Nova Canvas and Titan are documented for [generation](https://docs.litellm.ai/docs/providers/bedrock_image_gen) only
---
## Models
Source: https://stdapi.ai/md/models.md
# :material-view-list: Models
Every model stdapi.ai serves on AWS — Amazon Bedrock, Bedrock Mantle, Polly,
Transcribe and Comprehend — what it does, where it runs,
what AWS charges for it, and how it scores on public leaderboards.
All of them answer on **one endpoint under one API key**, through the
[OpenAI, Anthropic and Cohere APIs](features.md) your tools already speak.
:material-map-marker-radius: **“Available in EU” has two answers.** *Where I can
call it* is the AWS regions you can reach the model from. *Where it runs* is
where inference executes — a model reachable from Frankfurt through a **global**
inference profile is not a model running in Frankfurt. For sovereignty, read the
second.
:material-cash: **The prices are AWS's.** Published AWS rates in USD per billed
unit. You pay AWS directly on your own account; stdapi.ai adds no margin and
sends no invoice. The authority is the
[Amazon Bedrock pricing page](https://aws.amazon.com/bedrock/pricing/).
:material-check-decagram: **A blank score means blank.** A benchmark published
against the wrong model is worse than none, so an entry is attached only when
the evidence is unambiguous. Open a model to see which leaderboard entry each
score came from.
:material-history: **Legacy models stay listed.** Models AWS has marked legacy
are tagged and kept, so one you already run stays findable. Availability
reflects the AWS account this snapshot came from — your own
[`search_models`](api_search_models.md) is the authority for what *you* can call.
---
## :material-database-outline: Sources, licences and caveats
Snapshot taken on **2026-08-26** from a stdapi.ai instance, covering 138 models across 33 AWS regions. Prices and availability move — before you commit to a number, confirm it against the [Amazon Bedrock pricing page](https://aws.amazon.com/bedrock/pricing/) and your own [`search_models`](api_search_models.md).
Every number on this page comes from one of the sources below, reproduced unmodified. Mapping each entry onto an Amazon Bedrock model ID is our own work, and any error in that mapping is ours, not the source's.
| Source | Licence | Read on | Used here |
| --- | --- | --- | --- |
| The gateway's own [`search_models`](api_search_models.md) and [`model_pricing`](api_model_pricing.md) | — | 2026-08-26 | 138 |
| [Amazon Bedrock `ListFoundationModels`](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_ListFoundationModels.html), read raw so its undocumented fields survive | — | 2026-08-26 | capabilities, APIs, media types, limits |
| [LMArena Leaderboard](https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | 2026-08-24 | 79 of 793 entries |
| [MTEB — Massive Text Embedding Benchmark](https://github.com/embeddings-benchmark/results) | [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/) | 2026-08-26 | 5 of 12 entries |
| [Epoch AI — AI Benchmarking Hub](https://epoch.ai/benchmarks/use-this-data) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | 2026-08-18 | 59 of 655 entries |
| [Amazon Bedrock model cards](https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html) | [AWS documentation](https://aws.amazon.com/terms/) | 2026-08-26 | 112 of 124 model cards |
| [models.dev](https://models.dev/) | [MIT](https://github.com/anomalyco/models.dev/blob/dev/LICENSE) | 2026-08-26 | 65 of 120 models |
| [Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard) | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | 2026-08-26 | 2 of 65 entries |
- **LMArena Leaderboard** — Arena Elo ratings by LMArena (Arena Intelligence Inc.), reproduced unmodified under CC BY 4.0. The mapping to Amazon Bedrock model IDs is ours.
- **MTEB — Massive Text Embedding Benchmark** — Benchmark results from the MTEB results repository, dedicated to the public domain under CC0 1.0. The mapping to Amazon Bedrock model IDs is ours.
- **Epoch AI — AI Benchmarking Hub** — Benchmark results by Epoch AI, reproduced unmodified under CC BY 4.0. Rows derived from the Aider Polyglot and Terminal-Bench leaderboards keep their Apache-2.0 licence. The mapping to Amazon Bedrock model IDs is ours.
- **Amazon Bedrock model cards** — Context windows, output limits, knowledge cutoffs and lifecycle dates as stated on each model's Amazon Bedrock model card. Only those facts are taken; AWS's own descriptive copy is not reproduced.
- **models.dev** — Context windows, knowledge cutoffs and capability flags from models.dev, an open database of AI models, used under the MIT licence. Its Amazon Bedrock entries are keyed by Bedrock model ID, so the join is exact.
- **Open ASR Leaderboard** — Word error rates from the Open ASR Leaderboard, reproduced unmodified under Apache-2.0. The mapping to Amazon Bedrock model IDs is ours.
Some of what the table shows comes from parts of `ListFoundationModels` AWS does not document, so AWS may stop returning them at any time. A regeneration updates the published data set rather than replacing it: the last known value of such a field is kept, and a model AWS stops listing stays in the table, tagged `delisted`, with the date it was last seen.
The model names and brand logos in the table above are trademarks of their respective owners: AI21 Labs, Jamba; AWS, Amazon and Amazon product names and logos; Alibaba Cloud, Qwen; Anthropic, Claude; Cohere; DeepSeek; Google, Gemini, Gemma; Luma AI; Meta, Llama; MiniMax; Mistral AI; Moonshot AI, Kimi; NVIDIA, Nemotron; OpenAI, ChatGPT, GPT, Codex; Stability AI, Stable Diffusion; TwelveLabs, Marengo, Pegasus; Writer, Palmyra; Z.ai, Zhipu AI, GLM; xAI, Grok.
---
## Contact
Source: https://stdapi.ai/md/contact.md
# Contact
stdapi.ai is built and supported by **J. Goutin** ([jgoutin.dev](https://jgoutin.dev)). Questions go straight to product engineering, so expect a technical answer rather than a discovery call. Feel free to get in touch to discuss your needs.
!!! tip offer "Buy through an AWS Marketplace private offer"
A private offer is how an organization buys stdapi.ai: **custom terms and duration**, **committed usage**, and a **preferential rate of $0.09/container-hour** instead of $0.10. It is procured through your existing AWS relationship — no new vendor onboarding, no new supplier record, billed on your existing AWS invoice. Marketplace spend may count toward an EDP or Private Pricing Agreement; check your agreement to confirm whether yours does.
[Request your private offer](mailto:private@offers.stdapi.ai?subject=Private%20offer%20request%20-%20stdapi.ai&body=Company%3A%20%0AAWS%20account%20ID%3A%20%0AUse%20case%3A%20%0ADesired%20start%20date%3A%20){ .md-button .md-button--primary }
[private@offers.stdapi.ai](mailto:private@offers.stdapi.ai)
**How it works:**
1. **Email us** — use the pre-filled link above, including your company name, AWS account ID (required by AWS to create the offer), use case, and desired start date.
2. **Offer created** — typically ready in minutes.
3. **Accept it** in AWS Marketplace.
4. **Pay $0.09/container-hour** — billed automatically, same pay-per-use model as the public listing.
**Which AWS account ID?** An offer is extended to one specific account, not to an organization. Give the account ID that will hold the subscription — usually the linked account that runs the deployment, unless your organization subscribes centrally from the management (payer) account. Under consolidated billing the charges land on the payer's invoice either way. If you are unsure, tell us your structure in the email and we'll confirm before creating the offer.
**Already on the 14-day free trial?** The trial belongs to the public listing, and accepting a private offer moves you onto the new agreement. To use the trial in full, tell us the date it ends as your desired start date and we'll set the offer to begin then.
Your request is handled by an automated AI agent — it creates the offer and replies in your language, typically within minutes. A human follows every conversation and steps in whenever needed.
!!! info "Not ready to talk yet?"
- :material-scale-balance: [**Licensing & pricing**](operations_licensing.md) — Community (AGPL-3.0) vs commercial, and what each includes
- :material-cash-multiple: [**Cost management**](operations_cost_management.md) — What a deployment actually costs, tier by tier
- :material-docker: [**Run the free Community edition**](operations_getting_started_local.md) — Same endpoints, $0, one `docker run` command
---
!!! info email "Support"
[support@stdapi.ai](mailto:support@stdapi.ai)
For help, bug reports, or technical questions about stdapi.ai.
!!! info email "Business & general inquiries"
[contact@stdapi.ai](mailto:contact@stdapi.ai)
For sales, partnerships, or any other questions.
!!! abstract availability "Availability"
Replies typically within 1 business day (Europe/Paris timezone)
---
For business information (legal registration, address, VAT), see [jgoutin.dev](https://jgoutin.dev).
---
## Overview
Source: https://stdapi.ai/md/getting_started.md
# :material-rocket-launch: Get Started
Pick the path that fits where you are right now. Both use the same OpenAI, Anthropic, and Cohere-compatible API, and both are adopted the same way: point your client at the deployment's base URL, give it that deployment's API key, and change the model name only where it differs. Usually it does not — each model is published under the name its provider uses, so a client already asking for `claude-opus-5` or `gpt-5.6-sol` changes nothing but the base URL. Where a name *does* differ, [`MODEL_ALIASES`](operations_configuration.md#model-aliases) maps the name your application already sends onto a model the deployment serves. What the base URL buys is the catalogue behind it: it spans every provider your regions offer, not one vendor's list. [`GET /search_models`](api_search_models.md) lists it. The [Models](models.md) page shows the same catalogue now, before you deploy.
Looking for reference documentation rather than a quickstart? See [Features](features.md) for what the gateway does, and the [API Overview](api_overview.md) for endpoints, parameters, and SDK usage.
!!! tip trial "Start here: 14-day free trial on AWS"
Deploy the production-ready stack today — the stdapi.ai license is free for 14 days. AWS charges for what it runs (ALB, Fargate, KMS, NAT) and for Bedrock usage from the first minute. After the trial, the license is $0.10/container-hour — cancel anytime.
## :material-clipboard-check-outline: Before You Start
Both paths need:
- **An AWS account** with access to [Amazon Bedrock](https://aws.amazon.com/bedrock/). [Create one free](https://aws.amazon.com/free/) if you don't have one.
- **AWS credentials configured locally** — `aws configure` or `aws sso login` ([AWS CLI setup guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html)).
The AWS Terraform path additionally needs:
- **Administrator-level AWS permissions** — the module provisions IAM roles and policies, KMS keys, ECS/Fargate, ALB, and networking. A restricted developer profile will fail.
- **A sandbox AWS account is strongly recommended** for evaluation. Replicate into your target account once you've validated the stack.
- [Terraform](https://www.terraform.io/downloads) or [OpenTofu](https://opentofu.org/docs/intro/install/) >= 1.5.
- An **[AWS Marketplace subscription](https://aws.amazon.com/marketplace/pp/prodview-su2dajk5zawpo)** (14-day free trial of the license).
The Docker path additionally needs [Docker](https://docs.docker.com/get-started/get-docker/) or [Podman](https://podman.io/docs/installation).
---
## :material-directions-fork: Pick Your Path
- :material-aws:{ .lg .middle } __14-day free trial — Deploy on AWS with Terraform__
---
**Start here — recommended for production.**
- 2 Terraform commands → ECS Fargate with HTTPS, auto-scaling, optional WAF
- Hardened container image from **AWS Marketplace** — license free for 14 days, then $0.10/container-hour
- IP-restricted by default — safe to test right away
- Multi-region variants available (EU / US) for data residency
[:octicons-arrow-right-24: Deploy on AWS](operations_getting_started.md){ .md-button .md-button--primary }
- :material-docker:{ .lg .middle } __One command — Try locally with Docker__
---
**Lighter option, for:** local development, evaluation, open-source projects.
- One `docker run` command — no AWS infrastructure to provision
- Uses your local AWS credentials (mounted from `~/.aws`) to reach Bedrock
- No authentication or IP restriction by default — intended for local use only
- Free **community image** (AGPL-3.0)
- Full API compatibility — same endpoints as production
[:octicons-arrow-right-24: Run Locally with Docker](operations_getting_started_local.md){ .md-button }
---
## :material-help-circle-outline: Not Sure Which to Pick?
| You are... | Go with | Why |
|---|---|---|
| A developer evaluating for the first time | **Docker (local)** | Fastest feedback loop, no cloud resources to tear down. |
| A team lead validating for production use | **AWS (Terraform)** | Mirrors real deployment. 14-day free license trial during evaluation. |
| Already running a production workload elsewhere | **AWS (Terraform)** | Jump straight to the stack you'll operate long-term. |
| Contributing to an open-source project | **Docker (local)** | AGPL-3.0 community image is free to use and redistribute. |
You can start local and migrate to AWS later — same API, same SDKs, same client-side changes. Point the base URL at your AWS endpoint, add that deployment's API key, and check your model names against what its regions serve.
---
## :material-arrow-right: After Your First Call
- :material-star-four-points-outline: [**Features**](features.md) — What the gateway supports, substantiated claim by claim
- :material-book-open-variant: [**API Overview**](api_overview.md) — Endpoints, parameters, and SDK usage
- :material-magnify: [**Search Models**](api_search_models.md) — Find the right model ID by modality, route, region, or streaming support
- :material-view-list: [**Models**](models.md) — Browse every model with modalities, regions, AWS prices and leaderboard scores
- :material-puzzle: [**Use Cases**](use_cases.md) — Open WebUI, n8n, coding assistants, and more
- :material-cog: [**Configuration**](operations_configuration.md) — Every environment variable and option
- :material-wrench: [**Troubleshooting**](operations_troubleshooting.md) — Common first-deployment errors and fixes
- :material-server-network: [**Advanced Deployment**](operations_deploy_advanced.md) — VPC integration, multi-region, cost optimization
- :material-scale-balance: [**Licensing**](operations_licensing.md) — Community (AGPL) vs Commercial
---
## Deploy on AWS
Source: https://stdapi.ai/md/operations_getting_started.md
# :material-rocket-launch: Deploy stdapi.ai on AWS
Get a production-grade AI gateway running on AWS with two Terraform commands, speaking the OpenAI, Anthropic, and Cohere APIs. Terraform provisions the stack — ECS Fargate, HTTPS, auto-scaling, optional WAF, and optional monitoring. You bring the Marketplace subscription, AWS credentials, and — for a custom domain — the domain and its certificate.
!!! tip trial "14-Day Free Trial"
The AWS Marketplace subscription includes a **14-day free trial of the stdapi.ai license**. AWS charges for the infrastructure it deploys (ALB, Fargate, KMS, NAT) and for Bedrock usage apply from the first minute — see [Deployment Cost](#deployment-cost).
!!! info "Need help?"
For questions, issue reports, or assistance, see the [Contact](contact.md) page.
---
## :material-rocket-launch: Quick Start
### Prerequisites
1. **Subscribe on AWS Marketplace** — this is the action that starts your 14-day free trial:
[Subscribe on AWS Marketplace — starts your 14-day free trial](https://aws.amazon.com/marketplace/pp/prodview-su2dajk5zawpo){ .md-button .md-button--primary }
2. Install [Terraform](https://www.terraform.io/downloads) or [OpenTofu](https://opentofu.org/docs/intro/install/) >= 1.5.
3. Configure [AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) (`aws configure` or `aws sso login`).
AWS infrastructure cost depends entirely on your configuration — from a single scheduled Spot container with no load balancer to a full multi-AZ stack — so there's no one figure to quote; see [Deployment Cost](#deployment-cost) below. stdapi.ai's own license runs $0.10/container-hour ($0.09 via private offer); Bedrock usage is billed separately by AWS at cost.
!!! warning "Requires AWS administrator permissions"
The Terraform module provisions IAM roles and policies, KMS keys, ECS/Fargate, ALB, and networking. A restricted developer profile will fail during `terraform apply`.
**Strongly recommended:** deploy into a **sandbox / non-production AWS account first** to evaluate the stack, then replicate into your target account with scoped-down principals once you've validated it.
!!! tip "Confirm your AWS identity and region before deploying"
The AWS provider uses the region and profile from your environment — not a Terraform variable. Check both before running `terraform apply`:
```bash
aws sts get-caller-identity
aws configure get region
```
### Deploy
```bash
git clone https://github.com/stdapi-ai/samples.git
cd samples/getting_started_production/terraform
terraform init
terraform apply
```
??? info "No git? Download the ZIP"
```bash
curl -L https://github.com/stdapi-ai/samples/archive/refs/heads/main.zip -o samples.zip
unzip samples.zip
cd samples-main/getting_started_production/terraform
terraform init
terraform apply
```
That's it. Two Terraform commands, and you have:
- Production-grade ECS Fargate deployment with HTTPS
- Regional S3 buckets
- Auto-scaling and API key authentication
- Interactive API documentation at `/docs`
- IP-restricted access (your IP only)
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
openai[" OpenAI SDK"] -->|HTTPS| alb[" ALB"]
anthropic[" Anthropic SDK"] -->|HTTPS| alb
alb --> ecs[" stdapi.ai ECS Fargate"]
ecs --> bedrock[" Amazon Bedrock"]
ecs --> polly[" Amazon Polly"]
ecs --> transcribe[" Amazon Transcribe"]
ecs --> s3[" Amazon S3"]
ecs --> cloudwatch[" CloudWatch"]
```
### Get Your Credentials
```bash
terraform output -raw api_key
terraform output api_endpoint
terraform output docs_url
```
!!! tip "Ready-to-use Terraform example on GitHub"
:material-map-marker: **Single region** — [getting_started_production](https://github.com/stdapi-ai/samples/tree/main/getting_started_production)
!!! tip "Optional: expose the API as MCP tools"
The [MCP server](features.md#mcp-model-context-protocol) is off by default. Set `enable_mcp_streamable_http = true` on the Terraform module and every endpoint becomes a named MCP tool at `/mcp`, callable directly by Claude Code, LangGraph, or any MCP client.
Every exposed tool adds its schema to each MCP client's context window, so expose only the tools your agents actually use — for example `mcp_include_tools = "openai_chat_completion,openai_embedding,search_models"`. See the [MCP configuration reference](operations_configuration.md#summary-mcp).
!!! tip offer "Buying for an organization? Use an AWS Marketplace private offer"
Custom terms and duration, committed usage, and a preferential rate of **$0.09/container-hour** instead of $0.10. Procured through your existing AWS relationship — no new vendor onboarding, billed on your existing AWS invoice. Want to try first? Use the 14-day free trial of the public listing, then accept the private offer.
[:material-email-outline: Request a Private Offer](contact.md#private-offer)
---
## :material-check-circle: Make Your First API Call
**Want to explore the API without writing code?** Use the `docs_url` from `terraform output docs_url` to open the interactive Swagger documentation in your browser — you can browse all available endpoints and make live API calls directly from the page, no code required.
!!! info "If the docs page returns 503 or shows a TLS warning"
These are normal on a fresh deployment. The ECS service takes 2–3 minutes to pass health checks (→ 503), and the auto-generated `*.elb.amazonaws.com` domain has no trusted TLS certificate (→ browser warning; safe to bypass for testing). See [Troubleshooting](operations_troubleshooting.md) for a permanent HTTPS setup with a custom domain.
stdapi.ai is compatible with both OpenAI and Anthropic SDKs. If you've used either before, you already know how to use it — the base URL changes, along with the API key, and the model name only where it differs, since Anthropic's and OpenAI's own names for the models Bedrock serves resolve here as they stand. Here are the raw HTTP calls with `curl` so you can verify the endpoint from any shell:
=== "OpenAI-compatible"
```bash
API_ENDPOINT=$(terraform output -raw api_endpoint)
API_KEY=$(terraform output -raw api_key)
curl "$API_ENDPOINT/v1/chat/completions" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"messages": [{"role": "user", "content": "Hello! Tell me a joke."}]
}'
```
=== "Anthropic-compatible"
```bash
API_ENDPOINT=$(terraform output -raw api_endpoint)
API_KEY=$(terraform output -raw api_key)
curl "$API_ENDPOINT/anthropic/v1/messages" \
-H "x-api-key: $API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"max_tokens": 1000,
"messages": [{"role": "user", "content": "Hello! Tell me a joke."}]
}'
```
**Using the official SDKs?** Point the `base_url` (Python) / `baseURL` (Node.js) option at `$API_ENDPOINT/v1` (OpenAI SDK) or `$API_ENDPOINT/anthropic` (Anthropic SDK), and set the `model` field to a model from the catalog below. The rest of your existing code is unchanged. The [API Overview](api_overview.md) has SDK snippets for Python, Node.js, and more.
!!! tip "Discover the full model catalog"
Once your first call succeeds, switch the `model` field to any other Bedrock model — `anthropic.claude-fable-5`, `anthropic.claude-sonnet-5`, `qwen.qwen3-coder-next`, and more.
- **Browse before you switch:** the [Models](models.md) page lists everything with prices and scores
- **Browse active models (recommended):** `GET /search_models` — returns every discovered non-legacy model with full details (provider, modalities, supported routes, regions, streaming/legacy status). Add `legacy=true` to look up a deprecated model instead. Or open the interactive Swagger docs.
- **Find a model by capability:** the same endpoint filters by modality, route, region, streaming, Batch API support, or legacy status — e.g. `GET /search_models?input_modalities=IMAGE&route=/v1/chat/completions` returns only vision-capable chat models. This is also the recommended way for AI agents to discover the right model ID before calling another endpoint. See the [Search Models API](api_search_models.md) reference.
- **OpenAI SDK compatibility:** `GET /v1/models` is also available with the standard OpenAI listing format (lighter payload, no capability metadata) for tools that require the exact OpenAI schema.
**Using your own Terraform config instead of the sample?** The sample above enables authentication for you (`api_key_create = true`, retrieved above). Writing your own module config from scratch? stdapi.ai runs without authentication unless you set `api_key_create = true` — see [Authentication & Security](operations_authentication_security.md) for all options.
**Verify the deployment is healthy:**
```bash
curl $API_ENDPOINT/health
# → {"status": "ok"}
```
The `/health` endpoint requires no authentication and is used by the ALB health check.
---
## :material-wrench: Troubleshooting
The `503` and TLS-warning hiccups on first deployment are already covered above — see [Make Your First API Call](#make-your-first-api-call).
:material-arrow-right: **Full troubleshooting guide:** [Troubleshooting](operations_troubleshooting.md) — 401 auth errors, 404 model not found, ThrottlingException, S3 errors, VPC connectivity, Terraform IAM failures, and more.
!!! info "Prefer a hands-off setup?"
A [managed deployment service](https://aws.amazon.com/marketplace/pp/prodview-xknxzjgl7zi5s) is available if you'd rather not manage Terraform yourself. Choose between guided assistance (step-by-step support while you retain full control) or fully managed setup (handled on your behalf, inside your AWS account). Response time is 1 business day during the engagement.
---
## :material-currency-usd: Deployment Cost
AWS infrastructure cost is driven by configuration, not a fixed default: task count (one per Availability Zone unless overridden), Fargate Spot vs. on-demand, scheduled service hours, and whether an ALB is provisioned at all. A minimal deployment — one scheduled Fargate Spot task, no ALB, reached via Service Discovery — and a full multi-AZ production stack — ALB + WAF, one task per AZ, running 24/7 — sit at opposite ends of a wide range.
stdapi.ai's own license is billed separately at **$0.10/container-hour** ($0.09 via [private offer](contact.md#private-offer)); Bedrock and other AI service usage is billed by AWS at cost, with no stdapi.ai markup.
See [Cost-Optimized Deployment](operations_deploy_advanced.md#cost-optimized-deployment) for the Spot/scheduling configuration, and [Cost Management → Gateway Cost](operations_cost_management.md#gateway-cost) for the full tier-by-tier breakdown.
---
## :material-delete-outline: Cleaning Up
When you're done testing, tear down the stack to stop incurring AWS and license charges:
```bash
terraform destroy
```
Running from the ZIP download instead of `git clone`? Run the same command from the `samples-main/getting_started_production/terraform` directory.
---
## :material-arrow-right: Next Steps
- :material-book-open-variant: [**API Overview**](api_overview.md) — Endpoints, parameters, and usage examples
- :material-cog: [**Configuration**](operations_configuration.md) — All environment variables and options
- :material-server-network: [**Advanced Deployment**](operations_deploy_advanced.md) — VPC integration, multi-region, cost optimization, manual ECS
- :material-cash-multiple: [**Cost Management**](operations_cost_management.md) — Infrastructure, license, and AI usage cost breakdown
- :material-directions-fork: [**Resilience & Failover**](operations_resilience.md) — Multi-region routing, and the quota each enabled region adds
- :material-shield-lock: [**Data Sovereignty & Compliance**](operations_compliance.md) — GDPR-compliant region configuration
- :material-puzzle: [**Use Cases**](use_cases.md) — Open WebUI, n8n, coding assistants, and more
- :material-wrench: [**Troubleshooting**](operations_troubleshooting.md) — Common first-deployment errors and fixes
- :material-scale-balance: [**Licensing**](operations_licensing.md) — AGPL vs commercial options
---
## Run Locally with Docker
Source: https://stdapi.ai/md/operations_getting_started_local.md
# :material-docker: Local Development with Docker/Podman
Run stdapi.ai locally for development, testing, and evaluation using the free community container image (AGPL-3.0). Full API compatibility — the same endpoints and features as the production deployment.
!!! tip "New to Amazon Bedrock?"
To run stdapi.ai locally you need:
1. **[Docker](https://docs.docker.com/get-started/get-docker/) or [Podman](https://podman.io/docs/installation)** installed
2. An **AWS account** — [create one free](https://aws.amazon.com/free/)
3. **AWS credentials** configured locally — via `aws configure` or `aws sso login` ([AWS CLI setup guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html))
---
## :material-docker: Run It
**With AWS credentials (after `aws sso login`):**
```bash
docker run --rm -p 8000:8000 \
-v ~/.aws:/home/nonroot/.aws:ro \
-e AWS_BEDROCK_REGIONS=us-east-1,us-west-2 \
-e ENABLE_DOCS=true \
ghcr.io/stdapi-ai/stdapi.ai-community:latest
```
**With environment variables instead:**
```bash
docker run --rm -p 8000:8000 \
-e AWS_ACCESS_KEY_ID=your-access-key-id \
-e AWS_SECRET_ACCESS_KEY=your-secret-access-key \
-e AWS_SESSION_TOKEN=your-session-token \
-e AWS_BEDROCK_REGIONS=us-east-1,us-west-2 \
-e ENABLE_DOCS=true \
ghcr.io/stdapi-ai/stdapi.ai-community:latest
```
??? info "Podman on Fedora/RHEL with SELinux"
Add the `:z` SELinux label and `--userns=keep-id`:
```bash
podman run --rm -p 8000:8000 \
--userns=keep-id \
-v ~/.aws:/home/nonroot/.aws:ro,z \
-e AWS_BEDROCK_REGIONS=us-east-1,us-west-2 \
-e ENABLE_DOCS=true \
ghcr.io/stdapi-ai/stdapi.ai-community:latest
```
The `:z` flag relabels files for container access. Use `:Z` if multiple containers share the volume. `--userns=keep-id` maps your host user ID to the container user.
See also [Troubleshooting → Podman volume mount fails on Fedora/RHEL with SELinux](operations_troubleshooting.md#terraform-deployment) if you hit this after the fact.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
openai[" OpenAI SDK"] --> local[" stdapi.ai (community) Docker/Podman"]
anthropic[" Anthropic SDK"] --> local
local --> bedrock[" Amazon Bedrock"]
local --> polly[" Amazon Polly"]
local --> transcribe[" Amazon Transcribe"]
local --> s3[" Amazon S3"]
```
---
## :material-check-circle: Test It
```bash
# Check health
curl http://localhost:8000/health
# List all available models
curl http://localhost:8000/search_models
# Search models by capability (e.g. streaming-capable chat models only)
curl "http://localhost:8000/search_models?route=/v1/chat/completions&streaming=true"
# Chat completion
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
**Interactive API docs:** Open [http://localhost:8000/docs](http://localhost:8000/docs) for Swagger UI with all available endpoints.
**Point an application at it:** set the `base_url` (Python) / `baseURL` (Node.js) option to `http://localhost:8000/v1` (OpenAI SDK) or `http://localhost:8000/anthropic` (Anthropic SDK). Often that is the whole change: each model is published under the name its provider uses, so a `model` field already reading `claude-sonnet-5` or `gpt-oss-120b` resolves as it stands. Change it where the name differs — one your application hard-codes for a model this deployment does not serve, such as `gpt-4o` or `dall-e-3`, returns `404` here until you point it at a model from the catalog above or map it with [`MODEL_ALIASES`](operations_configuration.md#model-aliases), while every model your regions serve is one name away. No API key is required by default — pass any non-empty string if your client insists on one. The [API Overview](api_overview.md) has SDK snippets for Python, Node.js, and more.
!!! tip "Try other models"
`amazon.nova-micro-v1:0` is a fast, low-cost model — great for confirming the pipeline works. Once you see a response, switch the `model` field to `anthropic.claude-fable-5`, `anthropic.claude-sonnet-5`, or any other Bedrock model available in your configured regions.
Use `GET /search_models` (shown above) to discover what's available and filter by capability, or `GET /v1/models` for strict OpenAI SDK compatibility — see the [Search Models API](api_search_models.md) reference.
!!! tip "Optional: expose the API as MCP tools"
The [MCP server](features.md#mcp-model-context-protocol) is off by default. Add `-e ENABLE_MCP_STREAMABLE_HTTP=true` to the `docker run` command and every endpoint becomes a named MCP tool at `http://localhost:8000/mcp`, callable directly by Claude Code or any MCP client.
Every exposed tool adds its schema to each MCP client's context window, so expose only the tools you actually use — for example `-e MCP_INCLUDE_TOOLS=openai_chat_completion,openai_embedding,search_models`. See the [MCP configuration reference](operations_configuration.md#summary-mcp).
---
## :material-information: Technical Notes
**Building from source:** See the [Dockerfile](https://github.com/stdapi-ai/stdapi.ai/blob/main/Dockerfile) if you prefer to build the image yourself.
**Container runtime:** Both community and Marketplace images use [Granian](https://github.com/emmett-framework/granian), a high-performance Python ASGI server. Granian environment variables (e.g., `GRANIAN_PORT`, `GRANIAN_WORKERS`) are supported.
**Configuration:** See [Configuration Reference](operations_configuration.md) for all environment variables.
!!! tip "Ready for Production?"
When you're ready to deploy to AWS with HTTPS, auto-scaling, and enterprise features, the [production deployment guide](operations_getting_started.md) gets you running with two Terraform commands. The [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-su2dajk5zawpo) subscription includes a **14-day free trial of the license**.
---
## :material-arrow-right: Next Steps
stdapi.ai works with any OpenAI or Anthropic-compatible tool. Here are popular integrations to try locally:
- :material-chat: [**Open WebUI**](use_cases_openwebui.md) — Private ChatGPT-like interface with RAG, multi-modal support, and document upload
- :material-robot: [**n8n Workflows**](use_cases_n8n.md) — AI-powered automation with 400+ integrations
- :material-code-braces: [**AI Coding Assistants**](use_cases_coding_assistants.md) — Claude Code, Cline, OpenCode, Zed with Amazon Bedrock models
- :material-book-open-variant: [**API Overview**](api_overview.md) — All endpoints, parameters, and usage examples
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy the production stack on AWS with Terraform
- :material-wrench: [**Troubleshooting**](operations_troubleshooting.md) — Podman/SELinux, auth, model-not-found, and other common errors
---
## Troubleshooting
Source: https://stdapi.ai/md/operations_troubleshooting.md
# :material-wrench: Troubleshooting
Common issues when deploying stdapi.ai for the first time. If your error isn't listed here, see the [Contact](contact.md) page, or open an issue on [GitHub](https://github.com/stdapi-ai/stdapi.ai/issues).
---
## :material-cloud-upload: Terraform / Deployment
??? failure "`terraform apply` fails with AccessDenied on IAM, KMS, or ECS actions"
Your AWS profile does not have sufficient permissions. The stdapi.ai Terraform module provisions IAM roles, KMS keys, ECS, ALB, Route53 records, an optional WAF, and (for some samples) RDS and ElastiCache.
- Use an **administrator-level** AWS profile for the evaluation deployment.
- **Recommended**: deploy into a sandbox/non-production AWS account first, then replicate into your target account with scoped-down principals once validated.
- Verify your active identity: `aws sts get-caller-identity`.
??? failure "`terraform apply` succeeds but nothing is reachable"
Terraform completed but the ECS service is still coming up. The ALB returns `503 Service Unavailable` until tasks pass health checks.
- Wait 2–3 minutes after `terraform apply` completes.
- Check ECS service status: `aws ecs describe-services --cluster --services `.
- Check task logs in CloudWatch: `/aws/ecs/`.
??? failure "Wrong AWS region or profile used by Terraform"
The AWS provider uses the region/profile from your environment, not a Terraform variable.
- Confirm before applying:
```bash
aws sts get-caller-identity
aws configure get region
```
- Set explicitly with `AWS_PROFILE=... AWS_REGION=... terraform apply` if needed.
??? failure "ElastiCache creation failed — insufficient capacity in AZ (Open WebUI sample)"
The ElastiCache Valkey cache occasionally fails to create when the target availability zone is out of capacity.
```text
Error: waiting for ElastiCache Replication Group ... create: unexpected state 'create-failed',
wanted target 'available'
```
- Remove the failed Valkey cache from the ElastiCache console (disable backups first, then wait for full deletion) and re-run `terraform apply`.
- If the problem persists, change `node_type` in `valkey.tf` (e.g. `cache.t4g.micro` → `cache.t3.micro`) and retry.
??? failure "Podman volume mount fails on Fedora/RHEL with SELinux (local Docker)"
SELinux blocks container access to `~/.aws` without a relabel.
- Add `:z` (or `:Z` for exclusive use) to the volume and `--userns=keep-id`:
```bash
podman run --rm -p 8000:8000 \
--userns=keep-id \
-v ~/.aws:/home/nonroot/.aws:ro,z \
-e AWS_BEDROCK_REGIONS=us-east-1,us-west-2 \
-e ENABLE_DOCS=true \
ghcr.io/stdapi-ai/stdapi.ai-community:latest
```
- See [Local Development](operations_getting_started_local.md#run-it) for the full run command.
---
## :material-api: Runtime / First API call
??? failure "`503 Service Unavailable` — on the /docs page or any endpoint"
The ECS service is still starting up. Health checks take a few minutes.
- Wait 2–3 minutes after deployment and refresh.
- Check the ALB target group health in the AWS console.
- If it persists longer than 5 minutes, inspect CloudWatch logs for the ECS task.
??? failure "`ECONNREFUSED` — connection refused, but only from some clients"
The images bind IPv4 only (`GRANIAN_HOST=0.0.0.0`), so a client that resolves the server to an IPv6 address reaches a port nothing is listening on. Clients disagree about which address to try first, which is why the same deployment looks reachable from one language and dead from another: Node.js prefers the `AAAA` record and fails outright, while most Python clients fall back to the `A` record and hide the problem.
- Typically hit with **ECS service discovery**, which publishes an `AAAA` record for every task in an IPv6-enabled subnet. Deployments fronted by an ALB are unaffected — the load balancer terminates the client connection itself and reaches the task over IPv4.
- Set `GRANIAN_HOST=::` for a dual-stack socket answering both families; see the Container Runtime note in [Configuration](operations_configuration.md). The [Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) sets it when the VPC has IPv6 enabled.
- After switching, extend [`PROXY_TRUSTED_HOSTS`](operations_configuration.md#proxy-trusted-hosts) with the IPv4-mapped form of each range (`::ffff:10.0.0.0/112`) — a dual-stack listener reports IPv4 peers in that form, and an untrusted proxy means `X-Forwarded-For` is ignored and the load balancer's own address is logged as the client IP.
??? failure "Targets never become healthy after setting `TRUSTED_HOSTS`"
Host header validation applies to `/health` as well. A load balancer health check addresses the target directly, so its `Host` header carries the target's IP address — which a list of domain names does not match, and every probe is answered with `400`. The target group stays unhealthy and the ALB keeps returning `503`.
- Prefer host validation at the load balancer: an ALB listener rule on the `Host` header, with `TRUSTED_HOSTS` left unset.
- If the application-level allow-list is required, include the address the health check actually sends.
- The container's own `HEALTHCHECK` is unaffected: it derives its `Host` header from `TRUSTED_HOSTS`. See [`TRUSTED_HOSTS`](operations_configuration.md#trusted-hosts).
??? failure "The ECS task never reports healthy, or restarts in a loop"
ECS ignores the image's own `HEALTHCHECK`, so a task definition that declares no `healthCheck` gets no container-level probe at all, and one that declares a probe carried over from an earlier version runs a command the current image no longer provides. Either way the container is reported unhealthy and the service replaces it.
- Copy the `healthCheck` block from the [ECS task definition example](operations_deploy_advanced.md#ecs-task-definition-example), which declares the image's own probe, and re-copy it when upgrading.
- The [Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) declares it for you.
- Do not substitute a `curl` or `urllib` one-liner: it sends an untrusted `Host` and is answered with `400` as soon as [`TRUSTED_HOSTS`](operations_configuration.md#trusted-hosts) is set.
??? failure "Browser TLS warning on the /docs page"
The ALB uses the default `*.elb.amazonaws.com` domain, which has no trusted certificate. This is expected and safe to bypass for testing.
- For a production-grade certificate, configure a custom domain — the [Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) supports ACM-managed certificates via `alb_domain_name`.
??? failure "`401 Unauthorized` — client API key missing or wrong"
The API key is missing, wrong, or not configured.
- Pass the key in the `Authorization: Bearer ` header (OpenAI-style) or `X-API-Key` header.
- Retrieve the generated key with `terraform output -raw api_key`.
- If `api_key_create = true` was not set, no API key is configured and requests pass through without authentication by default (useful for testing behind IP-restricted ALB, not for production).
- See [Authentication & Security](operations_authentication_security.md) for all options.
??? failure "`503` saying the feature is not available on the current server — IAM permission denied on an AWS call"
The gateway reached the AWS service, but the **ECS task role** (or your local AWS credentials) lacks permission for the action it used. AWS returns `AccessDeniedException`, which stdapi.ai answers as a feature this deployment cannot run: HTTP `503`, error code `feature_unavailable`, and the same generic message whatever is missing. This is an IAM misconfiguration, **not** a client API-key problem, and the client is deliberately told nothing about it — **the server log names the operation, the model and the permission AWS refused**, under `error_detail`.
- Confirm the task role grants `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` (and the `bedrock:Converse*` actions) for the target model ARNs.
- For models invoked through an inference profile, allow both the profile ARN and the underlying foundation-model ARNs in the policy.
- Some models require one-time activation in the **Bedrock console → Model access** page before they can be invoked.
- Audio, embeddings, and file features need permissions for the relevant services (Polly, Transcribe, Translate, Comprehend, S3) — see [Configuration → IAM Permissions](operations_configuration.md#iam-permissions) for the full IAM reference.
- A `403 permission_error` on a model call means the opposite: [per-user cost attribution](operations_cost_management.md#per-user-attribution) is enabled and the **end user's** role was denied, so the policy to fix is that role's, not the task role's.
- A `400 invalid_request_error` naming an `s3://` input also means the opposite: the denial is on the object *the request pointed at*, in one of the external buckets declared in [`AWS_S3_ACCEPTED_BUCKETS`](operations_configuration.md#aws-s3-accepted-buckets). Only the caller can fix that one — a wrong key, or a bucket policy that does not grant this deployment's role `s3:GetObject` on the object. Objects in the deployment's own buckets keep the `503` above.
??? failure "`401 Unauthorized` — AWS credentials invalid or expired (often local Docker)"
stdapi.ai's **own** AWS credentials are missing, invalid, or expired — AWS returns `UnrecognizedClientException`, `InvalidSignatureException`, or `ExpiredTokenException`, which stdapi.ai maps to HTTP `401` with error type `authentication_error`. This is distinct from the client-facing API-key `401` above (which concerns your `Authorization` / `X-API-Key` header).
- Locally: refresh with `aws sso login` (or update `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_SESSION_TOKEN`) and restart the container.
- On ECS: confirm the task is assuming its IAM role rather than relying on stale static keys.
??? failure "`404 Not Found` — `The model ... does not exist or you do not have access to it`"
The name in the request is not one this deployment serves. Most names need no change when you adopt the gateway: the Anthropic, OpenAI and Cohere models Bedrock serves are published under their providers' own names as well as their Bedrock IDs, derived mechanically from the ID, so `anthropic.claude-opus-5` also answers to `claude-opus-5`, `openai.gpt-oss-120b-1:0` to `gpt-oss-120b` and `cohere.rerank-v3-5:0` to `rerank-v3.5`. This `404` is what a name that *does* differ looks like: one your application hard-codes for a model this deployment does not serve (`gpt-4o`, `gpt-3.5-turbo`, `text-embedding-3-small`, `dall-e-3`) resolves only if you map it yourself with [`MODEL_ALIASES`](operations_configuration.md#model-aliases). Nothing is substituted on your behalf, because a lookalike would serve a different model than the one you asked for.
- List what this deployment actually serves: `GET /search_models` (the default model-discovery endpoint). Filter by capability with query parameters — e.g. `GET /search_models?input_modalities=IMAGE&route=/v1/chat/completions` returns only vision-capable chat models. See the [Search Models API](api_search_models.md) reference.
- `GET /v1/models` is also available for strict OpenAI SDK compatibility (lighter payload, no capability metadata).
- Confirm the pipeline itself with a low-friction model: `amazon.nova-micro-v1:0` (available in all standard Bedrock regions).
- **Keep the name your application already sends** by mapping it onto a served model with [`MODEL_ALIASES`](operations_configuration.md#model-aliases) — an alias only ever points at a model this deployment serves, so what it resolves to is your choice, not a guess.
- Try the name with and without its `anthropic.` / `openai.` / `cohere.` prefix: both forms resolve for those three families, so `anthropic.claude-fable-5` and `claude-fable-5` reach the same model, as do `openai.gpt-oss-120b-1:0` and `gpt-oss-120b`. Cohere additionally spells its versions with a dot, so `cohere.embed-english-v3` answers to `embed-english-v3.0` and `cohere.rerank-v3-5:0` to `rerank-v3.5` — not to `embed-english-v3` or `rerank-v3-5`. Models from other providers are served under their Bedrock IDs only, so name the ID or alias it. A model version Bedrock has retired needs a current one, whichever form you use.
- Only if the model *is* one Bedrock serves and it is still missing: verify `AWS_BEDROCK_REGIONS` includes a region that offers it — see the [Bedrock model availability table](https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html). Adding regions never makes a name Bedrock does not serve resolve.
??? failure "Every Bedrock Mantle model is missing, while the classic Bedrock models are all there"
The catalog lists the classic Bedrock models and none of the ones served through Amazon Bedrock Mantle, in every configured region at once. A per-region or per-model gap looks different: this is the shape of a network policy that reaches one endpoint and not the other.
Bedrock Mantle is served from `bedrock-mantle..api.aws` — a different domain from classic Bedrock's `bedrock-runtime..amazonaws.com`, so an allowlist, firewall rule, proxy exception or VPC endpoint written for `amazonaws.com` does not cover it.
- Allow `bedrock-mantle..api.aws` outbound, for every region in `AWS_BEDROCK_REGIONS`.
- On a private deployment, create the interface VPC endpoint `com.amazonaws..bedrock-mantle` with private DNS enabled.
- Behind a proxy, make sure `HTTPS_PROXY` is set in the task environment and that `NO_PROXY` does not exclude `api.aws`.
- Confirm the task role carries the Bedrock Mantle permissions — without them the models are simply not listed rather than refused. See [IAM Permissions](operations_iam_permissions.md).
- Check what was discovered: `GET /search_models` returns every model the server found, with its regions.
See [Outbound Network Requirements](operations_deploy_advanced.md#outbound-network-requirements) for the full destination list.
The startup warning names the region, the endpoint address and the exception chain behind the failure, so a blocked route (`ConnectionTimeoutError`), a refused connection (`ConnectionRefusedError`), an intercepting proxy's certificate (`SSLCertVerificationError`) and an unresolvable address (`ClientConnectorDNSError`) are told apart without further instrumentation.
??? failure "Bedrock Mantle models are missing in one region only — `bedrock_mantle_regions_without_endpoint`"
The startup log lists the region under `bedrock_mantle_regions_without_endpoint` instead of `unreachable_bedrock_regions`. Bedrock Mantle is offered in fewer regions than classic Bedrock, and where it is not offered `bedrock-mantle..api.aws` has no DNS record at all — nothing to retry, and no network policy to change. See [model availability by endpoint](https://docs.aws.amazon.com/bedrock/latest/userguide/models-endpoint-availability.html).
- Remove the region from [`AWS_BEDROCK_MANTLE_REGIONS`](operations_configuration.md#bedrock-mantle-regions), or unset it to fall back to the regions of `AWS_BEDROCK_REGIONS` that offer Mantle. Classic Bedrock in that region is unaffected either way.
- If no configured region offers Mantle, the log says so as well and no Mantle model is served — set [`AWS_BEDROCK_MANTLE_ENABLED`](operations_configuration.md#bedrock-mantle-enabled) to `false` to stop the warning.
- If AWS has since added the region, list it explicitly in `AWS_BEDROCK_MANTLE_REGIONS`: an explicit list is used as given.
- If the address should resolve because a VPC endpoint provides it, this is instead the private-DNS case covered by *Every Bedrock Mantle model is missing* above.
??? failure "Startup takes tens of seconds — `server_start_time_ms` far above the usual few seconds"
The `start` log event reports `server_start_time_ms` in the tens of thousands where a healthy deployment reports a few thousand. Startup reads the model catalogs of every configured region, and the ECS task metadata endpoint before them; a destination that never answers is only given up on after a timeout.
- Read the `server_warnings` of the same event first. `ECS container metadata endpoint answered after N attempts in X s` accounts for that many seconds on its own: the endpoint is served by the ECS agent over the task ENI and answers slowly when the task is CPU-starved at boot. Raise the task CPU, or the `cpu` of the Fargate task definition, so the agent is scheduled promptly.
- `unreachable_bedrock_regions` and `bedrock_mantle_regions_without_endpoint` each name a region that spent its full timeout budget. Removing the region from `AWS_BEDROCK_REGIONS` or `AWS_BEDROCK_MANTLE_REGIONS` removes the delay.
- [`AWS_CONNECT_TIMEOUT`](operations_configuration.md#aws-connect-timeout) bounds each connection attempt and the model-catalog fetch that follows it, so lowering it lowers what an unreachable region can cost. It also bounds failover between healthy regions, so keep it above your real inter-region latency.
- Model discovery is per region and runs in parallel, so the count of regions costs far less than one unreachable region does.
??? failure "A deploy leaves work unfinished — `abandoned_background_tasks` in the `stop` log event"
Some work is started outside the request that asked for it, so the caller is answered without waiting: temporary file cleanups, vector store file indexing, and the release of live audio sessions. When a task is stopped, the server waits [`SHUTDOWN_DRAIN_TIMEOUT`](operations_configuration.md#shutdown-drain-timeout) seconds for that work, then cancels the rest — and reports the counts as `abandoned_background_tasks` in a `stop` log event raised to `warning`.
- A vector store file reported `failed` after a deployment, with `last_error` saying the indexing was interrupted, or a temporary object that outlived its request, is what those counts look like from the outside. Attach the file again, or let the object's lifecycle rule expire it. A file is never left `in_progress`: one whose indexing is gone is settled the next time it, its store, or the store's file list is read.
- Raise [`SHUTDOWN_DRAIN_TIMEOUT`](operations_configuration.md#shutdown-drain-timeout) **and** the container stop timeout together — the container runtime sends `SIGKILL` a fixed delay after the stop signal (30 seconds by default on Amazon ECS), so raising the wait alone changes nothing past that delay.
- Counts that persist after raising both mean the work itself is long, not that the wait is short: indexing a large file outlasts any stop timeout. Attach large files when a deployment is not in flight, and treat the operation as retryable.
- The wait is best effort in every deployment: a Spot interruption or a hard kill ends the process regardless, so never rely on it for anything whose completion matters.
??? failure "`429 Too Many Requests` — Bedrock throttling / quota"
AWS returned `ThrottlingException`, `TooManyRequestsException`, or `ServiceQuotaExceededException` — mapped to HTTP `429` with error type `rate_limit_error`. You've hit the per-region Bedrock quota.
- Add more regions to `AWS_BEDROCK_REGIONS`. Each region has its own independent quota — three regions ≈ triple the throughput.
- See [Resilience & Failover](operations_resilience.md) for multi-region routing configuration.
- Check quotas in the AWS Service Quotas console for **Amazon Bedrock**.
- When the router put a region on a quota backoff while serving the request, the response carries a `retry-after` header (in seconds) telling the client exactly how long to wait — OpenAI, Anthropic and Cohere SDKs honour it automatically instead of guessing an exponential backoff, up to the 60 s ceiling their retry loops apply to server-supplied delays.
??? failure "`400 Bad Request` — This model is not available under data retention mode 'default'."
A specific model is unavailable or requests to it are rejected because your account's data retention mode is incompatible with what that model requires.
Amazon Bedrock enforces retention compatibility at invocation time: each model declares the retention modes it accepts, and if your effective mode is not among them, the request is blocked.
**Common scenarios:**
- Your account is set to **zero data retention (`none`)** but the model requires `default` or `provider_data_share` for safety or abuse-prevention purposes. Bedrock blocks the request to honour your retention policy. To access the model, either switch to a compatible retention mode or contact your AWS account manager to request ZDR eligibility for that specific model.
- Your account is set to **`default`** but the model exclusively requires `provider_data_share` (typically models with mandatory provider-side safety review). The model will appear as unavailable. Enabling `provider_data_share` grants access but means AWS will share your inference data with the model provider — see [Data Privacy](operations_compliance.md#data-privacy) before enabling it.
??? failure "`400 Bad Request` — invalid parameters from Bedrock"
Bedrock rejected the request parameters (`ValidationException` / `BadRequestException`), mapped to HTTP `400` with error type `invalid_request_error` — for example an unsupported parameter for the chosen model, an out-of-range value, or content that exceeds the model's limits.
- Read the message detail returned in the response (correlate with `x-request-id` in the server logs).
- Confirm the parameter is supported by the model — see the per-API **Feature Compatibility** tables.
??? failure "`400 Bad Request` — a client's own control flag reaches Bedrock as a model parameter"
Request fields stdapi.ai does not declare are forwarded to Amazon Bedrock as provider-specific inference parameters, so any parameter a model accepts can be passed through — including the highly model-specific ones no common API surface exposes. Some OpenAI-SDK-based clients also use that same channel for their *client-side* settings — LiteLLM-derived ones send `drop_params`, `api_key` or `custom_llm_provider` in `extra_body` — and Bedrock answers `ValidationException` for a field no model declares. The symptom is a route that fails for one client and works for every other.
- The known LiteLLM control parameters are stripped by a built-in denylist, so this only appears for a name it does not yet cover. The rejected field is in the Bedrock message detail, correlated via `x-request-id` in the server logs.
- Add that name to [`EXTRA_MODEL_PARAMS_DENYLIST`](operations_configuration.md#extra-model-params-denylist) — it is merged with the built-in list, and every other extra parameter keeps being forwarded.
- If no client needs the passthrough, [`EXTRA_MODEL_PARAMS_DROP_ALL`](operations_configuration.md#extra-model-params-drop-all) disables it outright. Per-model defaults set through [`DEFAULT_MODEL_PARAMS`](operations_configuration.md#default-model-params) are unaffected — only request-supplied extras are dropped.
??? failure "A request's `service_tier`, guardrail or model parameters are ignored"
Some configuration reaches the model that the client did not send, or the value the client sent is not the one applied. Two layers of server-side configuration sit behind every request, and both are deliberate.
- The model name may be an alias carrying its own configuration — check the entry in [`MODEL_ALIASES`](operations_configuration.md#model-aliases-configuration) for that name. Requests naming it get its service tier, guardrail, metadata and model parameters; requests naming the target model directly do not.
- The reverse also happens: alias and server-wide configuration apply to models served through Amazon Bedrock's Converse and InvokeModel operations. A model served through Amazon Bedrock Mantle applies the request's own values only, so a configured service tier, metadata or model parameters are ignored there by design — see [the scope note](operations_configuration.md#model-aliases-configuration).
- A request value is discarded on purpose when its override setting is disabled: [`AWS_BEDROCK_ALLOW_SERVICE_TIER_OVERRIDE`](operations_configuration.md#aws-bedrock-allow-service-tier-override) for the tier, [`AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE`](operations_configuration.md#aws-bedrock-allow-guardrail-override) for the guardrail headers.
- Otherwise the value comes from the server-wide setting for that field — [`DEFAULT_MODEL_SERVICE_TIERS`](operations_configuration.md#default-model-service-tiers), [`DEFAULT_MODEL_PARAMS`](operations_configuration.md#default-model-params) or [`AWS_BEDROCK_GUARDRAIL_IDENTIFIER`](operations_configuration.md#aws-bedrock-guardrail-identifier). The order is always the request, then the alias, then the setting.
??? failure "Startup fails with a validation error naming a model alias"
An alias in [`MODEL_ALIASES`](operations_configuration.md#model-aliases-configuration) maps to an object that is not a valid alias configuration, so the server refuses to start rather than ignore it.
- `Extra inputs are not permitted` names a field that does not exist — check its spelling against the [alias fields](operations_configuration.md#model-aliases-configuration).
- `Field required` on `model` means the object gives configuration but no target model.
- A guardrail needs both `guardrail_id` and `guardrail_version`.
- An alias that only maps a name to a model stays a plain string: `{"my-model": "amazon.nova-lite-v1:0"}`.
??? failure "`400 Bad Request` — text-to-speech rejects a long `input`"
The message states the length the server accepts (`'input' is limited to 3,000 characters…`, or 20,000 with a generative voice). A generative voice speaks up to 20,000 characters unaided; every other voice, and longer generative input, is synthesized into an S3 bucket co-located with the region serving the request, and none is configured there.
- Set [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) for the first region of `AWS_BEDROCK_REGIONS`, and an [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets) entry for every other region that may serve speech — the server log names the one that was missing.
- Grant the task role `polly:StartSpeechSynthesisStream` (generative voices beyond 3,000 characters), `polly:StartSpeechSynthesisTask`, `polly:GetSpeechSynthesisTask`, and S3 read/write/delete on those buckets — see [IAM Permissions](operations_iam_permissions.md#text-to-speech-optional).
- A generative request that still behaves like the others — rejected without a bucket, or slower than expected with one — is missing `polly:StartSpeechSynthesisStream`; the server log names the failure.
- Up to 3,000 characters (6,000 including SSML markup) never needs a bucket; the limits and the expected latency are in [Long Input](api_openai_audio_speech.md#long-input).
??? failure "`400 Bad Request` — a transcription is rejected only for one model"
The message names the condition: `amazon.nova-2-sonic-v1:0` serves `json` and `text` only, and accepts at most 10 minutes of audio per request. It returns no timestamps and does not report a detected language, so subtitles, `verbose_json`, `diarized_json` and `timestamp_granularities` cannot be produced from it.
- Request `json` or `text`, or send the same audio to `amazon.transcribe`, which produces timestamps, SRT/VTT subtitles, speaker diarization and longer recordings — see [Transcriptions](api_openai_audio_transcriptions.md).
- The same model and the same limits apply on [Translations](api_openai_audio_translations.md).
??? failure "`400 Bad Request` — every transcription fails after setting an output encryption key"
The message is Amazon Transcribe's own failure reason for the job, and it names KMS. Only requests that stage audio in a bucket are affected: [`AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-transcribe-output-encryption-key-arn) encrypts the job's output, so streamed transcriptions, which write nothing, keep working — which is what makes the failure look model-specific at first.
- Grant the task role `kms:GenerateDataKey` and `kms:Decrypt` on that key, **and** allow the same role in the key's own policy — a grant on only one of the two denies the job. See [Speech-to-Text](operations_iam_permissions.md#speech-to-text-optional).
- With [`AWS_TRANSCRIBE_REGION`](operations_configuration.md#aws-transcribe-region) unset, a job runs in whichever candidate Region has a co-located bucket, so a single-Region key fails as soon as failover moves the job. Use a [multi-Region key](https://docs.aws.amazon.com/kms/latest/developerguide/multi-region-keys-overview.html), or pin the Region.
- A key policy conditioned on the [encryption context](operations_configuration.md#aws-transcribe-output-encryption-key-arn) must not require `stdapi-ai.user_id`: it is sent only when the request identifies an end user, so requiring it denies every anonymous call.
??? failure "`amazon.nova-2-sonic-v1:0` is missing from `/v1/models` or returns `404`"
The model is not offered in every AWS Region, and the catalog only lists what the configured Regions serve. [Check the model's regions](models.md).
- Add a Region that offers it to [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions); `us-east-1`, `us-west-2`, `ap-northeast-1` and `eu-north-1` did at the time of writing, while `eu-west-3` and `eu-central-1` did not.
- Request model access for it in the Amazon Bedrock console for that Region.
- Grant the task role `bedrock:InvokeModelWithBidirectionalStream`, which this model needs on top of the usual invoke permissions — see [IAM Permissions](operations_iam_permissions.md). Without it the session opens and then ends with a `503` saying live conversation is not available on this server; the server log names the permission.
- `/search_models?route=openai_audio_transcription` lists what this deployment can actually reach.
??? failure "`503 Service Unavailable` — a speech or speech-to-speech request fails a few seconds after it starts"
Real-time audio requests must become ready within [`AWS_CONNECT_TIMEOUT`](operations_configuration.md#aws-connect-timeout) in each candidate region — that budget covers the connection, the initial handshake and the first response together, not the connection alone. On a high-latency or NAT-fronted network the default of 5 seconds can expire in every region, and the request then ends as a `503`.
- Raise [`AWS_CONNECT_TIMEOUT`](operations_configuration.md#aws-connect-timeout) to 10 seconds or more; `AI_RESPONSE_TIMEOUT` governs the response itself and has no effect here.
- The server log names each Region that was abandoned and why.
??? failure "Conversation items are missing, or cannot be added"
A [conversation](api_openai_conversations.md) has a bounded lifetime and a bounded number of writes, and both are reached silently.
- **After 30 days**, a conversation and its items are removed and every route on it returns `404`. Long-lived agents must create a new conversation rather than reusing one indefinitely.
- **1,000 requests that add or delete items** is the per-conversation ceiling; a response bound to a conversation counts as one, whatever its number of output items. Past it, a listing stops early rather than adding failing: the gateway reads at most 1,000 invocation steps, and a single large item spans several. Start a new conversation, seeding it with the items you still need.
- **`503` saying the API is not available on the current server** means the IAM role is missing the [Bedrock Session Storage permissions](operations_iam_permissions.md#bedrock-session-storage-optional), including `bedrock:UpdateSession`, which only the metadata update uses — a deployment created before conversations shipped fails on `POST /v1/conversations/{id}` alone. The client message is the same whichever one is absent; the server log names it.
- **Items added by a streamed response appear when the stream ends**, not while it runs; a client that reads them from a callback fired on the terminal event must wait for the stream to close.
??? failure "S3 error on image generation or audio transcription"
The S3 bucket is missing, unreachable, or in the wrong region.
- The Terraform module creates the bucket automatically unless you pass your own via `aws_s3_bucket`.
- If you're using your own bucket: `AWS_S3_BUCKET` must point to a bucket in the same region as the **first** entry in `AWS_BEDROCK_REGIONS`.
- Verify the ECS task IAM role has `s3:PutObject` / `s3:GetObject` on the bucket.
- **`503` saying transcription is not available on the current server** means no region that can run a transcription has a bucket at all: set [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket), [`AWS_TRANSCRIBE_S3_BUCKET`](operations_configuration.md#aws-transcribe-s3-bucket) or an [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets) entry for it. The server log names the settings.
- **`503` saying the `'url'` response format is not available on the current server** means there is no bucket to host the images a `url` response points at: set [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket), or request `response_format="b64_json"`, which needs no storage. Image requests are refused before anything is generated, so a misconfigured deployment bills nothing for them.
??? failure "Connection timeout to AWS services from ECS"
Outbound traffic to AWS endpoints is blocked.
- Confirm the ECS task's security group allows outbound HTTPS (port 443).
- If using **VPC endpoints** (the Terraform module default), verify the endpoint security groups and policies permit traffic from the ECS task subnet.
- If ECS runs in a private subnet without VPC endpoints, confirm the NAT gateway / route table is configured.
??? failure "`413 Payload Too Large` — request or file rejected as oversized"
Either an attachment exceeds what the chosen model reads, the application-level file-size cap, or an edge control rejected the request.
- `Attachments larger than … are not available on the current server` means the attachment is too large to travel inside the request and there is nowhere to stage it: no region able to serve that model has an S3 bucket. Set [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) for the first region of `AWS_BEDROCK_REGIONS` and an [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets) entry for every other region that may serve the model — the server log names the one that was missing. The same request succeeds unchanged once a bucket exists.
- `An attached file is too large: this model accepts at most … bytes per file` means that model only reads attachments sent inside the request. Send a smaller file, or choose a model that reads attachments from storage — see [Attachment Size](features.md#attachment-size).
- `The attached files are too large: this model accepts at most … bytes of attachments per request` means the same, for the request as a whole: each file fits on its own, but their total does not. Split them across several requests, or choose a model that reads attachments from storage.
- Check [`MAX_INPUT_FILE_SIZE`](operations_configuration.md#max-input-file-size) — it caps the bytes of any single file loaded into memory for model input; disabled by default, so if it's set and the error appears, raise it or reduce the input size.
- If the deployment sits behind the Terraform module's WAF (`alb_waf_enabled=true`), check for a `SizeConstraintStatement` rule on the request body — see [Request Size & Resource Limits](operations_authentication_security.md#request-size-resource-limits).
- If fronted by Amazon API Gateway instead of an ALB, remember its hard 10 MB payload limit.
??? failure "Web search returns nothing, stale results, or is rejected"
The built-in [web search tool](api_openai_responses.md#openai-gpt-web-search) is gated by both an IAM permission and a server setting, and each failure looks different.
- **The model answers from its training data and says it could not search**: the task role is missing `bedrock-websearch:InvokeSearch` / `bedrock-websearch:InvokeFetch`. Add them — see [Web Search IAM](operations_iam_permissions.md#web-search-iam). The request itself still succeeds, so this shows up as a weak answer rather than an error.
- **`400` on `external_web_access`**: the request asked for a value the server does not allow. By default searches stay inside the AWS boundary; set [`AWS_BEDROCK_EXTERNAL_WEB_ACCESS`](operations_configuration.md#bedrock-external-web-access) to change what the server does, or [`AWS_BEDROCK_ALLOW_EXTERNAL_WEB_ACCESS_OVERRIDE`](operations_configuration.md#bedrock-allow-external-web-access-override) to let requests choose. It travels as an extra model parameter (a top-level request field), not as a field of the tool: a client that sets it on the tool changes nothing.
- **`400` on `external_web_access` saying it is not available with this model**: the override is enabled, but only the models that serve web search natively — the OpenAI GPT-5.x family — take a web access choice per request. Everywhere else the parameter must match the configured value, and is refused rather than accepted and ignored. Send it only with those models, or drop it.
- **External web access was enabled but results still look cached**: `bedrock-websearch:ExternalWebAccess` is missing from the task role, so the search falls back to the Amazon Bedrock web index.
- **`400` naming `filters.allowed_domains` or `user_location`**: the model's own search cannot restrict which sources it uses, and running it unrestricted would answer from the very sources the request excluded. Drop the restriction, or send the request to a model that serves web search natively — the OpenAI GPT-5.x family.
- **The tool is rejected outright**: web search is served on `/v1/responses` for the OpenAI GPT-5.x family, in `us-east-1`, `us-east-2` and `us-west-2`. On `/v1/messages` and `/v1/chat/completions` it is not available for these models.
??? failure "`503`/`504`/`408` — request times out mid-stream on long generations"
A slow or hung generation exceeded a timeout somewhere between the model and the client. The status code tells you where: `503` comes from stdapi.ai's own gateway timeout; `504`/`408` come from an edge or proxy timeout in front of it.
- **`503` from the gateway**: Check [`AI_RESPONSE_TIMEOUT`](operations_configuration.md#ai-response-timeout) — it closes stalled upstream model connections; raise it for workloads with long-running generations. The request is **not** retried in another region: the model already ran and AWS bills it either way, so a failover would pay twice for the same generation.
- **`504`/`408` from the edge/proxy**: Check the Terraform module's `alb_idle_timeout` (default: 3600 s) — if you lowered it, or front the deployment with your own load balancer or reverse proxy at a shorter idle timeout, streaming responses can be cut off mid-flight before the gateway's own timeout fires. See [ALB Resilience](operations_resilience.md#alb-resilience).
??? failure "Every request fails after enabling per-user cost attribution"
Model calls run under a session of [`AWS_BEDROCK_USER_ROLE_ARN`](operations_configuration.md#aws-bedrock-user-role-arn), and a session that cannot be opened fails the request rather than silently falling back to the server's identity. The server also reports this at startup, in the `server_warnings` field of its `start` log event. Five causes, in order of likelihood:
- **The trust policy allows only `sts:AssumeRole`.** Tagging the session is a separate action: add `sts:TagSession` to both the trust policy of the end user role and the server's own policy — see [Per-User Cost Attribution IAM](operations_iam_permissions.md#per-user-cost-attribution). Setting [`AWS_BEDROCK_USER_ROLE_TAG_KEY`](operations_configuration.md#aws-bedrock-user-role-tag-key) to null removes the need for it, at the cost of Cost Explorer grouping.
- **The role was just created.** A new or newly-edited trust policy takes a few seconds to propagate; a task started immediately after logs the startup warning and recovers on its own.
- **`403` on every request**: the end user role lacks `bedrock:InvokeModel` or `bedrock:InvokeModelWithResponseStream`, or its `Resource` list misses an ARN form requests actually reach. A cross-region inference profile also needs `arn:aws:bedrock:*::foundation-model/...` for every Region it routes to, and an application inference profile, a prompt router or a prompt ARN each has to be named in its own right — see [Per-User Cost Attribution IAM](operations_iam_permissions.md#per-user-cost-attribution).
- **`403` once a guardrail is configured**: a guardrail applied during an invocation — [`AWS_BEDROCK_GUARDRAIL_IDENTIFIER`](operations_configuration.md#aws-bedrock-guardrail-identifier), a model alias carrying one, or a request-level `moderation` parameter — is evaluated as part of the call the end user signed, so the end user role needs `bedrock:ApplyGuardrail` on the guardrail ARN as well.
- **`400` naming `safety_identifier`**: [`AWS_BEDROCK_USER_ROLE_REQUIRE_IDENTITY`](operations_configuration.md#aws-bedrock-user-role-require-identity) is enabled and the client sends no end user identifier. Either have the client send one, or disable that setting.
??? failure "A batch is refused, stuck, or its requests fail one by one"
The [Batch API](api_openai_batches.md) and the [Message Batches API](api_anthropic_batches.md) refuse at submit what the backend would otherwise fail hours later, so most surprises land in the create call.
- **`503` (or `529` on `/anthropic/...`) on every batch endpoint**: the deployment declares no batch service role. Set [`AWS_BEDROCK_BATCH_ROLE_ARN`](operations_configuration.md#aws-bedrock-batch-role-arn) and grant the policies in [Batch Inference IAM](operations_iam_permissions.md#batch-inference). The server also reports the disabled feature in the `server_warnings` field of its `start` log event.
- **`400` naming a minimum of 100 requests**: batches run on a backend with a floor of 100 requests **per model**. A batch naming several models must reach it for each of them; combine the small ones or send them without batching. The 100 is the default of the per-model Amazon Bedrock quota *Minimum number of records per batch inference job* ([Amazon Bedrock quotas](https://docs.aws.amazon.com/general/latest/gr/bedrock.html)), and it is the value checked here whatever your own account's quota says.
- **The batch stays `validating` (`in_progress`) for several minutes**: expected. Validation alone takes a few minutes before any request runs, and the whole batch has a 24-hour window. Poll rather than resubmit — a resubmission is a second, separately billed batch.
- **`400` naming tool use or a structured output schema**: neither is available in a batch. Remove `tools`/`tool_choice` and `response_format` of type `json_schema`, or send those requests without batching.
- **`400` saying the model is not available for batched requests**: not every model can run batched. Pick another one; when the batch named several models, no sibling job is left running.
- **A model that batches fine is missing from `search_models?batch=true`, or reports `batch: false`**: that flag is a discovery hint published on a best-effort basis and is never used to reject anything — submit the batch and let the answer decide. It is reported for no model at all while [`COST_TRACKING`](operations_configuration.md#cost-tracking) is disabled, and for a few seconds after startup while the catalogue is still being built.
- **`503` on creation, with nothing wrong with the request**: the backend refused the job for a reason that is not the model — the account's batch quota for that model, the service role, or a restriction such as a model the provider marked legacy and the account has not used in the last 30 days. The client message is deliberately generic; the server log carries the reason the backend gave, as a warning.
- **`503` on creation, after the endpoints answered normally**: the task role is missing `bedrock:CreateModelInvocationJob` or the `iam:PassRole` statement on the batch service role; the server log names which. A batch that starts and then fails without results usually means the service role itself cannot read or write the bucket under [`AWS_S3_BATCHES_PREFIX`](operations_configuration.md#aws-s3-batches-prefix) — the reason the backend gives is logged as a warning when a job reports `Failed`.
- **A batch reports no cached tokens, whatever its requests asked for**: prompt caching does not apply to batched requests, on any model. A cache hint — `cache_control` on `/anthropic/v1/messages/batches`, `prompt_cache_key` or `prompt_cache_breakpoint` on `/v1/batches` — is accepted and dropped rather than refused, so the request is answered normally and no cached tokens are reported for it. There is no discount to lose: batched requests are billed at the batch rate already.
- **The first read after the batch ends is slow**: the results are translated and published on that read. Later reads are immediate.
- **The result files are gone, or never go away**: a batch created with `output_expires_after` deletes both files that long after they are written, and one created without it keeps them until they are deleted with the [Files API](api_openai_files.md). The clock starts when the results are published, not when the batch was created.
??? failure "A vector store file stays in progress, fails, or returns nothing"
Indexing runs after the response is sent, so a file is `in_progress` for a moment by design — see [Vector Stores](api_openai_vector_stores.md#indexing-is-asynchronous).
- **`503` on every vector store endpoint**: the deployment declares no vector storage. Set [`AWS_S3_VECTORS_BUCKET`](operations_configuration.md#aws-s3-vectors-bucket) and [`AWS_S3_VECTORS_REGION`](operations_configuration.md#aws-s3-vectors-region), keep [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) set, and grant the [Vector Stores IAM permissions](operations_iam_permissions.md#vector-stores-optional).
- **`503` on one operation only** (creating a store, searching, deleting): a single `s3vectors` action is missing from the task role. The client message is deliberately the same as above; the server log names the action and the bucket.
- **Creating a store fails**: the vector bucket must already exist, in the Region named by [`AWS_S3_VECTORS_REGION`](operations_configuration.md#aws-s3-vectors-region). A vector bucket is a Region-local resource with no failover, so a bucket in another Region is not reachable at all.
- **A file settles as `failed` with `unsupported_file`**: the store does not index that file type. Read `last_error.message` — it names what **this** store indexes, since that differs per store. A store the server owns indexes text only, so a PDF or an office document settles here; convert it first — [RAG Pipelines](use_cases_rag.md#document-parsing) shows a conversion stage — or attach it to a [knowledge base store](api_openai_vector_stores.md#knowledge-base-stores), which indexes those formats as they stand. When the message names formats and the file is already one of them, its bytes are not what the content type claims.
- **A file settles as `failed` with `server_error`**: indexing hit a backend error, or was interrupted before it finished — `last_error.message` says which. Interrupted means the server was replaced, scaled in or killed while it was indexing, and nothing else is wrong. Check the `background` log event sharing the request's `id` for the backend case; attach the file again in both. To stop losing that work at every deployment, give the deployment an indexing queue — [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url) — and another server finishes the job instead.
- **A file stays `in_progress` far longer than the others**: a large file is many passages, each embedded in turn, and indexing is bounded server-wide, so a file attached while others are being indexed waits its turn. A file no server is indexing any more settles as `failed` rather than waiting for good, so an unchanging `in_progress` is work that is still queued.
- **A search returns nothing after attaching**: the store is still indexing (`status` is `in_progress`), the store has passed its expiration (`status` is `expired`), or the `filters` match no file. A filter applies to the file's `attributes`, never to its content.
- **`409` on an update**: several requests are changing the same store at once. Retry the request.
??? failure "Vector store indexing is not picked up by the queue, or a queued file never settles"
Only deployments that set [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url) hand indexing to a queue; without it, indexing runs in the server that accepted the request and everything below is expected behaviour rather than a fault.
- **The server refuses to start, naming the setting**: the URL is not an Amazon SQS queue URL (`https://sqs..amazonaws.com//`), it names a FIFO queue, or [`AWS_S3_VECTORS_BUCKET`](operations_configuration.md#aws-s3-vectors-bucket) is unset. FIFO is refused because its deduplication would silently drop a legitimate re-attach of the same files.
- **A `start` log warning says the queue could not be described**: the queue does not exist, or the task role lacks `sqs:GetQueueAttributes`. The deployment still runs and still queues, but it cannot read your redrive policy, so it falls back to its own retry count. Grant the [Durable Vector Store Indexing permissions](operations_iam_permissions.md#durable-vector-store-indexing).
- **A `start` log warning says the queue has no dead-letter queue**: add a redrive policy. Without one, the message of a file that cannot be indexed is dropped once its retries run out instead of being kept for inspection.
- **Every file still settles as `failed` after a deployment**: the send is failing, which the server log reports at `error` naming `sqs:SendMessage`. A deployment that cannot queue keeps indexing in-process, which is exactly the behaviour the setting was meant to replace, so the symptom looks like the setting doing nothing.
- **Files sit `in_progress` for minutes under load**: a server only takes jobs off the queue while it is not busy answering requests, so indexing yields to clients by design. Scale out, or wait.
- **A file settles as `failed` although the queue is configured**: the job ran out of deliveries. Its message is in your dead-letter queue; the server log says so at `error`. Attach the file again once the underlying cause is fixed.
??? failure "A `vs_kb_...` vector store answers `404`, or refuses a file attached to it"
A [knowledge base store](api_openai_vector_stores.md#knowledge-base-stores) is addressed, never created, so most of these are configuration rather than a bad request.
- **`404` on every route of a `vs_kb_...` identifier**: the knowledge base is not listed in [`AWS_BEDROCK_KNOWLEDGE_BASE_IDS`](operations_configuration.md#aws-bedrock-knowledge-base-ids), or it does not exist in the first [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) entry, or the task role lacks the read permission on it. The three cases answer identically **by design**, so the allowlist cannot be probed by a client; the server log says which one it was. Check the setting first, then grant the [Knowledge Base Vector Stores permissions](operations_iam_permissions.md#knowledge-base-vector-stores) on the knowledge base ARN.
- **`503` when attaching a file, while search and listing work**: the knowledge base has more than one data source, so which one a document belongs to is ambiguous. Name it in the setting as `/`.
- **`400` saying files cannot be attached to the store, while search and listing work**: the allowlisted data source keeps its corpus in sync from somewhere else — a bucket, or another connected service — and takes no file handed to it. Only a **custom** data source does. Point the entry at one, as `/`; the server log names the data source that refused. A knowledge base can hold both kinds, and the store keeps serving search and listing meanwhile.
- **`400` when deleting a file a search returned**: that document belongs to the corpus behind the store rather than to the files attached here, so it is readable and never removable. Remove it where the corpus comes from.
- **`400` on an update, a delete, a chunking strategy, a file batch, a `score_threshold` or a file's `content`**: none of these apply to a store managed outside the server. The [refusal table](api_openai_vector_stores.md#knowledge-base-stores) lists each one and what to do instead.
??? failure "All Bedrock spend still lands on one identity"
[Per-user attribution](operations_cost_management.md#per-user-attribution) reaches the AWS bill through two AWS-side steps that are easy to miss, and neither is instant:
- **The Cost and Usage Report export must include caller identity.** Create a Data Exports CUR 2.0 export with *Include caller identity (IAM principal) allocation data* enabled; an existing export cannot be changed and must be re-created. The identity then appears in `line_item_iam_principal` as `assumed-role//`.
- **The session tag must be activated as a cost allocation tag**, in the AWS Billing console under **Cost allocation tags**, filtered by type **IAM principal**. It is only listed there after that identity has made at least one call, and takes up to 24 hours to appear in Cost Explorer.
- **Requests that identify no end user are billed to the server**, by design. The request log's `aws_role_session_name` field is absent on exactly those requests — use it to find the clients that send no identifier, then enable [`AWS_BEDROCK_USER_ROLE_REQUIRE_IDENTITY`](operations_configuration.md#aws-bedrock-user-role-require-identity).
- **Only model invocations are attributed.** Video generation, guardrail evaluations, speech, transcription and translation stay on the server's own identity.
??? failure "A WebSocket upgrade to `/v1/realtime` answers `404`"
The upgrade request never reached the [Realtime API](api_openai_realtime.md) route — either the URL is wrong, or something in front of the deployment does not forward a WebSocket upgrade at all.
- Check the path: the route lives at `${OPENAI_ROUTES_PREFIX}/v1/realtime` (`/v1/realtime` with the default empty prefix). A client dialling the wrong prefix reaches no route and gets `404` from the framework itself, before authentication is even checked.
- An Application Load Balancer forwards a WebSocket upgrade by default, but anything placed **in front of** it — a CDN, an API Gateway REST API (which has no WebSocket support at all), a reverse proxy that does not pass through `Connection: Upgrade` / `Upgrade: websocket` — answers its own `404` or refuses the upgrade first. See [WebSocket-Capable Deployment](operations_deploy_advanced.md#websocket-capable-deployment-realtime-api).
- Confirm the deployment actually ships the Realtime API: this is a versioned feature, not every earlier release includes it.
??? failure "`POST /v1/realtime/calls` answers `404`"
That endpoint is the upstream API's WebRTC and SIP transport negotiation, and it is **not served here** — every realtime session runs over the WebSocket. Nothing in the deployment enables it: no setting, no IAM permission and no load balancer configuration changes the answer. See [Transports](api_openai_realtime.md#transports).
- **A browser does not need it.** It connects to the same `WS /v1/realtime` with an [ephemeral client secret](api_openai_realtime.md#ephemeral-client-secrets) in the `Sec-WebSocket-Protocol` list, and captures and plays the audio itself.
- **A client that must speak WebRTC or SIP** — a phone line, or a browser on a lossy network that needs jitter and packet-loss handling — belongs behind a voice-agent framework that terminates the media itself and reaches this API over the WebSocket. See [Put WebRTC or a phone line in front of the gateway](api_openai_realtime.md#put-webrtc-or-a-phone-line-in-front-of-the-gateway).
- **Do not try to route UDP through the deployment's ALB**: its listeners carry HTTP and HTTPS only. A media terminator you run yourself needs its own ingress — see [WebRTC and SIP need their own ingress](operations_deploy_advanced.md#webrtc-and-sip-need-their-own-ingress).
??? failure "A realtime session closes exactly at 8 minutes, or unexpectedly earlier"
Two different things end a [Realtime](api_openai_realtime.md) session, and the close code tells them apart — inspect it in the client's WebSocket close handler.
- **Close code `1000`, reason `session_expired`**: expected. Every session is capped at 8 minutes; reconnect to continue the conversation — see [Session Lifecycle and Limits](api_openai_realtime.md#session-lifecycle-and-limits).
- **Closes with no code from the gateway at all, well before 8 minutes**: the load balancer's idle timeout fired on a quiet stretch between spoken turns. Raise `alb_idle_timeout` to at least 8 minutes (480 seconds) — see [The idle timeout bounds a session](operations_deploy_advanced.md#the-idle-timeout-bounds-a-session).
- **Closes during a deployment, scale-in, or Spot interruption**: the ECS task holding the session was replaced. There is no live handoff between tasks — see [A deploy truncates open sessions](operations_deploy_advanced.md#a-deploy-truncates-open-sessions).
- **Close code `1001`, reason `server_shutdown`**: the deployment was shutting down when the session was still open; reconnect once it is back.
- **Close code `3000`**: a fatal error, not a limit. The reason is `.`, and a terminal `error` event carrying the same detail was sent just before the close frame.
??? failure "A raw WebSocket client gets `403` that looks like an authentication failure"
If the deployment's WAF is enabled with the AWS-managed Common Rule Set (`alb_waf_enabled = true`), its `NoUserAgent_HEADER` rule blocks any request — including a WebSocket upgrade — that carries no `User-Agent` header, with a plain `403` that is easy to mistake for a rejected credential.
- Every mainstream WebSocket client library sets a `User-Agent` automatically; this only surfaces with a hand-rolled client (a bespoke SIP/telephony bridge, a minimal test script).
- Check the WAF sampled requests in the console to confirm `NoUserAgent_HEADER` is the rule that matched, before assuming the API key or ephemeral secret is wrong.
- Have the client send any non-empty `User-Agent`, or exclude the rule for the Realtime path. See [AWS WAF's `NoUserAgent_HEADER` rule reads like an auth failure](operations_deploy_advanced.md#aws-wafs-nouseragent_header-rule-reads-like-an-auth-failure).
??? failure "An ephemeral client secret works on one instance and is rejected on another"
A [Realtime API ephemeral client secret](api_openai_realtime.md#ephemeral-client-secrets) is a signed token with nothing stored server-side, verified by re-checking its signature against a shared key — every instance must sign with the **same** key for that to work.
- With no [`API_KEY`](operations_configuration.md#api-key)-family setting configured at all, each instance falls back to a **random signing key generated per process**, so a secret minted by one instance never verifies on another — the symptom is intermittent rejection that tracks which instance the client's connection happened to land on.
- Set [`REALTIME_CLIENT_SECRET_KEY`](operations_configuration.md#realtime-client-secret-key) explicitly to a value shared by every instance; this also covers a deployment with no API key by design (e.g. behind an IP-restricted ALB).
- A deployment that already configures an API key is unaffected: the signing key is derived from it automatically, and that same key is already shared across instances.
- Rotating the API key or `REALTIME_CLIENT_SECRET_KEY` invalidates every client secret minted before the change, the same as an expired one.
### AWS error → HTTP status mapping
stdapi.ai translates upstream AWS error codes into standard HTTP responses with an OpenAI/Anthropic-style error type. Use this table to map a status code back to its likely AWS cause. HTTP status and error type are as returned on OpenAI-compatible routes (`/v1/...`); Anthropic-compatible routes (`/anthropic/...`) diverge on the two footnoted rows.
| HTTP | Error type | AWS error codes | Typical cause |
|-------|------------------------------|---------------------------------------------------------------------------------------------------------|-----------------------------------------------|
| `400` | `invalid_request_error` | `ValidationException`, `BadRequestException` | Unsupported/invalid request parameters |
| `400` | `invalid_request_error` | `AccessDenied` — on the object an `s3://` input named | The caller's own object cannot be read[^4] |
| `401` | `authentication_error` | `UnrecognizedClientException`, `InvalidSignatureException`, `ExpiredTokenException` | stdapi.ai's AWS credentials missing/expired |
| `403` | `permission_error` | `AccessDeniedException` — on a model call an end user's own role signed | That end user is not allowed that model[^3] |
| `404` | `invalid_request_error`[^1] | `ResourceNotFoundException` | Model or resource not available in the region |
| `429` | `rate_limit_error` | `ThrottlingException`, `TooManyRequestsException`, `ServiceQuotaExceededException` | Bedrock quota / throttling |
| `503` | `feature_unavailable` | `AccessDeniedException`, `AccessDenied` — every other denial | IAM task role lacks permission / model access |
| `503` | `server_error`[^2] | `ServiceUnavailableException`, `InternalServerException`, `ServiceFailureException`, `ReadTimeoutError` | Transient AWS-side error — retry |
[^1]: Anthropic-compatible routes return `not_found_error` instead.
[^2]: Anthropic-compatible routes return HTTP `529` with error type `overloaded_error` instead.
[^3]: Only when [per-user cost attribution](operations_cost_management.md#per-user-attribution) is enabled: the call then carries the end user's identity, and AWS evaluated a policy written about them.
[^4]: Only for a bucket declared in [`AWS_S3_ACCEPTED_BUCKETS`](operations_configuration.md#aws-s3-accepted-buckets), which the deployment reads but does not own — so the refused object is the one the request named. The message names that input, and nothing else. A denial on the deployment's own buckets stays `feature_unavailable`.
!!! note "Where to find the detail"
For security, `401`, `403` and `feature_unavailable` responses returned to clients contain only a generic message — the same one whatever is missing, so that the difference between "no permission" and "not configured" is not disclosed. The full diagnostic detail is captured in the server logs under `error_detail` and can be correlated via the `x-request-id` response header (`request-id` on Anthropic-compatible `/anthropic/...` routes) — see [Logging & Monitoring](operations_logging_monitoring.md).
---
## :material-key-variant: Authentication & Identity
??? failure "Bearer token works, but Anthropic SDK requests fail"
The Anthropic SDK uses a different auth header than OpenAI.
- Use `x-api-key: ` (not `Authorization: Bearer`).
- Set the base URL to `https:///anthropic` (not `/v1`).
- See [API Overview → Anthropic-Compatible API](api_overview.md#using-the-anthropic-compatible-api).
??? failure "A valid Amazon Cognito token is rejected with 401"
The response body is always the same opaque `Unauthorized`; the check that failed is in the server's request log (`error_detail`). Work through the checks in order:
- **Wrong app client**: the token's app client must be listed in [`AWS_COGNITO_CLIENT_IDS`](operations_configuration.md#aws-cognito-client-ids).
- **Wrong token**: an identity token is rejected unless [`AWS_COGNITO_ACCEPT_ID_TOKEN`](operations_configuration.md#aws-cognito-accept-id-token) is enabled. Send the **access** token.
- **Wrong issuer**: pools on the Essentials and Plus tiers can issue `https://issuer-cognito-idp..amazonaws.com/...`. Set [`AWS_COGNITO_ISSUER_TYPE`](operations_configuration.md#aws-cognito-issuer-type) to `updated` for those, and confirm the pool ID matches the pool that minted the token.
- **Missing scope**: a token obtained by signing in with a username and password carries only `aws.cognito.signin.user.admin`. Clear [`AWS_COGNITO_REQUIRED_SCOPES`](operations_configuration.md#aws-cognito-required-scopes), or have clients obtain tokens from the pool's OAuth 2.0 token endpoint.
- **Expired token**: tokens are accepted up to one minute past expiry only. Refresh the token, and check the container clock if expiry errors are constant.
- See [Authentication & Security → Amazon Cognito User Pool Tokens](operations_authentication_security.md#amazon-cognito-user-pool-tokens).
??? failure "The server does not start after enabling authentication"
A half-applied credential configuration is refused rather than accepted, so the deployment never runs unauthenticated by accident. The startup log's `error_detail` names the exact rule:
- **Missing allowlist**: [`AWS_COGNITO_CLIENT_IDS`](operations_configuration.md#aws-cognito-client-ids) is required with a pool.
- **Setting without a pool**: any other `AWS_COGNITO_*` variable requires [`AWS_COGNITO_USER_POOL_ID`](operations_configuration.md#aws-cognito-user-pool-id).
- **Mode conflict**: [`AUTHENTICATION_MODE`](operations_configuration.md#authentication-mode) must not demand a method that is unconfigured, nor ignore one that is configured — use `any` to accept both.
- **Empty API key**: the SSM parameter or Secrets Manager secret named by [`API_KEY_SSM_PARAMETER`](operations_configuration.md#api-key-ssm) or [`API_KEY_SECRETSMANAGER_SECRET`](operations_configuration.md#api-key-secretsmanager-secret) exists but holds an empty value — populate it, or unset the setting to run without an API key deliberately.
- **Signing keys unreachable**: the pool's public keys are read at startup over HTTPS. Check the pool ID, and that the task can reach the internet or a suitable endpoint for outbound HTTPS.
??? failure "An MCP client or agent cannot discover how to authenticate"
Discovery is off until it is configured, and a client that finds nothing falls back to asking the user for a key.
- **Nothing published**: `GET /.well-known/oauth-protected-resource` answering `404` means [`OAUTH_RESOURCE_IDENTIFIER`](operations_configuration.md#oauth-resource-identifier) is unset. That is the default, not a fault — set it to turn discovery on. With an [`AWS_COGNITO_USER_POOL_ID`](operations_configuration.md#aws-cognito-user-pool-id) configured it is the only setting needed; otherwise set [`OAUTH_AUTHORIZATION_SERVERS`](operations_configuration.md#oauth-authorization-servers) too.
- **Nothing in the challenge**: a `401` carrying a bare `WWW-Authenticate: Bearer` means the same thing. Once configured it also carries `resource_metadata="…"` and, with [`OAUTH_SCOPES_SUPPORTED`](operations_configuration.md#oauth-scopes-supported), `scope="…"`.
- **Browser-hosted client**: a client running in a page reads the document cross-origin and needs its origin in [`CORS_ALLOW_ORIGINS`](operations_configuration.md#cors-allow-origins).
- **The client asks for `/.well-known/openid-configuration`**: stdapi.ai is a resource server and deliberately does not serve it; the client should follow `authorization_servers` to the issuer's own document.
- See [Authentication & Security → Authentication Discovery for Agents](operations_authentication_security.md#authentication-discovery-for-agents).
??? failure "A client reports the protected resource does not match the expected URL"
Clients compare the published `resource` against the URL they dialled character by character, so any difference aborts the flow.
- **Scheme**: set [`OAUTH_RESOURCE_IDENTIFIER`](operations_configuration.md#oauth-resource-identifier) to `https://…` when clients reach the deployment over TLS, even though the container itself listens on plain HTTP behind the load balancer.
- **Host**: it must be the public hostname clients use, not the internal service or task address.
- **Port**: an explicit `:443` on the client's URL does not match an identifier without one, and vice versa. Drop the default port on both sides.
- **Path**: the identifier is an origin. Do not append `/mcp`, `/v1`, or a trailing slash — one document at the root already covers every surface of the deployment.
??? failure "The server does not start after configuring authentication discovery"
The three settings describe one document, so an incomplete set is refused rather than published half-formed. The startup log's `error_detail` names the rule:
- **No authorization server**: [`OAUTH_AUTHORIZATION_SERVERS`](operations_configuration.md#oauth-authorization-servers) is required with [`OAUTH_RESOURCE_IDENTIFIER`](operations_configuration.md#oauth-resource-identifier) unless [`AWS_COGNITO_USER_POOL_ID`](operations_configuration.md#aws-cognito-user-pool-id) is set, which supplies the issuer; a document naming none leaves a client unable to obtain a token.
- **Issuer contradicting the pool**: with a user pool configured, the published issuers must include the pool's own — otherwise clients are sent to an authorization server whose tokens every request refuses. Leave [`OAUTH_AUTHORIZATION_SERVERS`](operations_configuration.md#oauth-authorization-servers) empty to publish exactly the pool issuer, or list it alongside the others.
- **Setting without an identifier**: the authorization servers and the scopes describe a document that is not published without [`OAUTH_RESOURCE_IDENTIFIER`](operations_configuration.md#oauth-resource-identifier).
- **Malformed value**: the identifier is an origin with no path or query, each issuer is an `https` URL with no query or fragment, and a scope carries no space or quote.
??? failure "OIDC/Cognito redirect loop or 401 from the ALB"
Authentication is enforced by the ALB listener, not stdapi.ai.
- Verify the OIDC issuer URL, client ID, client secret, and redirect URI in the ALB listener rule.
- For Cognito, confirm the app client is configured as a "confidential" client with a client secret.
- See [Authentication & Security → via Application Load Balancer (ALB)](operations_authentication_security.md#via-application-load-balancer-alb).
---
## :material-arrow-right: Next Steps
- :material-cog: [**Configuration Reference**](operations_configuration.md) — Every environment variable
- :material-server-network: [**Advanced Deployment**](operations_deploy_advanced.md) — VPC integration, manual ECS, multi-region
- :material-lock: [**Authentication & Security**](operations_authentication_security.md) — API keys, OIDC/Cognito, IAM
- :material-terraform: [**Terraform Module Docs**](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) — All module inputs and outputs
- :material-github: [**GitHub Issues**](https://github.com/stdapi-ai/stdapi.ai/issues) — Report a bug or ask a question
- :material-email-outline: [**Contact**](contact.md) — Reach the team directly
---
## Releases & Roadmap
Source: https://stdapi.ai/md/roadmap.md
# :material-timeline: Releases & Roadmap
**stdapi.ai is under active development** with regular feature releases.
## :material-tag-multiple: Recent Releases
See [Release History below](#release-history) for the full changelog of all releases.
**Latest: v1.16.1** – A maintenance update to v1.16.0, which added four new API surfaces — conversations, batches, vector stores and realtime speech — plus retrieval a model calls for itself, a vector store served from your own Amazon Bedrock knowledge base, long and streamed text-to-speech, live transcription, per-caller authentication, and end-user cost attribution on the AWS bill. See the [full release notes](#v1160-conversations-batches-vector-stores-realtime-speech-per-user-identity-with-v1161-maintenance-update) below.
---
## :material-rocket-launch: Roadmap (Tracked on GitHub)
Pending features and current deployment state are tracked on the [GitHub Project](https://github.com/orgs/stdapi-ai/projects/1).
---
## :material-history: Release History
### v1.16.0 – Conversations, Batches, Vector Stores, Realtime Speech & Per-User Identity (with v1.16.1 maintenance update)
This release adds four API surfaces and finishes the speech story. **New APIs**: [**Conversations**](api_openai_conversations.md) keep a thread server-side, so a client continues it by id instead of resending the history; the OpenAI [**Batch API**](api_openai_batches.md) and Anthropic [**Message Batches API**](api_anthropic_batches.md) run large request sets asynchronously at the discounted batch price; [**Vector Stores**](api_openai_vector_stores.md) index and search files by meaning — or address [a knowledge base you already run](api_openai_vector_stores.md#knowledge-base-stores) — with a model reaching either kind for itself through [`file_search`](api_openai_responses.md#file-search); and the [**Realtime API**](api_openai_realtime.md) holds a spoken conversation over one WebSocket. **Speech**: 100,000-character [synthesis](api_openai_audio_speech.md#long-input) spoken as it is produced, [live transcription](api_openai_audio_transcriptions.md#streaming) needing no bucket, and [**Amazon Nova Sonic**](api_openai_audio_transcriptions.md#amazon-nova-sonic) as the lowest-cost speech-to-text backend here. **Identity per caller**: [Amazon Cognito tokens](operations_configuration.md#cognito-authentication) alongside or instead of the API key, [published discovery](operations_configuration.md#oauth-discovery) so an agent authenticates itself, and [per-user cost attribution](operations_cost_management.md#per-user-attribution) reporting each end user's spend from the AWS invoice rather than an estimate.
!!! warning "New Required IAM Permissions"
v1.16.0 adds one action every deployment needs, a handful that belong to statements you may already grant, and one statement per optional feature. See [IAM Permissions](operations_iam_permissions.md) for the policies in full.
Enough of them together that **they no longer fit one policy**: IAM caps a customer managed policy at 6,144 characters, and a deployment enabling most of these exceeds it. Attach several policies to the role rather than widening actions to save room — the Terraform module now ships two, one for Amazon Bedrock and one for the supporting services, and does that for you.
**Required on upgrade, whatever the deployment does:**
- **`bedrock:InvokeModelWithBidirectionalStream`** — serves every model invoked over a two-way connection — the [Realtime API](api_openai_realtime.md), and Amazon Nova Sonic transcription and translation. It belongs to the [core Bedrock policy](operations_iam_permissions.md#bedrock-iam); no route-specific action exists for any of them.
**Add to a statement you already grant, if the deployment uses that feature:**
- **`bedrock:UpdateSession`**, on the [session storage statement](operations_iam_permissions.md#bedrock-session-storage-optional) — the conversation metadata update (`POST /v1/conversations/{id}`) and nothing else. The rest of the Conversations API uses the actions stored responses already require.
- **`bedrock-mantle:CountTokens`**, on the [Bedrock Mantle statement](operations_iam_permissions.md#bedrock-mantle-iam) — counts the tokens of a Mantle-served model on `/anthropic/v1/messages/count_tokens`, since Amazon Bedrock's own `CountTokens` takes Anthropic models only. Needed by any deployment serving Mantle models, which is the default.
- **`transcribe:StartStreamTranscription`**, on the [speech-to-text statement](operations_iam_permissions.md#speech-to-text-optional) — serves [`stream=true`](api_openai_audio_transcriptions.md#streaming) on `/v1/audio/transcriptions`. Add it with the upgrade: without it a streamed request that names its language answers `503` `feature_unavailable`, and the server log names the permission. It stages nothing, so a deployment with no bucket at all grants this one alone.
- **`polly:StartSpeechSynthesisStream`, `polly:StartSpeechSynthesisTask` and `polly:GetSpeechSynthesisTask`**, plus `s3:PutObject`, `s3:GetObject` and `s3:DeleteObject` on each bucket serving an Amazon Polly Region, on the [text-to-speech statement](operations_iam_permissions.md#text-to-speech-optional) — they serve [input above 3,000 characters](api_openai_audio_speech.md#long-input) and nothing else. With a bucket configured and these missing, long requests are accepted and then fail on the permission: grant the whole set, or leave the bucket unconfigured and keep the 3,000-character answer.
- **`translate:ListLanguages`**, on the [translation statement](operations_iam_permissions.md#text-translation-optional) — read once at startup so an unsupported language pair is refused before the audio is transcribed. Genuinely optional: without it the check stays off and translation still works, reporting the unsupported pair once the translation call itself fails.
**New statements, one per optional feature:**
- [**Vector stores**](operations_iam_permissions.md#vector-stores-optional) — `s3vectors:CreateIndex`, `DeleteIndex`, `PutVectors`, `GetVectors`, `QueryVectors` and `DeleteVectors`, scoped to your vector bucket and its indexes. No bucket-level create or delete is granted: the gateway creates and deletes the indexes inside the bucket, never the bucket.
- [**Knowledge base vector stores**](operations_iam_permissions.md#knowledge-base-vector-stores) — `bedrock:GetKnowledgeBase`, `Retrieve`, `ListDataSources`, `IngestKnowledgeBaseDocuments`, `ListKnowledgeBaseDocuments`, `GetKnowledgeBaseDocuments` and `DeleteKnowledgeBaseDocuments`, one statement per allowlisted knowledge base ARN. `bedrock:ListKnowledgeBases` is deliberately **not** granted and not needed — the server only ever addresses the identifiers it was given.
- [**Batch inference**](operations_iam_permissions.md#batch-inference) — `bedrock:CreateModelInvocationJob`, `GetModelInvocationJob` and `StopModelInvocationJob` on the server's role, plus `iam:PassRole` conditioned on `bedrock.amazonaws.com`. The service role Amazon Bedrock assumes carries its own policy: `s3:GetObject`, `s3:PutObject` and `s3:ListBucket` on the batch prefix, and `bedrock:InvokeModel` on the models you batch.
- [**Per-user cost attribution**](operations_iam_permissions.md#per-user-cost-attribution) — `sts:AssumeRole` and `sts:TagSession` on the server's role *and* in the end user role's trust policy (both actions: without `TagSession`, every tagged session is denied), and `bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream` and `bedrock:ApplyGuardrail` on the end user role itself, since AWS authorizes those against the caller of the invocation.
- [**Web search**](operations_iam_permissions.md#web-search-iam) — `bedrock-websearch:InvokeSearch` and `bedrock-websearch:InvokeFetch`, plus `bedrock-websearch:ExternalWebAccess` only where a request may reach the open internet. Leaving that last one out is what keeps every search inside the AWS boundary. A missing web-search permission produces no error and no server log entry: the model answers without having searched, so check these before suspecting the model.
- [**Transcription output encryption**](operations_iam_permissions.md#speech-to-text-optional) — `kms:GenerateDataKey` and `kms:Decrypt` on the key named by [`AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-transcribe-output-encryption-key-arn), in the key policy as well as on the role.
- [**Durable vector store indexing**](operations_iam_permissions.md#durable-vector-store-indexing) — `sqs:SendMessage`, `sqs:ReceiveMessage`, `sqs:DeleteMessage`, `sqs:ChangeMessageVisibility` and `sqs:GetQueueAttributes`, on the single queue named by [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url) and never on `*`. Needed only if you configure that queue; leave it unset and nothing here applies. No queue is ever created, deleted or reconfigured, so none of those actions is granted.
!!! note "Five features stay inert until you create the resource they need"
Nothing in this release is breaking — but these five answer `503`, or stay off, until the resource exists in your own account:
- **Batches** — an IAM service role Amazon Bedrock assumes to read the requests and write the results ([`AWS_BEDROCK_BATCH_ROLE_ARN`](operations_configuration.md#aws-bedrock-batch-role-arn)), plus the bucket it reads and writes.
- **Vector stores** — an [Amazon S3 vector bucket](operations_configuration.md#aws-s3-vectors-bucket) you create yourself, and the [Region](operations_configuration.md#aws-s3-vectors-region) it lives in.
- **Knowledge base vector stores** — an allowlist of the knowledge bases this deployment may address ([`AWS_BEDROCK_KNOWLEDGE_BASE_IDS`](operations_configuration.md#aws-bedrock-knowledge-base-ids)), empty by default. One that is not on it answers exactly as a store that does not exist, so the setting cannot be probed for what a deployment holds.
- **Cognito authentication** — a user pool and its app clients ([`AWS_COGNITO_USER_POOL_ID`](operations_configuration.md#aws-cognito-user-pool-id)); until then the API key remains the only method, exactly as before.
- **Per-user cost attribution** — a role for the end user sessions ([`AWS_BEDROCK_USER_ROLE_ARN`](operations_configuration.md#aws-bedrock-user-role-arn)); off by default, and every call keeps being billed to the deployment's own identity until it is set.
Conversations, the Realtime API and streamed transcription need no new resource. Long speech input needs a [bucket for the serving region](operations_configuration.md#aws-s3-regional-buckets), which is the same one the rest of the gateway already uses — except on generative voices, which speak up to 20,000 characters without one.
!!! warning "Behavior Changes"
Review these before upgrading — they may change what existing clients or dashboards observe:
- **A missing deployment permission is no longer reported as the caller's.** An `AccessDeniedException` on the gateway's own AWS calls reached clients as `403 permission_error` — which every OpenAI and Anthropic SDK reads as *their* key being refused. Every route now answers `503` `feature_unavailable`, with the server log naming the operation, model and permission. Clients matching `403` for a backend permission error should match `503`/`feature_unavailable` instead; a `403` now means only that [per-user attribution](operations_cost_management.md#per-user-attribution) is on and *that end user's* role was denied.
- **Built-in web search now appears in usage and cost reporting.** Queries were recorded as nothing at all, so a measured turn under-reported its cost by 58%. Nothing AWS charges changed; what the gateway reports does. Web access is also an operator setting now ([`AWS_BEDROCK_EXTERNAL_WEB_ACCESS`](operations_configuration.md#bedrock-external-web-access)), defaulting to the previous behaviour.
- **A request that would be answered without what it asked for is refused.** A `/v1/responses` `web_search` restricting its sources (`filters.allowed_domains`, `user_location`) was accepted and dropped, so answers came back sourced from domains the caller had excluded. Now a `400` on models that cannot serve it; [Bedrock Mantle](features.md#bedrock-mantle-models) models receive the options unchanged. The same rule governs [file search](api_openai_responses.md#file-search) filters and score thresholds.
- **Two output-shaping hints that returned `400` now succeed.** `prediction` and `verbosity` on chat completions are accepted and dropped, as the Responses surface already did; `truncation="disabled"` is likewise accepted, while `truncation="auto"` is still refused.
- **`/v1/responses` forwards undeclared request fields to the model**, as chat completions and messages already did, so the backend may refuse one it does not recognise. Conversely, client-side control fields no provider treats as parameters (LiteLLM's `drop_params` among them) are dropped rather than forwarded. Both are governed by [`EXTRA_MODEL_PARAMS_DENYLIST`](operations_configuration.md#extra-model-params-denylist) and [`EXTRA_MODEL_PARAMS_DROP_ALL`](operations_configuration.md#extra-model-params-drop-all).
- **Attachments are measured against what the model actually accepts.** The old guard compared raw bytes where the backend enforces base64 length, so it was ~33% too permissive. Oversized attachments are now staged and referenced where the model reads from storage, or refused with `413` naming the size it accepts. Smaller attachments are unaffected — see [Attachment Size](features.md#attachment-size).
- **The server's own connections follow the proxy environment.** `HTTPS_PROXY`, `HTTP_PROXY` and `NO_PROXY` were honoured by the AWS SDK and ignored by everything else, so a proxied deployment saw no [Bedrock Mantle](features.md#bedrock-mantle-models) models. Two connections deliberately still bypass it: container metadata, and the fetch of a caller-supplied URL, where a proxy would defeat address validation. See [proxied deployments](operations_deploy_advanced.md#proxied-deployments).
- **A declared upload checksum is now verified.** The value was stored and never looked at, so a corrupted upload completed like a clean one. It covers the file's contents, **not** the storage layer's multipart identifier — declaring the latter is now refused.
- **An unknown model name answers with a sentence, not the catalogue.** The `404` body carried every served identifier, roughly 2,500 characters. Clients that parsed it for a model list should call [`/v1/models`](api_openai_models.md).
- **Bedrock Mantle is only probed in the Regions that serve it**, so a deployment listing others no longer warns at every start. An explicit [`AWS_BEDROCK_MANTLE_REGIONS`](operations_configuration.md#bedrock-mantle-regions) list is still used exactly as given.
- **The container health probe's command changed.** Deployments that re-declare the probe instead of running the image's own — an [ECS task definition](operations_deploy_advanced.md#ecs-task-definition-example) among them — should take the command from the image.
#### :material-api: New APIs
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/conversations`](api_openai_conversations.md) – create, retrieve, update and delete a conversation, list and manage its items, and continue it from the Responses API with the `conversation` parameter | {: style="height:20px;width:20px"} Amazon Bedrock - session management |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/vector_stores`](api_openai_vector_stores.md) – attach files, follow the indexing as it progresses, then search by meaning with attribute filters and per-passage scores | {: style="height:20px;width:20px"} Amazon S3 Vectors, {: style="height:20px;width:20px"} Amazon Bedrock - embedding models |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/vector_stores`](api_openai_vector_stores.md#knowledge-base-stores) – address an Amazon Bedrock knowledge base you already run as a vector store, Bedrock managed or customer-managed: search it, attach, list, read and delete documents. Allowlisted per knowledge base, never created or deleted here | {: style="height:20px;width:20px"} Amazon Bedrock - Knowledge Bases |
| {: style="height:20px;width:20px"} **OpenAI** | [`file_search` on `/v1/responses`](api_openai_responses.md#file-search) – a chat model answers from the stores you name, reporting the searches it ran and citing a `file_citation` per file it drew on | {: style="height:20px;width:20px"} Amazon S3 Vectors, {: style="height:20px;width:20px"} Amazon Bedrock - Knowledge Bases |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/batches`](api_openai_batches.md) – run a JSONL file of chat completion or embedding requests asynchronously at the batch price: submit, poll, cancel, read the result files | {: style="height:20px;width:20px"} Amazon Bedrock - batch inference |
| {: style="height:20px;width:20px"} **Anthropic** | [`/anthropic/v1/messages/batches`](api_anthropic_batches.md) – the same asynchronous, batch-priced run for the Messages API, results streamed back as JSONL; each request may name its own model, up to eight per batch | {: style="height:20px;width:20px"} Amazon Bedrock - batch inference |
| {: style="height:20px;width:20px"} **OpenAI** | [`WS /v1/realtime`](api_openai_realtime.md) – a live speech-to-speech session over one WebSocket, with a transcript of both sides, server-side turn detection or manual turns, barge-in, and G.711 for telephony | {: style="height:20px;width:20px"} Amazon Bedrock - Amazon Nova Sonic |
| {: style="height:20px;width:20px"} **OpenAI** | [`POST /v1/realtime/client_secrets`](api_openai_realtime.md#ephemeral-client-secrets) – mint a short-lived, browser-safe credential carrying a session configuration; signed and stateless, so any instance verifies one minted by any other | {: style="height:20px;width:20px"} Amazon Bedrock - Amazon Nova Sonic |
!!! note "Limits worth knowing before building on these"
**Realtime**: a session lasts at most 8 minutes and calls no tools, and a spoken answer is guardrail-checked once complete, so a blocked one may already have been partly heard ([coverage](api_openai_realtime.md#guardrail-coverage)). **WebRTC and SIP are not served** — put [LiveKit Agents or Pipecat](api_openai_realtime.md#transports) in front for a browser media path or a phone line. The [compatibility table](api_openai_realtime.md#feature-compatibility) lists every event the session does not emit.
**Knowledge-base stores** address a knowledge base that already exists and refuse, naming why, anything that would reshape it — creating, deleting, renaming, expiry, chunking strategy, attribute rewrites and the file-batch routes. Attaching needs a **custom** data source. Retrieval scores are reported as the backend states them rather than rescaled into similarities, and unknown values are reported unknown rather than invented. See [Knowledge Base Stores](api_openai_vector_stores.md#knowledge-base-stores).
#### :material-microphone: Speech & Audio
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/audio/speech`](api_openai_audio_speech.md#long-input) – up to 100,000 billed characters per request, 24× the upstream 4,096, with no API change and no new request field | {: style="height:20px;width:20px"} Amazon Polly |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/audio/speech`](api_openai_audio_speech.md#long-input) – long input is spoken as it is synthesized instead of after a whole job finishes; generative voices reach 20,000 characters with no bucket at all, and each request takes whichever path can serve it, so long input is no longer tied to one voice | {: style="height:20px;width:20px"} Amazon Polly |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/audio/transcriptions`](api_openai_audio_transcriptions.md#amazon-nova-sonic) – naming Amazon Nova Sonic transcribes at the lowest cost available here, streamed as it is recognized; `json` and `text` only, up to 10 minutes, no timestamps. No existing request is re-routed | {: style="height:20px;width:20px"} Amazon Bedrock - Amazon Nova Sonic |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/audio/translations`](api_openai_audio_translations.md) – Amazon Nova Sonic translates speech to English itself, in one request | {: style="height:20px;width:20px"} Amazon Bedrock - Amazon Nova Sonic |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/audio/transcriptions`](api_openai_audio_transcriptions.md#streaming) – `stream=true` returns each phrase as it is recognized, whenever the request names the language to expect; needs no bucket. `gpt-live-transcribe` is now an alias, and requests naming no language are unchanged unless [`AWS_TRANSCRIBE_STREAM_LANGUAGES`](operations_configuration.md#aws-transcribe-stream-languages) says which to expect | {: style="height:20px;width:20px"} Amazon Transcribe |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/audio/transcriptions`](api_openai_audio_transcriptions.md) – per-language custom vocabularies and language models, so a request identifying between several languages can apply the right resources to each one instead of being refused. Accepted only where the backend would use them: alongside a single fixed language, where they would apply to nothing, they are still refused | {: style="height:20px;width:20px"} Amazon Transcribe |
| **stdapi.ai** | [`AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-transcribe-output-encryption-key-arn) – encrypt a transcription's output with a key you name rather than the bucket's own. The job's request identifiers travel as the encryption context, so a key policy can be scoped to this workload instead of to the whole bucket | {: style="height:20px;width:20px"} Amazon Transcribe, AWS KMS |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/audio/translations`](api_openai_audio_translations.md) – the supported language pairs are read once at startup and checked before the call, so a pair that cannot be served is named as the request problem it is instead of surfacing as a failure after the audio was transcribed. The permission that reads them is optional: without it the check stays off and everything else works | {: style="height:20px;width:20px"} AWS Translate |
#### :material-account-key: Identity & Cost Attribution
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| **stdapi.ai** | [Amazon Cognito user pool tokens](operations_configuration.md#cognito-authentication) – accept access tokens instead of, or alongside, the API key, so each caller reaches the API with their own credential; validated in-process against the pool's published keys, with no AWS call on the request path | {: style="height:20px;width:20px"} Amazon Cognito |
| **stdapi.ai** | [`AUTHENTICATION_MODE`](operations_configuration.md#authentication-mode) – assert the posture rather than infer it: the server refuses to start when the selected method is not configured, or when a configured method would be silently ignored | {: style="height:20px;width:20px"} Amazon Cognito |
| **stdapi.ai** | [Authentication discovery for agents](operations_configuration.md#oauth-discovery) – an OAuth 2.0 protected resource metadata document, pointed at by every unauthorized response, so an MCP client finds the authorization server and the scope it needs without being configured for this deployment; published only once an authorization server is declared | {: style="height:20px;width:20px"} Amazon Cognito, or any OAuth 2.0 authorization server |
| **stdapi.ai** | [Per-user cost attribution](operations_cost_management.md#per-user-attribution) – model calls issued under a short-lived role session tagged with the caller, so AWS reports each end user's spend in Cost Explorer and the Cost and Usage Report, from the invoice rather than an estimate. Off by default; a deployment can also [require](operations_configuration.md#aws-bedrock-user-role-require-identity) every call to name its end user rather than bill it to the deployment | {: style="height:20px;width:20px"} Amazon Bedrock, AWS STS |
| **stdapi.ai** | [Vector store cost reporting](operations_cost_management.md#vector-stores) – a search against a Bedrock-managed knowledge base is recorded and priced like every other billed unit; what cannot be accounted for is stated rather than approximated | {: style="height:20px;width:20px"} Amazon Bedrock - Knowledge Bases |
#### Platform Features
| Feature | Description |
|----------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Aliases that carry configuration](operations_configuration.md#model-aliases-configuration) | A `MODEL_ALIASES` entry may map a public name to the target model *plus* the service tier, guardrail, metadata and extra parameters applied to requests naming it, so one model is published under several names with different policies. The plain-string form is unchanged, and a malformed alias stops startup naming itself rather than failing once per request |
| [Attachment size policy](features.md#attachment-size) | On the multimodal routes served by Amazon Bedrock, an attachment is measured before the request is built and travels inline or by reference according to the limits each model class declares. Staging is per model and per media kind: of the families measured, only the Amazon Nova families and TwelveLabs Pegasus accept a reference |
| Ephemeral secret signing key | [`REALTIME_CLIENT_SECRET_KEY`](operations_configuration.md#realtime-client-secret-key) signs the Realtime API's client secrets. A deployment with an API key already shares one and needs nothing; one running with no API key at all should set it, or a secret minted by one instance fails to verify on another |
| Dual-stack container listener | The image's bind address moved out of its command into `GRANIAN_HOST`, so a deployment that needs a dual-stack socket — an ECS service whose discovery record includes an AAAA record, for instance — sets one variable instead of replacing the whole command. The IPv4-only default is unchanged |
| Faster container health probe | The probe ships as a module of the application itself, byte-compiled with the rest of the package and covered by the linters and the test suite; it speaks HTTP over a socket rather than pulling in 123 modules per probe, cutting roughly 250 ms of import work per run in the community image and halving its peak memory |
| OpenAI Daybreak models | Daybreak Red (GPT-5.6 Cyber) and Daybreak Blue (GPT-5.6 Sol) are served and priced with the rest of the GPT-5.6 family, image input included. Both answer on the Responses API through Bedrock's next-generation inference endpoint, in US East (Ohio) only, and both are gated on enrollment with OpenAI's Daybreak programme — an account without it does not see them in the catalogue at all |
| Capability discovery | The [model catalogue](api_search_models.md) advertises what this release added — speech to speech, its transcription and translation, the search surfaces, and whether a model can be used with the Batch API — filterable over HTTP and through the same tool an agent reads before it calls anything. Web search is credited to every model that provides it, not only to the family the last release added it for |
| [Durable vector store indexing](api_openai_vector_stores.md#durable-indexing) | [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url) hands indexing to an Amazon SQS queue you create, so a file keeps being indexed — and finishes — when the server that accepted it is replaced. Off by default; needs a standard queue with a dead-letter queue and the [durable indexing permissions](operations_iam_permissions.md#durable-vector-store-indexing) ([resilience](operations_resilience.md#vector-store-indexing)) |
| End-to-end client coverage | The suite driving complete, unmodified third-party clients against a live gateway gains five: **LiteLLM**, **Docling Serve**'s vision pipeline, the **OpenAI Agents SDK** (realtime voice, conversations, web search, vector-store retrieval), and **LiveKit Agents** and **Pipecat** running the exact [WebRTC and telephony configurations](api_openai_realtime.md#transports) this documentation prints. Deployment guides follow for [LobeHub](use_cases_lobehub.md) and [RAGFlow](use_cases_ragflow.md) |
#### :material-bug: Fixes { #fixes-3 }
- **Vector store durability**: an attached file whose indexing was interrupted is now reported `failed` with a `last_error` instead of sitting `in_progress` for ever; a detached file leaves every listing and search immediately and is gone only once its passages are, so a server replaced mid-delete no longer leaves a deleted document answering searches; indexing is bounded for the whole server rather than per request, so memory and embedding quota no longer scale with the number of callers; and deployments that would rather the work finished than reported can [hand indexing to a queue](api_openai_vector_stores.md#durable-indexing)
- **Work a request left running is finished before the server stops**: a deployment, scale-in or Spot interruption used to drop temporary file cleanups, vector store indexing and live audio session releases with nothing in the logs. Shutdown now waits under [`SHUTDOWN_DRAIN_TIMEOUT`](operations_configuration.md#shutdown-drain-timeout) (10 seconds by default), settles whatever the deadline leaves, and counts it in the `stop` event. Raise it together with your container runtime's kill delay, never one alone
- **Streamed responses run the work they scheduled**: the drain was attached before the body produced a byte, so three leaks followed — a vector store searched only through streamed answers could expire mid-query, an expired store left a paid index behind, and a streamed transcription falling back to a job left its audio, transcript and job record behind on every request
- **Realtime speaks the released vocabulary, not the beta one**: item events were unparsable, the caller's transcript event was dropped for a missing field, and the item lifecycle clients wait on was never emitted. Barge-in did not work at all — the session refused truncation precisely while an answer was playing, the only moment it is ever sent. Truncate, retrieve and delete now work against the tracked conversation, a written turn is answered instead of timing out, and every answer reports the six response fields upstream always sends
- **Batch API**: listing a batch neither settled nor published it, so a client that only ever listed never had its usage recorded; every validation failure at submission was reported as an unsupported model, quota and role failures included; and the batch record was written only after the jobs started, leaving billable work running with nothing on disk to stop it
- **Knowledge-base stores answer in this API's own words**: attaching to a managed knowledge base always failed, listing its files always failed, deleted documents never left a listing, and internal bookkeeping reached the caller as attributes. A file refused because the corpus is maintained elsewhere is now told so in the store's terms, and a refused file is explained by the store that refused it rather than by one fixed sentence describing limits the caller never met
- **Cost reporting matches what AWS charges**: GPT-5.6 Luna was reported at five times its real cost and Terra a quarter over, since AWS repriced them on the model's own page rather than in the live price catalogue; a [Global cross-Region](operations_cost_management.md#routing-tier-pricing) call — the only way Luna, Sol and Terra are ever served — was costed at the In-Region rate, about 10% over; and moderation usage naming an alias matched no price at all. Every source page is now re-checked weekly
- **Reasoning and web search reach the models that serve them**: Amazon Nova 2 and DeepSeek V3 refused the token budget the Anthropic dialect requires, leaving [extended thinking](api_anthropic_messages.md#extended-thinking) unreachable on that route while the identical ask worked elsewhere; and a `web_search` sent to a GPT-5.6 model resolved to its non-Mantle twin travelled as an ordinary function tool, so no search ran and nothing said so — now a `400` naming both ways to route the model to the endpoint that serves it
- **The Messages surface reports what an answer cost and why it stopped**: refusals carry the policy category, the reasoning-token breakdown is reported, and service tier and per-TTL cache-creation counts are populated — most visibly on a batch, which claimed no tier while being billed as one
- **Audio, embeddings and attachments are bounded correctly**: inline audio was measured against raw bytes where the backend enforces the encoded length, so files between ~18.75 MB and 25 MB passed and were refused downstream; long text-to-speech now answers with the length a bucket-less deployment can honour; and models that embed one input per call no longer open a connection per chunk
- **The interactive documentation pages render with no outbound access**: [`/docs` and `/redoc`](operations_configuration.md#enable-docs) pulled the icon, Swagger UI, ReDoc and a web font from three third parties — blank pages in an air-gapped VPC, and elsewhere a report to those hosts of who was reading this API and when, running whatever a floating major version tag resolved to that day. Both are now served whole from the image, pinned to exact releases verified by SHA-256 at build time, with upstream licences beside them
- **MCP tools return what they produce**: every route answering with bytes was published as a tool an agent could call and then could not use. An image now arrives as an image and audio as audio, anything the protocol cannot carry arrives as a reference rather than failing, and the 4 MiB body cap that blocked image edits now follows [`MAX_INPUT_FILE_SIZE`](operations_configuration.md#max-input-file-size)
- **Addresses and listings are the ones this deployment serves**: a custom [route prefix](operations_configuration.md#openai-routes-prefix) still quoted default paths to [`search_models`](api_search_models.md) and to video job polling, neither recoverable client-side; and a file's `created_at` and its place in a listing came from two different clocks, so a [multipart upload](api_openai_files.md#uploads-api) sat among older files reporting a later time. The [Anthropic listing](api_anthropic_files.md#list-files) also answered oldest first, hiding every recent file, and now runs newest first as upstream does
- **Diagnostics name their cause**: an unreachable Region rendered six different conditions as one identical sentence, and the slow startup beside it was the container metadata lookup retrying, unreported; a request abandoned mid-flight left its OpenTelemetry trace current, so later work was recorded under a closed request's trace id; and behind a proxy, `client_ip` recorded the load balancer whatever [`PROXY_TRUSTED_HOSTS`](operations_configuration.md#proxy-trusted-hosts) allowed
- **The API describes itself, not the service behind it**: a synthesis limit credited to the service enforcing it, prices credited to their catalogue and a moderation route naming the engine underneath all shipped in the OpenAPI document and in the tool descriptions agents read before calling
#### Fixes & Maintenance (v1.16.1)
- Update `cryptography` to 50.0.1, rebuilt against OpenSSL 4.0.2
### v1.15.0 – Reliability, Performance & Feature Completeness
This release focuses on making the whole gateway better rather than just bigger. **Reliability and quality**: the largest correctness pass to date — three successive deep audits plus an independent full-branch review closed hundreds of fidelity gaps across all three API dialects, every fix pinned by tests and the whole surface validated by **real, unmodified client applications**. **Performance**: hot paths now run in compiled native code and independent work in parallel, [measurably cutting the gateway's processing overhead](features.md#performance). **Feature completeness**: existing capabilities are rounded out end to end — [**explicit prompt caching**](api_openai_chat_completions.md#prompt-caching), operator [**reasoning controls**](operations_configuration.md#chat-completions-reasoning-field), the Responses API [`prompt` parameter](operations_configuration.md#bedrock-allow-prompt-arn) from **Amazon Bedrock Prompt Management**, native mid-conversation system messages on Claude 4.8+, richer speech and transcription (Polly speech marks, Transcribe/Translate extras, generic Converse speech-to-text), Cohere `embedding_types` and Rerank v1 structured documents, guardrail enforcement on every route, and an inline guardrail-checks moderation backend.
!!! warning "Behavior Changes"
Review these before upgrading — they may change what existing clients observe:
- **Unsupported parameters are accepted and ignored, not rejected.** Parameters the AWS backends cannot honour (e.g. `known_speaker_*`, `partial_images`, unsupported image `quality`/`style`, programmatic tool calling) now behave like they do on OpenAI: the request succeeds, the parameter is dropped, and a warning is recorded in the request log. Requests that returned `400` on v1.14 may now succeed.
- **Impossible combinations are now clean `400`s instead of silent degradation**: subtitle or diarized formats with `stream=true`, contradictory Amazon Transcribe settings, and web-search filters Nova grounding cannot apply are rejected with actionable messages.
- **Speech output quality**: `wav`/`flac`/`aac` are now encoded from lossless PCM instead of Ogg Vorbis, and the default `pcm` output is resampled to 24 kHz for OpenAI parity (pass an explicit `SampleRate` to keep Polly's native rate). Same formats, different — better — bytes.
- **Error responses no longer expose backend internals.** Server-side (`5xx`) error messages are generic with details kept in the server log, and Anthropic error types now match the official SDK exactly.
- **Comprehend-backed moderation always analyses text as English** — the only language the AWS API accepts at runtime.
- **A configured guardrail now applies to every route.** Embeddings, rerank, images, videos, and the audio routes enforce it through the ApplyGuardrail API ([route coverage](operations_configuration.md#route-coverage)) — requests that silently bypassed the guardrail on v1.14 may now return `400` (code `content_filter`) or masked text, and each check is billed as guardrail text units.
- **SSRF protection covers every non-globally-reachable address.** With [`SSRF_PROTECTION_BLOCK_PRIVATE_NETWORKS`](operations_configuration.md#ssrf-protection-block-private-networks) enabled (the default), a user-supplied URL resolving to shared address space (100.64.0.0/10, used by EKS custom networking and Hybrid Nodes) or another special-purpose range is now rejected with `403`, alongside the RFC 1918 ranges.
- **Usage reporting is additive but richer**: cached-token buckets are folded into `prompt_tokens` with `prompt_tokens_details` on every surface, and the Anthropic API now reports `cache_creation_input_tokens` (it was always `null` on v1.14).
- **Two request-body keys are reserved.** `model_id` and `additional_request_fields` (plus `stop_sequences` on the legacy `/v1/completions`, where `stop` is the parameter to use) collide with the gateway's own request-building parameters: instead of being forwarded to Bedrock as [provider extras](api_openai_chat_completions.md#provider-specific-parameters), they return a `400 invalid_request_error` naming the key.
- **The container health probe now respects `TRUSTED_HOSTS`.** The image's `HEALTHCHECK` requests `/health` with a `Host` header derived from [`TRUSTED_HOSTS`](operations_configuration.md#trusted-hosts) — a correct list keeps the container healthy with no extra entry. Deployments that re-declare the probe, such as an [ECS task definition](operations_deploy_advanced.md#ecs-task-definition-example), should run the image's own command. Note that a load balancer health check still sends the target's IP address as the `Host` and is rejected with `400` when the allow-list is enabled.
#### :material-cached: Explicit Prompt Caching
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | [Explicit cache breakpoints](api_openai_chat_completions.md#prompt-caching) on chat completions and responses, mapped to Bedrock `cachePoint` blocks with the per-request block budget enforced | {: style="height:20px;width:20px"} Amazon Bedrock - Converse API |
| {: style="height:20px;width:20px"} **OpenAI** | `prompt_cache_options` – cache TTL control on chat completions and responses, honoured on models that support it | {: style="height:20px;width:20px"} Amazon Bedrock - Converse API |
Prompt caching itself is not new — this release adds *explicit placement and lifetime control* on the OpenAI dialect, and fixes the caching plumbing that already existed (see [Fixes](#fixes-2)).
#### :material-brain: Reasoning Controls
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| **stdapi.ai** | [`CHAT_COMPLETIONS_REASONING_FIELD`](operations_configuration.md#chat-completions-reasoning-field) – return thinking text under `reasoning_content`, `reasoning`, or suppress it with `none`; applied identically to streamed deltas and final messages | {: style="height:20px;width:20px"} Amazon Bedrock - Converse API & Mantle |
| {: style="height:20px;width:20px"} **OpenAI** | OpenRouter-style `reasoning` request object accepted on chat completions (`effort`, `max_tokens`, `enabled`, `exclude`) | {: style="height:20px;width:20px"} Amazon Bedrock - Converse API & Mantle |
#### :material-api: New API Features
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | Responses API `prompt` parameter – serve prompts stored in Bedrock Prompt Management, with versions and variables (opt-in, [`AWS_BEDROCK_ALLOW_PROMPT_ARN`](operations_configuration.md#bedrock-allow-prompt-arn)) | {: style="height:20px;width:20px"} Amazon Bedrock - Prompt Management |
| {: style="height:20px;width:20px"} **Anthropic** / {: style="height:20px;width:20px"} **OpenAI** | Mid-conversation system messages forwarded natively on Claude 4.8+ and Claude 5 family models instead of being folded into the system prompt | {: style="height:20px;width:20px"} Amazon Bedrock - Converse API & Mantle |
| **stdapi.ai** | Guardrail asynchronous stream processing via the `X-Amzn-Bedrock-GuardrailStreamProcessingMode` request header | {: style="height:20px;width:20px"} Amazon Bedrock - Guardrails |
| **stdapi.ai** | Configured guardrails enforced on every route: embeddings, rerank, images, videos, and audio now apply them via the ApplyGuardrail API ([route coverage](operations_configuration.md#route-coverage)) | {: style="height:20px;width:20px"} Amazon Bedrock - Guardrails |
| {: style="height:20px;width:20px"} **OpenAI** | [Moderations](api_openai_moderations.md) `amazon.bedrock-runtime-guardrail-checks` model – inline guardrail content filter checks with no guardrail resource required, the new default fallback for `omni-moderation-*` in supported regions | {: style="height:20px;width:20px"} Amazon Bedrock - Guardrails |
#### :material-microphone: Speech & Audio
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/audio/speech`](api_openai_audio_speech.md) – Polly `SpeechMarkTypes` for word/sentence/viseme/SSML timing marks | {: style="height:20px;width:20px"} Amazon Polly |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/audio/transcriptions`](api_openai_audio_transcriptions.md) – Amazon Transcribe extra parameters: multi-language identification, custom vocabularies, PII redaction, and more | {: style="height:20px;width:20px"} Amazon Transcribe |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/audio/transcriptions`](api_openai_audio_transcriptions.md) – `gpt-transcribe` context inputs: `keywords` and multi-language `languages`, with detected languages reported in the response | {: style="height:20px;width:20px"} Amazon Transcribe |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/audio/transcriptions`](api_openai_audio_transcriptions.md) & translations – any Converse-capable speech-input Bedrock model transcribes through a generic default (Voxtral rebuilt on the Converse API); uploads outside the accepted formats are transcoded automatically | {: style="height:20px;width:20px"} Amazon Bedrock - Converse API |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/audio/translations`](api_openai_audio_translations.md) – AWS Translate `Formality`, `Profanity`, `Brevity`, and custom terminologies | {: style="height:20px;width:20px"} AWS Translate |
#### :material-vector-combine: Cohere Embed & Rerank
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **Cohere** | [`embedding_types`](api_cohere_embed.md) – quantized (`int8`/`uint8`/`binary`/`ubinary`) and `base64` embeddings on both embed routes; image embedding metadata reported | {: style="height:20px;width:20px"} Amazon Bedrock - embedding models |
| {: style="height:20px;width:20px"} **Cohere** | [Rerank v1](api_cohere_rerank.md) – structured JSON documents with `rank_fields` selection | {: style="height:20px;width:20px"} Amazon Bedrock - Rerank API |
#### Platform Features
| Feature | Description |
|----------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Stateless MCP transport | The [MCP server](operations_configuration.md#mcp-model-context-protocol) can serve `/mcp` without server-side sessions ([`MCP_STATELESS_HTTP`](operations_configuration.md#mcp-stateless-http)), so any replica may answer any request, alongside a `GET /ping` health probe kept out of request logs |
| `Retry-After` on `429` | Throttled responses advertise the region router's computed backoff, so well-behaved clients retry exactly when capacity returns |
| AWS request-ID correlation | Request logs record every AWS API call's request ID (and incoming ALB/CloudFront trace headers), so a gateway request ties directly to CloudTrail and AWS support cases |
| Programmatic tool calling types | The OpenAI SDK's programmatic tool calling type surface parses on every request union, accepted and ignored on models without the capability |
| [Performance](features.md#performance) | Hot paths run in compiled native code and independent work runs in parallel: a 1 MB request costs 30% less CPU, multi-image generations finish in the time of the slowest image, and every optimization is pinned by regression tests |
| MCP context efficiency | Tool schemas hide parameters MCP callers cannot use (streaming modes, token-level tuning, caller identifiers) and tool results return as compact JSON, cutting the tokens each call costs the calling agent; every exposed MCP tool is exercised end to end through a real MCP client in the test suite |
| Slimmer container image | The community image shrinks from 230 MB to 156 MB: unused dependency payloads are removed (language-name data, `cryptography`, `rich`/`typer`, uvicorn extras) and AWS service models are pruned to the services actually used, guarded by a build-time smoke test; the package inventory stays complete for vulnerability scanners (self-built ffmpeg registered, Python package metadata retained) and every redistributed component keeps its [licence and notice files](operations_licensing.md#frequently-asked-questions) |
| Metadata filter for MCP clients | Listing stored chat completions accepts the metadata filter as a single [`metadata={"key": "value"}` JSON object](api_openai_chat_completions.md#stored-chat-completions) as well as the OpenAI SDK's `metadata[key]=value` pairs, so clients that can only send one query parameter per field — MCP tool calls among them — can filter too |
#### :material-robot-happy: Verified with Real Clients
Compatibility claims in this release are backed by a new test tier that runs **complete, unmodified third-party client software against a live gateway** — not just HTTP assertions. Coding agents (Claude Code, Codex, pi, Qwen Code), the n8n workflow platform, Open WebUI, Home Assistant's voice bridge, a Haystack RAG pipeline, and the LangChain and pydantic-ai libraries drive real multi-turn tool-calling, retrieval, and speech sessions across dozens of models and all three API dialects, in isolated sandboxes. Alongside them, every served model is **empirically probed** for the parameters it genuinely honours, with the results recorded and pinned by tests. See [Quality Assurance](features.md#quality-assurance) for the full methodology.
#### :material-bug: Fixes { #fixes-2 }
Three audit passes and an independent full-branch review closed over a hundred fidelity gaps. The user-visible highlights:
- **MCP tool calls match their schemas**: union-typed parameters no longer advertise a contradictory single `type` (which made valid string arguments randomly fail schema validation), and the JSON image edit/variation bodies accept the plain string references the tool schemas advertise
- **Clean errors on log-exempt paths**: a 404 or 405 on paths kept out of request logs (e.g. `/favicon.ico`, auto-requested by every browser visiting `/`) returned a 500 with a traceback instead of the JSON error envelope
- **Prompt caching plumbing**: Anthropic `cache_control` breakpoints land on their marked block; cache reads and writes are counted, priced, and reported consistently in responses, request logs, and `count_tokens`
- **Reasoning**: `reasoning_effort="max"` accepted end to end; thinking text returned by Bedrock Mantle models is surfaced on every API instead of dropped; unsigned reasoning is no longer replayed to models that would reject it
- **Streaming parity**: streamed and non-streamed results are now identical (tool-call indices, same-role message merging, text concatenation); mid-stream errors emit proper error events on every API instead of ending streams silently; redacted thinking and web-search results round-trip exactly as native Anthropic emits them; failed generations return `502` instead of an empty `200`
- **Routing & billing**: a read timeout on an already-sent request is no longer re-invoked in another region (no double billing), on Converse and Mantle alike; region failover now tries each candidate region at most once per request instead of looping back over regions it just marked as throttled, so a single request can no longer escalate a region's quota backoff toward the one-hour ceiling; Bedrock prompt-router usage is billed against the actually-invoked model; the [price card](api_model_pricing.md) reprices a standard-tier row served as a tier fallback at the rate that tier actually bills; `store=true` degrades gracefully with a logged warning in regions without the session API
- **Responses API parity**: the type surface is synchronized with the current OpenAI SDK (tool fields, error codes, tool-call `caller` provenance); Anthropic `count_tokens` counts exactly what generation sends, and error bodies carry the `request_id`
- **Audio & images**: the transcoding pipeline is fully bounded — a stalled or failed encode returns a clean error instead of holding the connection open; multipart forms bind every list field the OpenAI SDK sends; `size="auto"` works on generation, edits, and variations; `zh-TW`/`pt-PT` stay distinct in translation; PII-redacted transcripts are read from the key Amazon Transcribe actually writes; Polly voice auto-selection is deterministic; batch-purpose files apply the documented 30-day default expiry, and an expired file now disappears from [file listings](api_openai_files.md#upload-with-expiry) instead of being listed with an entry that 404s on retrieve
### v1.14.0 – Bedrock Mantle, Video Generation, Cohere APIs, Moderation & Stored Conversations
This release adds enabled-by-default [**Amazon Bedrock Mantle** support](features.md#bedrock-mantle-models) — models served by the Bedrock Mantle endpoint (OpenAI GPT-5.4/5.5/5.6, xAI Grok 4.3, Google Gemma 4, Qwen3, GLM, DeepSeek, MiniMax, Kimi, Nemotron, and more) become available through all four text APIs, with transparent API conversion, native stored conversations, and independent throughput quotas. It also turns stdapi.ai into a three-dialect gateway with the new **Cohere-compatible API** ([Rerank](api_cohere_rerank.md) and [Embed](api_cohere_embed.md)), adds the OpenAI-compatible [**Videos API**](api_openai_videos.md) for asynchronous video generation, [**content moderation**](api_openai_moderations.md) backed by Amazon Bedrock Guardrails or Amazon Comprehend toxicity detection, **stored responses and chat completions** with `store=true`, `previous_response_id` multi-turn continuation, and a full list/retrieve/update/delete lifecycle on Amazon Bedrock session storage, and [**conversation compaction**](api_openai_responses.md#conversation-compaction). The Responses API gains [**extended reasoning**](api_openai_responses.md#extended-reasoning): Bedrock `reasoningContent` now surfaces as native reasoning output items, both non-streaming and streamed, with signatures and redacted payloads round-tripping through an `encrypted_content` envelope. A broader compatibility pass brings request/response parity closer to the OpenAI SDK — hosted and agent tool types (web search, computer use, custom tools) are now accepted and ignored instead of rejected, streams correctly terminate with `response.incomplete`/`response.failed`, cached tokens are counted in `input_tokens`, and citation annotations are emitted with their streaming events — validated end-to-end against the OpenAI Codex CLI as an agent client. Operations gain a [model pricing API](api_model_pricing.md), multi-region failover for every AWS AI service, fault-tolerant startup, real AWS-billed usage and costs in request logs (optionally exported as CloudWatch metrics), and a [security hardening pass](#security-hardening) covering SSRF protection, input validation, and log/error redaction.
!!! warning "New Required IAM Permissions"
v1.14.0 requires two new IAM permissions:
- **`bedrock:Rerank`** — needed for the [Cohere-compatible Rerank API](api_cohere_rerank.md) (`/cohere/v2/rerank`). See [IAM Permissions](operations_configuration.md#bedrock-iam).
- **`bedrock:ListAsyncInvokes`**, plus **`bedrock:ListTagsForResource`** on `arn:aws:bedrock:*:*:async-invoke/*` — needed for `GET /v1/videos` (listing video generation jobs across regions). See [IAM Permissions](operations_configuration.md#bedrock-iam).
Ensure your IAM role or user policy includes both statements before upgrading to v1.14.0.
!!! note "Session storage and Comprehend permissions already covered"
The IAM permissions for [stored responses/chat completions](operations_configuration.md#bedrock-session-storage-optional) (`bedrock:CreateSession` and related session actions) and [Comprehend-based moderation](operations_configuration.md#iam-permissions) (`comprehend:DetectToxicContent`) were already added to the official [stdapi-ai Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) ahead of this release. Deployments using a hand-written policy still need to add those statements if they haven't already. Without the session permissions, `store=true` (previously accepted and ignored) is still ignored — a warning is recorded in the request log instead of failing the request.
#### :material-layers-triple: Amazon Bedrock Mantle
Enabled-by-default support ([`AWS_BEDROCK_MANTLE_ENABLED`](operations_configuration.md#bedrock-mantle-enabled)) for models served by the **Amazon Bedrock Mantle** endpoint — OpenAI GPT-5.4/5.5/5.6 (Sol, Terra, Luna), xAI Grok 4.3, Google Gemma 4, Qwen3, GLM 4.x/5, DeepSeek V3.x, MiniMax M2.x, Kimi K2.5, Nemotron, and more — alongside the classic Bedrock Converse catalog:
- All four text APIs (chat completions, responses, messages, legacy completions) are served for every Mantle model — native passthrough where the model supports the API upstream, transparent conversion otherwise
- Models available on both bedrock-runtime and Mantle are served by bedrock-runtime by default; [`AWS_BEDROCK_MANTLE_PREFERRED_MODELS`](operations_configuration.md#bedrock-mantle-preferred-models) or the opt-in `x-stdapi-service` request header ([`AWS_BEDROCK_MANTLE_SERVICE_HEADER`](operations_configuration.md#bedrock-mantle-service-header)) route them through Mantle instead — e.g. to tap Mantle's independent throughput quotas
- Native Mantle stored conversations on `/v1/responses` (`store`, `previous_response_id`, retrieval and deletion) — 30-day retention, region-local, project-scoped
- Multi-region failover and quota backoff across [`AWS_BEDROCK_MANTLE_REGIONS`](operations_configuration.md#bedrock-mantle-regions), matching classic Bedrock region routing
- Authentication via short-term bearer tokens derived from the server's AWS credential chain — no static secrets
- Usage recorded and priced at bedrock-mantle rates, including cached tokens and service tiers
- Optional Bedrock Project/Workspace attribution for cost tracking via [`AWS_BEDROCK_MANTLE_PROJECT`](operations_configuration.md#bedrock-mantle-project), with per-request override ([`AWS_BEDROCK_ALLOW_MANTLE_PROJECT_OVERRIDE`](operations_configuration.md#bedrock-allow-mantle-project-override)) through the `OpenAI-Project` / `anthropic-workspace` header
[:octicons-arrow-right-24: Bedrock Mantle Models](features.md#bedrock-mantle-models)
!!! warning "Additional IAM Permissions (opt-in feature)"
Enabling `AWS_BEDROCK_MANTLE_ENABLED` requires the `bedrock-mantle:CreateInference`, `bedrock-mantle:GetInference`, `bedrock-mantle:DeleteInference`, `bedrock-mantle:ListModels`, `bedrock-mantle:GetModel`, and `bedrock-mantle:CancelInference` permissions on `arn:aws:bedrock-mantle:*:*:project/*`, plus `bedrock-mantle:CallWithBearerToken` on `*`. See [IAM Permissions](operations_iam_permissions.md#bedrock-mantle-iam).
#### :material-api: New APIs
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/videos`](api_openai_videos.md) – create, poll, list, download, and delete video generation jobs | {: style="height:20px;width:20px"} Amazon Bedrock - Amazon Nova Reel, Luma Ray 2 |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/moderations`](api_openai_moderations.md) – text and image content classification | {: style="height:20px;width:20px"} Amazon Bedrock - Guardrails, Amazon Comprehend |
| {: style="height:20px;width:20px"} **Cohere** | [`/cohere/v2/rerank`](api_cohere_rerank.md) – document reranking (Amazon Rerank 1.0, Cohere Rerank 3.5) | {: style="height:20px;width:20px"} Amazon Bedrock - Rerank API |
| {: style="height:20px;width:20px"} **Cohere** | [`/cohere/v2/embed`](api_cohere_embed.md) – embeddings over all Bedrock embedding models | {: style="height:20px;width:20px"} Amazon Bedrock - embedding models |
| **stdapi.ai** | [`/model_pricing`](api_model_pricing.md) – exact AWS unit prices per model | AWS Price List API |
#### :material-brain: Extended Reasoning
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/responses`](api_openai_responses.md#extended-reasoning) – Bedrock `reasoningContent` returned as reasoning output items | {: style="height:20px;width:20px"} Amazon Bedrock - Converse API |
| {: style="height:20px;width:20px"} **OpenAI** | Streaming `response.output_item.added` / `response.reasoning_text.delta` / `.done` events for reasoning content | {: style="height:20px;width:20px"} Amazon Bedrock - Converse API |
| {: style="height:20px;width:20px"} **OpenAI** | `include=["reasoning.encrypted_content"]` – signature/redacted round-trip for multi-turn reasoning continuation | {: style="height:20px;width:20px"} Amazon Bedrock - Converse API |
#### :material-chat: Conversations
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `store=true` + `GET/DELETE /v1/responses/{id}`, input items listing, and `previous_response_id` continuation | {: style="height:20px;width:20px"} Amazon Bedrock - session management |
| {: style="height:20px;width:20px"} **OpenAI** | `POST /v1/responses/{id}/cancel` – endpoint parity for the cancel lifecycle (always fails for session-stored responses, which never run in background mode; Mantle-stored responses are cancelled upstream) | {: style="height:20px;width:20px"} Amazon Bedrock - session management |
| {: style="height:20px;width:20px"} **OpenAI** | `store=true` + `GET/DELETE /v1/chat/completions/{id}`, `GET /v1/chat/completions` listing, `POST /v1/chat/completions/{id}` metadata updates, and input messages listing | {: style="height:20px;width:20px"} Amazon Bedrock - session management |
| {: style="height:20px;width:20px"} **OpenAI** | [`/v1/responses/compact`](api_openai_responses.md#conversation-compaction) – stateless conversation compaction | {: style="height:20px;width:20px"} Amazon Bedrock - Converse API |
| {: style="height:20px;width:20px"} **OpenAI** | `moderation` request parameter on chat completions and responses, with results reported in the response | {: style="height:20px;width:20px"} Amazon Bedrock - Guardrails |
#### Platform Features
| Feature | Description |
|-----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Claude 5 models | Explicit support for the Claude 5 generation — **Opus 5**, **Sonnet 5**, **Fable 5**, and **Mythos** — with each model's server tool set and reasoning configuration matched to what Bedrock actually accepts (Opus 5 exposes no computer use tool; Fable and Mythos always reason and reject a disabled configuration). Model matching covers unreleased versions of each family, so a new minor or major release inherits its family's behavior instead of a generic fallback. Validated end-to-end across the full Claude feature matrix, from Claude 4.5 through Claude 5 |
| Multi-region AWS AI services | Automatic multi-region failover for Amazon Polly, Transcribe, Translate, and Comprehend (per-engine voice discovery, co-located Transcribe buckets, latency-ordered region pools) |
| Fault-tolerant, faster startup | Unreachable Bedrock regions or Polly engines no longer abort startup; they are skipped with a warning and retried on the next refresh — and startup is faster overall |
| Usage & cost tracking | Request logs report the usage actually billed by AWS with its cost computed from live AWS pricing, optionally exported as CloudWatch metrics ([`CLOUDWATCH_METRICS`](operations_logging_monitoring.md#cloudwatch-metrics-emf)); the previous token estimation is removed and its `TOKENS_ESTIMATION*` settings are deprecated and ignored |
| [Cost attribution](operations_cost_management.md#aws-cost-attribution) | Request, server, and user correlation metadata is now attached to every synchronous Bedrock inference call — the `InvokeModel` family included, not only `Converse` — so Bedrock invocation logs can be filtered and costs attributed per request or per user |
| Smaller container images | The published images shrink by around 40% — 413 MB to 253 MB for the AWS Marketplace image, 377 MB to 230 MB for the community image — cutting pull time and storage. ffmpeg is now built with only the audio encoders the server uses, and the unused OpenTelemetry gRPC exporter is no longer installed |
| Video retention (`AWS_S3_VIDEOS_EXPIRES_AFTER`) | Optional retention period for generated videos, reported as `expires_at` and enforced on download |
| Upload expiry (`expires_after`) | Multipart upload sessions honor the OpenAI `expires_after` policy on the resulting file |
| Session storage encryption | Optional KMS key for Amazon Bedrock session storage (`AWS_BEDROCK_SESSION_ENCRYPTION_KEY_ARN`) |
| Proxy trust (`PROXY_TRUSTED_HOSTS`) | `X-Forwarded-*` headers are only honored when sent by a trusted reverse-proxy address |
| Input file size limit (`MAX_INPUT_FILE_SIZE`) | Optional cap on the size of downloaded/decoded input files, with bounded download concurrency (`MAX_CONCURRENT_INPUT_DOWNLOADS`) |
| Legacy model opt-in fix | `AWS_BEDROCK_LEGACY` now also exposes models whose AWS legacy date has already passed (e.g. Amazon Nova Reel) |
#### :material-shield-lock: Security Hardening
- **MCP transports and `/search_models` now require authentication** when an API key is configured — clients that relied on these endpoints being open must now send the API key
- SSRF protection hardened against IP-literal encoding and DNS-rebinding bypasses on URL file inputs
- `s3://` file inputs are restricted to the server's allowed buckets, and multipart upload filenames are validated
- Decoded image size is capped against decompression-bomb payloads
- ARNs and AWS account IDs are redacted from client-facing error messages, and presigned URL signatures are stripped from logs and traces
- An empty resolved API key (e.g. a blank secret value) now disables authentication cleanly instead of matching an empty bearer token, and CORS no longer allows credentialed cross-origin requests
- **Reduced container attack surface**: the images no longer carry the video codec, X11 and font libraries that a distribution ffmpeg package links — x264, x265, AOM, dav1d, SVT-AV1 and SDL2 among them — nor the gRPC stack. None were reachable from the audio transcoding ffmpeg is used for, and they accounted for the bulk of the images' third-party native code. ffmpeg is built from the same version the base distribution ships, with only the audio encoders in use, and enables no GPL-licensed component
#### :material-robot-outline: Agent SDK Compatibility
The Responses API request/response surface was audited and hardened against the OpenAI SDK and real agent clients, end-to-end tested against the **OpenAI Codex CLI**:
- Hosted and agent tool types (`web_search`, `computer_use`, `file_search`, `custom`/`namespace` tools, and other items without a Bedrock equivalent) are now accepted and dropped instead of rejected with `400`, preserving compatibility with existing agent tooling
- Streaming responses now correctly terminate with `response.incomplete` or `response.failed` (matching upstream behavior) instead of always reporting `response.completed`
- Mid-stream errors emit the spec-compliant `error` SSE event
- `input_tokens` usage now includes cache read/write tokens, matching OpenAI's accounting
- `url_citation` annotations are emitted alongside their streaming events
- Echoed reasoning items tolerate the field variations produced by different SDKs and agent clients
#### :material-bug: Fixes
- Rerank models are no longer incorrectly advertised on Converse-based chat routes and MCP tools
- Fixed per-request model parameter overrides (`default_model_params`) occasionally leaking into subsequent requests for the same model
- 5xx provider errors now report server-side error types (`server_error`/`api_error`) in OpenAI and Anthropic error envelopes instead of `invalid_request_error`
- Unknown paths (`404`) and wrong methods (`405`) now return the error envelope of the API family they were sent to, instead of the framework's default `detail` payload
- The Anthropic Messages API now returns `404` instead of `400` for an unknown model, matching the upstream API, and rejects a `top_p` above `1.0`
- Audio transcription returns plain text for `response_format=text` and now defaults `verbose_json` to segment timestamps
- Responses API usage reports `input_tokens_details.cache_write_tokens`, which recent OpenAI SDKs require to parse a response
- Files API listing and cursor pagination order by creation time again: file IDs now use an order-preserving alphabet, where the previous one could sort a newer file first. IDs issued before this release keep working, but sort among themselves as before until they expire
- Newer Anthropic client request fields (free-form JSON Schema keywords in tool `input_schema`, adaptive thinking `display`) are accepted instead of rejected in strict validation mode
- Amazon Nova 2 no longer fails on `max_tokens` combined with high reasoning effort (the cap is dropped with a logged warning)
- Explicit cache points are kept off tool-related content blocks for models without tool caching support
- The Files API unavailable error no longer exposes the S3 bucket configuration detail
- Fixed input files from one request occasionally leaking into later requests served by the same connection, which could fail those requests with internal errors
- Anthropic Messages streams now emit an empty tool-input delta for tool calls without arguments, so SDK stream accumulators no longer fail on argument-less tool calls
- JSON-body image edit and variation requests now accept the `model` field instead of rejecting the request
- Model listings now report `service: "AWS Bedrock Runtime"` for classic Bedrock models (previously `"Amazon Bedrock"`), distinguishing them from `"AWS Bedrock Mantle"`
- High reasoning effort now maps to the intended thinking-token budget on Anthropic Claude models (the budget factor was previously miscomputed)
- Setting `log_level` to `disabled` now suppresses all log output as documented, instead of publishing every event
- Server startup no longer fails when the ECS container metadata endpoint answers slowly, which could prevent small Fargate tasks from starting: the lookup is retried, then falls back to the STS caller identity with a startup warning
- Multipart upload parts are numbered from the parts already stored in S3 instead of a per-instance counter: with several server instances behind a load balancer, two parts of one upload could be given the same number, overwriting each other and failing the upload
- Multi-region failover now covers a region that does not offer the service at all: with no [`AWS_COMPREHEND_REGION`](operations_configuration.md#aws-comprehend-region) set, a Bedrock region without Amazon Comprehend moves on to the next one as documented, instead of failing language detection and Comprehend moderation
---
### v1.13.0 – Terraform Module Compliance & Security Hardening
This release focuses on the [stdapi-ai Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) and its child modules — [VPC](https://github.com/JGoutin/terraform-aws-vpc), [KMS](https://github.com/JGoutin/terraform-aws-kms-key), and [ECS Fargate](https://github.com/JGoutin/terraform-aws-ecs-fargate) — adding detailed AWS Security Hub control documentation and closing several compliance gaps: default security group lockdown, ALB access logging, EFS POSIX user enforcement with native backups, and optional compliance/GuardDuty/DNS Firewall VPC integrations. All four modules now also accept a `tags` variable for custom resource tagging.
!!! info "Documentation-first release"
Every module README now includes a full Security Hub Foundational Security Best Practices (FSBP) control mapping. See [Authentication & Security](operations_authentication_security.md#aws-security-hub-guardduty-dns-firewall-integration) for a summary and links to each module.
#### :material-bug: Fixes
- Added the missing `1h` and `5m` values to `PromptCacheRetention` for Bedrock-specific prompt cache TTLs in the OpenAI Responses API
#### :material-shield-star: Security Hub & Compliance Hardening
| Feature | Module | Description |
|-----------------------------------------|----------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Security Hub FSBP control documentation | VPC, KMS, ECS Fargate, stdapi-ai | Per-control (pass/fail/conditional/N-A) tables added to each module README |
| Default security group lockdown | VPC | New `aws_default_security_group` resource revokes all default ingress/egress rules (EC2.2 / CIS 5.4) |
| VPC Flow Logs retention | VPC | Default retention increased from 7 to 365 days (EC2.6) |
| Compliance VPC endpoints | VPC | New `compliance_vpc_endpoints_enabled` variable adds ECR, SSM, SSM Contacts, and SSM Incidents interface endpoints |
| GuardDuty VPC endpoint | VPC | New `guardduty_vpc_endpoint_enabled` variable adds the `guardduty-data` interface endpoint |
| Route 53 Resolver DNS Firewall | VPC | New `dns_firewall_enabled` variable blocks/alerts on DNS queries to known-malicious domains (AWS Managed Domain Lists, plus DGA/DNS-tunneling detection via `dns_firewall_advanced_enabled`); dedicated VPC only |
| ALB access logging | stdapi-ai | New `alb_access_logging_enabled` variable (default `true`) logs ALB access to a dedicated, encrypted S3 bucket |
| EFS POSIX user enforcement | ECS Fargate | `mount_points` now accepts an `efs_posix_user` object to enforce a POSIX identity on EFS access points (EFS.4) |
| EFS native backups | ECS Fargate | New `mount_points_efs_backup_enable` variable enables native EFS automatic backups, independent of the existing AWS Backup plan (EFS.7) |
| Resource tagging | VPC, KMS, ECS Fargate, stdapi-ai | New `tags` variable propagates custom tags to nearly all created resources (IAM.24 / EC2.48) |
#### :material-cog-outline: Other Infrastructure Changes
| Feature | Description |
|-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| AWS provider version bump | Requirement raised to `>= 6.27.0` across all four modules |
| S3 object tag rename | Files API objects and the corresponding Terraform lifecycle rule now use the `stdapi-ai.expires` tag key instead of `expires`; a temporary backward-compatible rule still expires legacy-tagged objects |
| `aws-apn-id` resource tagging | AWS resources created at runtime (Bedrock async jobs, Transcribe jobs, S3 objects) are tagged with `aws-apn-id`, the standard AWS Marketplace attribution tag — an internal, vendor-side tag, not user-configurable |
#### :material-robot-outline: MCP Token Optimization
- Significantly reduced the size of MCP tool descriptions across the API, lowering the token cost of every AI agent session connected to this server
- No change in functionality: all parameter constraints and usage guidance remain intact
---
### v1.12.0 – Completions API, Video Understanding & File References
This release adds the OpenAI-compatible [`/v1/completions`](api_openai_completions.md) endpoint for text-first coding agents and legacy completion clients, **TwelveLabs Pegasus** video understanding for analyzing `video/*` inputs in chat completions, and an input token counting endpoint for the Responses API. Files uploaded through the Files API can now be referenced anywhere a URL is accepted using the new `file-id:` URI scheme. The Anthropic Messages API now accepts `system`-role messages (merged into the system prompt for compatibility), reasoning can be explicitly enabled or disabled, and a new `DEFAULT_MODEL_SERVICE_TIERS` setting applies per-model service tiers automatically.
#### :material-chat: Chat Completions
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/completions` – text completion endpoint for text-first coding agents | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/responses/input_tokens` – input token counting | {: style="height:20px;width:20px"} Amazon Bedrock - CountTokens API |
| {: style="height:20px;width:20px"} **Anthropic** | `/v1/messages` – accepts `system`-role messages (merged into the system prompt) | {: style="height:20px;width:20px"} Amazon Bedrock - Claude models |
| {: style="height:20px;width:20px"} **Twelve Labs** | Pegasus video understanding (`video/*` inputs) | {: style="height:20px;width:20px"} Amazon Bedrock - TwelveLabs Pegasus |
#### :material-microphone: Speech & Audio
| Provider | Endpoint/Feature | AWS Backend |
|--------------------------------------------------------------------------------|-------------------------------------------------------------------|----------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/audio/speech` – case-insensitive voice names & default model | {: style="height:20px;width:20px"} Amazon Polly |
#### Platform Features
| Feature | Description |
|-------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `file-id:` URI scheme | Reference Files API uploads via `file-id:` anywhere a URL is accepted — embeddings, audio transcription/translation, chat, images, and messages |
| Default model service tiers (`DEFAULT_MODEL_SERVICE_TIERS`) | Automatically apply a per-model service tier (`default`, `flex`, `priority`, `reserved`) when none is provided in the request |
| Explicit reasoning enable/disable | Reasoning/thinking can now be explicitly enabled or disabled via request parameters |
| Service tier & guardrail support for Pegasus | TwelveLabs Pegasus requests honor `service_tier` and Bedrock Guardrail configuration |
| MCP speech streaming defaults to SSE | `/v1/audio/speech` defaults `stream_format` to `sse` when invoked as an MCP tool for broader client compatibility |
| Full regional S3 bucket handling | The Terraform module resolves regional S3 buckets via resource-level region (requires AWS provider >= 6.0.0) |
| Reliable cross-region model identifiers | Region routing no longer fails intermittently with "The provided model identifier is invalid": a region whose inference profile is missing or not yet propagated is skipped, and a geo-scoped profile is never sent to a different region |
---
### v1.11.0 – MCP Server, Agent Discovery & Model Search (with v1.11.1–v1.11.4 maintenance updates)
This release introduces a **Model Context Protocol (MCP) server**, making all stdapi.ai API endpoints directly accessible as MCP tools for AI agents and agentic workflows. A new `/search_models` endpoint enables precise discovery of models by route, MCP tool, region, streaming support, and legacy status. Agent-friendly discovery metadata is now exposed via RFC 8288 Link headers and an RFC 9727 machine-readable API catalog at `/.well-known/api-catalog`. Endpoints that previously required binary `multipart/form-data` uploads now also accept an `application/json` body for MCP and HTTP client compatibility. The Anthropic Messages API now accepts `xhigh` as a `reasoning_effort` value.
#### :material-robot-outline: MCP Server
| Feature | Description |
|------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
| MCP server (Streamable HTTP & SSE) | All API endpoints exposed as MCP tools; Streamable HTTP and SSE transports can be independently enabled or disabled via configuration |
| Configurable MCP tool exposure | Individual MCP tools can be selectively enabled or restricted via configuration |
| JSON body for binary endpoints | Audio transcription, audio translation, and image edit endpoints now accept `application/json` with files as base64, data URI, HTTP URL, or S3 URI |
#### :material-magnify: Model Search
| Feature | Description |
|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `/search_models` | New official endpoint to filter models by route, MCP tool name, input/output modalities, region, streaming, and legacy status; returns richer metadata than `/v1/models` or Anthropic `/v1/models`, designed for LLM-driven model selection (replaces BETA and undocumented `/available_models`) |
#### :material-access-point: Agent Discovery
| Feature | Description |
|-------------------------------------------------------|-----------------------------------------------------------------------------------------------|
| RFC 8288 Link headers | Root (`/`) endpoint returns Link headers for resource discovery |
| RFC 9727 API catalog (`/.well-known/api-catalog`) | Machine-readable API catalog for automated agent and tool discovery |
| MCP Server Card (`/.well-known/mcp/server-card.json`) | Advertises available MCP transports and capabilities to AI agents (SEP-1649) |
| `robots.txt` AI signals | Updated `robots.txt` with `Content-Signal` directives and explicit `/.well-known/` allow rule |
#### :material-chat: Chat Completions & Messages
| Provider | Endpoint/Feature | AWS Backend |
|-----------------------------------------------------------------------------------------|------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **Anthropic** | `/v1/messages` `reasoning_effort=xhigh` support | {: style="height:20px;width:20px"} Amazon Bedrock - Claude models |
#### :material-label-off: Deprecation Mappings
- Added automatic fallback for `amazon.nova-reel-v1:0` and `anthropic.claude-3-haiku-20240307-v1:0` to their respective replacements
#### Fixes
- Fix reasoning token double-counting in usage calculation in OpenAI Responses API adapter
- Fix missing `file_id` inputs for image and file processing in OpenAI Responses API adapter
- Remove `store` parameter from unsupported validations in chat completions to ensure client compatibility
#### Fixes & Maintenance (v1.11.1–v1.11.4)
**v1.11.1**
- Make `max_tokens` optional in Anthropic `/v1/messages` to align with the Anthropic API specification
- Remove unsupported reasoning configuration checks for broader client compatibility
- Rename `/v1/responses` route tag from "Responses" to "Chat" in OpenAPI documentation for consistency
**v1.11.2-v1.11.3**
- Add missing MCP dependencies to container image.
**v1.11.4**
- Upgrade Starlette dependency to fix CVE-2026-48710.
---
### v1.10.0 – OpenAI Responses API
This release adds support for the OpenAI [`/v1/responses`](api_openai_responses.md) endpoint—OpenAI's next-generation API designed for building agents and multi-step AI workflows. Drop-in compatible with the OpenAI SDK, it works with all Amazon Bedrock Converse-compatible models and supports streaming, function tools, built-in tools (web search, code interpreter, image generation), extended reasoning, and structured output.
#### :material-chat: Responses (OpenAI-Compatible)
| Provider | Endpoint/Feature | AWS Backend |
|--------------------------------------------------------------------------------|---------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/responses` | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/responses` – `web_search` / `web_search_preview` built-in tool | {: style="height:20px;width:20px"} Amazon Nova models |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/responses` – `code_interpreter` built-in tool | {: style="height:20px;width:20px"} Amazon Nova models |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/responses` – `image_generation` built-in tool | {: style="height:20px;width:20px"} Amazon Bedrock - image models |
#### Fixes
- Fix prompt caching error when messages contain tool-related content on models that do not support tool caching
- Make `signature` field optional in Anthropic message types
- Fix model legacy detection when the end-of-life date falls before the next cache refresh
---
### v1.9.0 – Files API & Images API JSON Body
This release introduces a Files API backed by Amazon S3, available through both the OpenAI-compatible and Anthropic-compatible interfaces. Files uploaded via either API share the same S3 storage and can be referenced across both interfaces. Large files can be uploaded incrementally using the OpenAI multipart uploads API. Stored files can be referenced by ID directly in image edit and variation requests (JSON body), as well as in chat completion messages as document or image inputs. The image edits endpoint now also accepts an `application/json` body as an alternative to multipart form-data, making it easier to chain pipeline steps without re-uploading files.
!!! warning "New Required Configuration"
Files API requires `AWS_S3_BUCKET` to be configured (shared with the image URL response feature). The S3 prefix for stored files defaults to `files/` and is configurable via `AWS_S3_FILES_PREFIX`. Ensure your IAM role includes read, write, delete, and list permissions on the files prefix in addition to the existing S3 permissions for presigned URLs.
#### :material-folder: Files & Storage
| Provider | Endpoint/Feature | AWS Backend |
|-----------------------------------------------------------------------------------------|---------------------------------------|-------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/files` – CRUD operations | {: style="height:20px;width:20px"} Amazon S3 |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/uploads` – multipart uploads | {: style="height:20px;width:20px"} Amazon S3 |
| {: style="height:20px;width:20px"} **Anthropic** | `/v1/files` – CRUD operations | {: style="height:20px;width:20px"} Amazon S3 |
#### :material-image: Image Generation
| Provider | Endpoint/Feature | AWS Backend |
|--------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/images/edits` – JSON body with `images`/`mask` referencing Files API IDs or URLs | {: style="height:20px;width:20px"} Amazon Bedrock - image models |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/images/variations` – JSON body with `image` referencing a Files API ID or URL | {: style="height:20px;width:20px"} Amazon Bedrock - image models |
#### :material-chat: Chat Completions & Messages
| Provider | Endpoint/Feature | AWS Backend |
|-----------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | Files API file IDs usable as document/image inputs in chat completions | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
| {: style="height:20px;width:20px"} **Anthropic** | Files API file IDs usable as document/image inputs in messages | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
#### Fixes
- Document inputs via S3 URLs are not supported as Bedrock Converse API inputs for some models (e.g., Claude) — now properly detected and handled
---
### v1.8.0 – Broader Model Compatibility & Structured Output
This release focuses on improving reliability and compatibility across a wide variety of models. Structured response formats (JSON object and JSON schema) are now supported on OpenAI chat completions, and request metadata can be forwarded to Bedrock. Tool handling has been significantly improved—both for model-specific system tools and for Amazon Nova's grounding tool, including multi-turn support. Region routing is now more robust, correctly enforcing non-global inference profiles for region-restricted models and handling edge cases gracefully.
!!! warning "New Required IAM Permissions"
v1.8.0 requires two new IAM permissions to attach request metadata tags to jobs:
- **`bedrock:TagResource`** on `arn:aws:bedrock:*:*:async-invoke/*` — needed for Bedrock asynchronous invocation jobs (see [IAM Permissions](operations_configuration.md#bedrock-iam)). The `twelvelabs.marengo-embed-3-0-v1:0` and `twelvelabs.marengo-embed-2-7-v1:0` models rely on asynchronous invocation and will fail with an access denied error if this permission is missing.
- **`transcribe:TagResource`** on `arn:aws:transcribe:*:*:transcription-job/*` — needed for Amazon Transcribe transcription jobs (see [IAM Permissions](operations_configuration.md#speech-to-text-optional)). The `amazon.transcribe` model will fail with an access denied error if this permission is missing.
Ensure your IAM role or user policy includes both statements before upgrading to v1.8.0.
#### :material-chat: Chat Completions
| Provider | Endpoint/Feature | AWS Backend |
|-----------------------------------------------------------------------------------------------|-------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `response_format` – JSON object and JSON schema structured output | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
| {: style="height:20px;width:20px"} **OpenAI** | `metadata` – request metadata forwarding to Bedrock | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
| {: style="height:20px;width:20px"} **Amazon Nova** | Nova Code Interpreter global profile support | {: style="height:20px;width:20px"} Amazon Bedrock - Nova models |
#### :material-message: Messages (Anthropic-Compatible)
| Provider | Endpoint/Feature | AWS Backend |
|-----------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **Amazon Nova** | `nova_grounding` responses mapped to `web_search` content blocks | {: style="height:20px;width:20px"} Amazon Bedrock - Nova models |
| {: style="height:20px;width:20px"} **Amazon Nova** | Multi-turn conversation support with `nova_grounding` | {: style="height:20px;width:20px"} Amazon Bedrock - Nova models |
#### Platform Features
| Feature | Description |
|--------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Non-global profiles for region-restricted models | Region-restricted models are now always assigned non-global inference profiles, preventing requests from bypassing configured region restrictions |
| Region routing edge case handling | Region routing gracefully handles cases where no usable regions are available |
| ECS-based server ID | When running on ECS, `server_id` in logs is set to `task_id.container_name` for precise instance identification across tasks and containers |
| Request metadata tagging | stdapi.ai request context (`request_id`, `server_id`, `user_id`) is automatically attached as tags to every Bedrock and Amazon Transcribe job, making it easy to trace API calls across AWS service logs |
#### Fixes
- Fix `systemTool_` prefix handling: removed broken auto-promotion logic; system tools require specific tool output handling not compatible with generic tool forwarding
- `AWS_BEDROCK_LEGACY` default changed from `true` to `false` to prevent access denied errors on legacy models that have not been actively used recently
- Bedrock read timeouts are now handled as standard model errors (503) instead of unhandled exceptions, and are properly retried across regions when multi-region routing is enabled
---
### v1.7.0 – Automatic Region Routing, Deprecated Model Fallback & Resilience Improvements
The headline feature of v1.7 is **automatic multi-region routing**: stdapi.ai now intelligently distributes requests across your configured AWS regions, failing over automatically on quota limits or unavailability—and because each region carries its own independent quota, adding regions directly multiplies your effective tokens-per-minute and daily limits. Alongside this, deprecated model IDs are transparently redirected to their replacements so clients survive AWS model retirements without any code changes. This release also adds S3 URL support for file inputs across all relevant endpoints, a configurable AI response timeout, and memory efficiency improvements.
#### Platform Features
| Feature | Description |
|-------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Automatic region routing with configurable strategies | Intelligently distributes Bedrock requests across configured AWS regions with automatic failover on quota limits or unavailability; supports `ordered`, `lowest_latency`, and `round_robin` strategies |
| Deprecated model fallback | Transparently reroute deprecated model IDs to their replacements; extend or override the built-in mapping; warns on legacy model usage |
| AI response timeout | Configurable timeout for AI model responses to prevent indefinitely hanging requests |
| Expanded file input support | File inputs (images, documents, audio) now support S3 URLs in addition to HTTP URLs, data URIs, and plain base64 across all relevant endpoints; improves memory efficiency by releasing file data as early as possible |
| Model lifecycle timestamps | Model created/updated timestamps now derived from lifecycle data (`startOfLifeTime`, `endOfLifeTime`) |
#### Fixes
- Fix SSE stream error handling in monitoring to handle specific API and AWS client errors gracefully
- Fix audio MIME type detection failure when `libmagic`'s in-memory buffer path silently returns `application/octet-stream`; fall back to file-based detection to ensure correct format is sent to Bedrock
---
### v1.6.0 – Anthropic API Compatibility & Advanced Claude Capabilities
Introduces a full Anthropic-compatible API layer, enabling direct use of the Anthropic SDK and Claude-native tools with Amazon Bedrock. Adds Claude server tools support via OpenAI chat completions, token count estimation, automatic Anthropic beta flag filtering, and configurable route prefixes.
#### :material-chat: Chat Completions
| Provider | Endpoint/Feature | AWS Backend |
|--------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/chat/completions` Claude server tools (`bash`, `str_replace_based_edit_tool`, `computer`, `memory`) | {: style="height:20px;width:20px"} Claude models on Amazon Bedrock |
#### :material-message: Messages (Anthropic-Compatible)
| Provider | Endpoint/Feature | AWS Backend |
|-----------------------------------------------------------------------------------------------|-----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **Anthropic** | `/v1/messages` – Full Anthropic Messages API | {: style="height:20px;width:20px"} Amazon Bedrock - Converse API |
| {: style="height:20px;width:20px"} **Anthropic** | `/v1/messages/count_tokens` – Token counting | {: style="height:20px;width:20px"} Amazon Bedrock - CountTokens API |
| {: style="height:20px;width:20px"} **Claude** | Claude server tools (bash, text editor, computer, memory) | {: style="height:20px;width:20px"} Amazon Bedrock - Claude models |
| {: style="height:20px;width:20px"} **Amazon Nova** | Web search tool (`web_search` → `nova_grounding`) | {: style="height:20px;width:20px"} Amazon Bedrock - Nova models |
#### :material-format-list-bulleted: Model Discovery (Anthropic-Compatible)
| Provider | Endpoint/Feature | AWS Backend |
|-----------------------------------------------------------------------------------------|-----------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **Anthropic** | `/v1/models` – List models (Anthropic format) | {: style="height:20px;width:20px"} Amazon Bedrock - model catalog |
| {: style="height:20px;width:20px"} **Anthropic** | `/v1/models/{model_id}` – Get model details | {: style="height:20px;width:20px"} Amazon Bedrock - model catalog |
#### Platform Features
| Feature | Description |
|---------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
| `ANTHROPIC_ROUTES_PREFIX` configuration | Configurable base path prefix for Anthropic-compatible routes (default: `/anthropic`) |
| `OPENAI_ROUTES_PREFIX` configuration | Configurable base path prefix for OpenAI-compatible routes |
| Real usage tracking (`usage` in logs) | Token counts sourced directly from AWS billing data (replaces tiktoken-based estimation) |
| Anthropic beta flag filtering (`ANTHROPIC_BETA_FILTER`) | Automatically filter unsupported `anthropic-beta` flags to prevent Bedrock `ValidationException` errors; extensible via `ANTHROPIC_BETA_ALLOWLIST` |
| Claude model name aliases | Use official Anthropic model names (e.g., `claude-opus-4-8`) auto-resolved to Amazon Bedrock identifiers |
---
### v1.5.0 – Advanced Reasoning & Model Compatibility (with v1.5.1–v1.5.2 maintenance updates)
Introduces advanced reasoning capabilities with Amazon Nova 2 and Anthropic Claude 4.6+ adaptive reasoning, enhanced system prompt handling for broader model compatibility.
#### :material-chat: Chat Completions
| Provider | Endpoint/Feature | AWS Backend |
|-----------------------------------------------------------------------------------------------|-----------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | System prompt handling for unsupported models | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
| {: style="height:20px;width:20px"} **Amazon Nova** | Nova 2 chat model reasoning implementation | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
| {: style="height:20px;width:20px"} **Claude** | Claude 4.6+ adaptive reasoning configuration | {: style="height:20px;width:20px"} Amazon Bedrock - Claude models |
#### Fixes & Maintenance (v1.5.1–v1.5.2)
**v1.5.2**
- Add "/" route to avoid 404 errors on root endpoint
- Fix empty system content block handling (improves Amazon Bedrock Converse API compatibility)
**v1.5.1**
- Fix Amazon Nova Canvas image editing to fall back to TEXT_IMAGE task type when no mask is provided
---
### v1.4.0 – Audio Enhancements & Model Compatibility
Expands audio capabilities with Mistral Voxtral support, speaker diarization, audio formats for chat completions, and introduces prompt caching TTL and model aliasing for better OpenAI compatibility.
#### :material-chat: Chat Completions
| Provider | Endpoint/Feature | AWS Backend |
|--------------------------------------------------------------------------------|----------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/chat/completions` audio format support | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/chat/completions` extended Bedrock finish reasons mapping | {: style="height:20px;width:20px"} Amazon Bedrock |
| {: style="height:20px;width:20px"} **OpenAI** | Prompt caching TTL support | {: style="height:20px;width:20px"} Amazon Bedrock - prompt caching |
#### :material-microphone: Speech & Audio
| Provider | Endpoint/Feature | AWS Backend |
|-------------------------------------------------------------------------------------|---------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/audio/transcriptions` `diarized_json` format | {: style="height:20px;width:20px"} Amazon Transcribe |
| {: style="height:20px;width:20px"} **Mistral** | Voxtral audio model | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
#### Platform Features
| Feature | Description |
|----------------------------------|--------------------------------------------------------------|
| Model alias support | Seamless OpenAI compatibility via model name aliasing |
#### Fixes
- Fix chat completion file input handling and refactor base64 decoding and MIME handling for file processing.
- Re-raise startup exceptions and disable botocore logging to improve error visibility
---
### v1.3.0 – Image Editing & Variation Support (with v1.3.1–v1.3.5 maintenance updates)
Adds support for OpenAI's image editing and variation endpoints, enabling image manipulation capabilities backed by Amazon Bedrock. Includes maintenance updates for content block handling, tool call validation, streaming fixes, and TTS optimization.
#### :material-image: Image Generation
| Provider | Endpoint/Feature | AWS Backend |
|--------------------------------------------------------------------------------|-------------------------|-------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/images/edits` | {: style="height:20px;width:20px"} Amazon Bedrock - image models |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/images/variations` | {: style="height:20px;width:20px"} Amazon Bedrock - image models |
#### :material-microphone: Speech & Audio (v1.3.2)
| Feature | Description |
|--------------------------------|---------------------------------------------------------------|
| `DEFAULT_TTS_LANGUAGE` setting | Configurable default language for TTS to optimize performance |
#### Fixes & Maintenance (v1.3.1–v1.3.5)
**v1.3.5**
- Refactor content block handling to skip empty entries in assistant responses
**v1.3.4**
- Handle invalid tool call arguments with robust JSON content validation
- Add deprecation mapping for `amazon.titan-image-generator-v2:0` → `amazon.nova-canvas-v1:0`
**v1.3.3**
- Remove premature stop condition for `contentBlockStop` in streaming chat completions
**v1.3.2**
- Support `image[]` array-style notation for OpenAI image edits
- Handle empty audio segments in transcription duration calculation
**v1.3.1**
- Improve JSON parsing for tool arguments and results
- Correct `example` → `examples` in OpenAPI model path parameter
---
### v1.2.0 – Service Tiers, System Tools & Performance Enhancements
Introduces service tiers and latency headers for all Bedrock routes, Bedrock-specific system tools (Nova grounding), GPT5.2 API compatibility, configurable guardrail overrides, and Python 3.14 optimization.
#### :material-chat: Chat Completions
| Provider | Endpoint/Feature | AWS Backend |
|--------------------------------------------------------------------------------|-----------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/chat/completions` `service_tier` parameter | {: style="height:20px;width:20px"} Amazon Bedrock - service tiers |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/chat/completions` Bedrock-specific system tools (Nova grounding) | {: style="height:20px;width:20px"} Amazon Bedrock - system tools |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/chat/completions` GPT5.2 API update (`reasoning_effort=xhigh`) | |
#### :material-shield-check: Content Safety & Moderation
| Feature | AWS Backend |
|-------------------------------------------------|---------------------------------------------------------------------------------------------------------------|
| Configuration flag for guardrail override allow | {: style="height:20px;width:20px"} Amazon Bedrock Guardrails |
#### Platform Features
| Feature | AWS Backend / Description |
|--------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
| Service tiers and latency headers (all Bedrock routes) | {: style="height:20px;width:20px"} Amazon Bedrock - service tiers |
| Python 3.14 support | Upgraded to Python 3.14 with performance optimization |
| Dependency update | Direct aiobotocore usage (replaced aioboto3) |
#### Fixes
- Fix warnings for duplicated FastAPI routes (`/docs` and `/openapi.json`).
---
### v1.1.0 – Embeddings Enhancement, Prompt Caching & Advanced Routing
Expands multimodal embedding capabilities, adds prompt caching support, and introduces advanced routing with application inference profiles and prompt routers.
#### :material-chat: Chat Completions
| Provider | Endpoint/Feature | AWS Backend |
|--------------------------------------------------------------------------------|---------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | Prompt caching `/v1/chat/completions` `prompt_cache_key` | {: style="height:20px;width:20px"} Amazon Bedrock - prompt caching |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/chat/completions` GPT5.1 API update (`reasoning_effort=none`) | |
#### :material-vector-polyline: Embeddings
| Provider | Endpoint/Feature | AWS Backend |
|-----------------------------------------------------------------------------------------------|-------------------------------------------|----------------------------------------------------------------------------------------------------|
| | Intelligent S3 multimodal upload | {: style="height:20px;width:20px"} Amazon S3 |
| | Intelligent Sync/async Bedrock invocation | {: style="height:20px;width:20px"} Amazon Bedrock |
| {: style="height:20px;width:20px"} **Amazon Nova** | Multimodal embeddings models | |
| {: style="height:20px;width:20px"} **Twelve Labs** | Marengo V3 models | |
#### :material-directions-fork: Advanced Routing
| Feature | AWS Backend |
|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------|
| Application inference profiles | {: style="height:20px;width:20px"} Amazon Bedrock - application inference profiles |
| Prompt routers | {: style="height:20px;width:20px"} Amazon Bedrock - prompt routers |
| Server-side ARN mapping | {: style="height:20px;width:20px"} Amazon Bedrock |
| Client-side ARN passing (optional) | {: style="height:20px;width:20px"} Amazon Bedrock |
#### Fixes
- `/v1/chat/completions`: Fix default value passed to the converse API for tools without parameters.
- [stdapi-ai Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai): Fix error if alarms_enabled = true but sns_topic_arn undefined.
---
### v1.0.0 – Foundation Release
The initial release establishes core OpenAI API compatibility with Amazon Bedrock backing.
#### :material-chat: Chat Completions
| Provider | Endpoint/Feature | AWS Backend |
|--------------------------------------------------------------------------------------|----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/chat/completions` | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
| | All models supporting Converse/ConverseStream APIs | {: style="height:20px;width:20px"} Amazon Bedrock - Converse API |
| {: style="height:20px;width:20px"} **Deepseek** | `/v1/chat/completions` `reasoning_content` | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
| {: style="height:20px;width:20px"} **Qwen** | `enable_thinking` + `thinking_budget` parameter | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
| {: style="height:20px;width:20px"} **Qwen** | `top_k` parameter | {: style="height:20px;width:20px"} Amazon Bedrock - foundation models |
#### :material-vector-polyline: Embeddings
| Provider | Endpoint/Feature | AWS Backend |
|----------------------------------------------------------------------------------------------|-----------------------|-----------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/embeddings` | {: style="height:20px;width:20px"} Amazon Bedrock - embedding models |
| {: style="height:20px;width:20px"} **Cohere** | Embed V3 & V4 models | |
| {: style="height:20px;width:20px"} **Twelve Labs** | Marengo V2 models | |
| {: style="height:20px;width:20px"} **Amazon Titan** | Embed V1 & V2 models | |
#### :material-microphone: Speech & Audio
| Provider | Endpoint/Feature | AWS Backend |
|--------------------------------------------------------------------------------|----------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/audio/speech` | {: style="height:20px;width:20px"} Amazon Polly + Amazon Comprehend |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/audio/transcriptions` | {: style="height:20px;width:20px"} Amazon Transcribe |
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/audio/translations` | {: style="height:20px;width:20px"} Amazon Transcribe + Amazon Translate |
#### :material-image: Image Generation
| Provider | Endpoint/Feature | AWS Backend |
|-------------------------------------------------------------------------------------------------|-----------------------------------------|-------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/images/generations` | {: style="height:20px;width:20px"} Amazon Bedrock - image models |
| {: style="height:20px;width:20px"} **Amazon Nova** | Canvas V1 models | |
| {: style="height:20px;width:20px"} **Amazon Titan** | Image Generator V1 & V2 models | |
| {: style="height:20px;width:20px"} **Stability AI** | Image Core, Ultra et SD3.5 Large models | |
#### :material-format-list-bulleted: Model Discovery
| Provider | Endpoint/Feature | AWS Backend |
|--------------------------------------------------------------------------------|------------------|--------------------------------------------------------------------------------------------------------------------|
| {: style="height:20px;width:20px"} **OpenAI** | `/v1/models` | {: style="height:20px;width:20px"} Amazon Bedrock - model catalog |
#### Platform Features
| Feature | AWS Backend |
|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Bedrock Features** | |
| Content filtering and safety | {: style="height:20px;width:20px"} Amazon Bedrock Guardrails |
| Cross-region inference | {: style="height:20px;width:20px"} Amazon Bedrock - global/regional |
| Application inference profiles | {: style="height:20px;width:20px"} Amazon Bedrock - inference profiles |
| Model parameters (temperature, top_p, etc.) | {: style="height:20px;width:20px"} Amazon Bedrock - native parameters |
| Multi-region failover | {: style="height:20px;width:20px"} Amazon Bedrock - multi-region |
| Bedrock guardrails | {: style="height:20px;width:20px"} Amazon Bedrock Guardrails |
| **AWS Services** | |
| File storage | {: style="height:20px;width:20px"} Amazon S3 - presigned URLs, Transfer Acceleration |
| **Authentication** | |
| Static token authentication | {: style="height:20px;width:20px"} AWS SSM Parameter Store / {: style="height:20px;width:20px"} Secrets Manager |
| Development mode (no auth) | |
| **Observability** | |
| Distributed tracing | {: style="height:20px;width:20px"} AWS X-Ray + OpenTelemetry |
| Structured logging | {: style="height:20px;width:20px"} Amazon CloudWatch (When running on ECS/EKS) |
| Health check endpoint | |
| **HTTP/Security** | |
| CORS support | |
| Trusted host validation | |
| Proxy headers (X-Forwarded-*) | |
| GZip compression | |
| **📚 Documentation** | |
| Interactive API docs & OpenAPI schema | |
| **🔌 Compatibility** | |
| Provider-specific parameters | |
---
## Trademarks
Source: https://stdapi.ai/md/trademarks.md
# Trademarks
This page lists the third-party names and logos that appear anywhere in the
stdapi.ai documentation, together with the owner each one belongs to. Every page
of this site also carries, in its footer, the subset of these marks that the page
itself uses.
## :material-scale-balance: How these marks are used
- **They belong to their owners.** Every name and logo listed below is a trademark
or a registered trademark of its respective owner. Nothing on this site
transfers, dilutes, or claims any right in them.
- **They are used for identification only.** stdapi.ai is a gateway that speaks
other vendors' API dialects and routes requests to other vendors' models and
services. Naming those products, and showing their logos next to their names, is
how a reader can tell which product is being discussed. That is the only purpose
the marks serve here.
- **No affiliation, no endorsement.** The owners of these marks are not affiliated
with, and do not endorse, sponsor, or certify, stdapi.ai or JGoutin-dev SARL.
stdapi.ai is an independent product.
- **No licence is claimed.** Using a mark to refer to its owner's product is not a
licence, and none is asserted. Where an owner publishes trademark guidelines,
this site aims to follow them; if you own a mark listed here and want its use
changed or removed, please [get in touch](contact.md) and it will be.
- **Owner names are reported as published.** Some marks belong to companies, some
to foundations, and some to community projects with no incorporated owner. Each
row below names the owner in the form that owner publishes. If a row is wrong,
tell us and it will be corrected.
- **The list is not exhaustive.** It covers the products whose logos this site
displays, the applications the deployment samples run, the model vendors named
across the API reference, and the tools the documentation asks you to install.
Other third-party names appear in passing — for instance when listing the
ecosystems stdapi.ai is compatible with — and are equally the property of their
respective owners, whose use here is likewise for identification only and implies
no affiliation or endorsement in either direction.
!!! info "stdapi.ai's own marks"
The stdapi.ai name and logo belong to JGoutin-dev SARL. The copyright notice
in the footer of every page is separate from this trademark notice and is not
affected by it.
## :material-format-list-bulleted: Marks and their owners
## :material-clipboard-text: Attribution notices requested by owners
Some owners publish a specific wording they ask third parties to reproduce. Those
requests are honoured here.
- Docker and the Docker logo are trademarks or registered trademarks of
Docker, Inc. in the United States and/or other countries. Docker, Inc. and other
parties may also have trademark rights in other terms used herein.
- Postgres, PostgreSQL and the Slonik logo are registered trademarks of the
PostgreSQL Community Association of Canada.
- Kubernetes is a registered trademark of The Linux Foundation.
---
## Overview
Source: https://stdapi.ai/md/api_overview.md
# :material-api: API Overview
stdapi.ai provides OpenAI-, Anthropic-, and Cohere-compatible APIs backed by Amazon Bedrock and AWS AI services. Any application that works with OpenAI, Anthropic, or Cohere works with stdapi.ai by simply changing the API endpoint.
!!! tip "One catalog, discovered automatically"
Amazon Bedrock, Bedrock Mantle, Amazon Polly, Amazon Transcribe and Amazon Comprehend all surface as **models in a single catalog**. stdapi.ai discovers them from your AWS account at startup — there is no model list to declare or maintain, and a model AWS adds appears without a configuration change. They are interchangeable by name on a shared endpoint: [`GET /v1/models`](api_openai_models.md) lists them together, [`GET /search_models`](api_search_models.md) filters them by capability, and the endpoint routes to whichever AWS service backs the model you named — `POST /v1/audio/transcriptions` reaches Amazon Transcribe or a Bedrock audio model, and `POST /v1/moderations` reaches Bedrock Guardrails or Amazon Comprehend, from the same request, and the [Models](models.md) page shows the whole catalogue with prices and scores.
!!! tip "Attachments, however large"
Every multimodal route takes its images, documents, audio and video as base64, a data URI, an HTTPS URL, an `s3://` URI or a Files API ID. On chat completions, messages and responses served by Amazon Bedrock — Bedrock Mantle models excepted — an attachment past what the chosen model reads inside a request is delivered by reference instead, with no change to the request, wherever that model reads that kind of attachment from storage; the models that read it inline only refuse it with `413`, stating the size they accept. See [Attachment Size](features.md#attachment-size).
## :material-book-open-variant: Documentation & Tooling
stdapi.ai provides multiple resources for exploring and testing the API—choose the one that fits your workflow:
### :material-book-open-variant: Documentation Resources
* **Per-endpoint guides** – The pages in this section (linked from the [endpoint tables below](#supported-endpoints)) with parameter details, feature tables, and examples
* **[API Reference](api_reference.md)** – Browsable rendering of the full OpenAPI specification (request/response schemas for every endpoint)
* **[OpenAPI Specification](openapi.yml)** – Full machine-readable schema for integration and tooling
### :material-play-circle: Live API Playground
**When running the server**, access these interactive interfaces (can be enabled via [configuration options](operations_configuration.md)):
| Interface | URL | Best For |
|--------------------|---------------------------------|--------------------------------------------------------------------------------|
| **Swagger UI** | `http://localhost/docs` | Testing endpoints directly in your browser with live request/response examples |
| **ReDoc** | `http://localhost/redoc` | Reading and searching through clean, organized documentation |
| **OpenAPI Schema** | `http://localhost/openapi.json` | Generating client code or importing into API tools like Postman |
## :material-api: Supported Endpoints
### { style="height: 1.2em; vertical-align: text-bottom;" } OpenAI-Compatible API
| Category | Endpoint | Capability | Documentation |
|-------------------|-----------------------------------|-----------------------------------------------------------------------------|--------------------------------------------------------|
| **💬 Chat** | `POST /v1/chat/completions` | Multi-modal conversations with text, images, video, documents | [Chat Completions →](api_openai_chat_completions.md) |
| | `GET /v1/chat/completions` | List stored chat completions | [Chat Completions →](api_openai_chat_completions.md) |
| | `GET/POST/DELETE /v1/chat/completions/{id}` | Retrieve, update metadata, or delete a stored chat completion | [Chat Completions →](api_openai_chat_completions.md) |
| | `GET /v1/chat/completions/{id}/messages` | List the messages of a stored chat completion | [Chat Completions →](api_openai_chat_completions.md) |
| | `POST /v1/completions` | Simple prompt-to-text completion — recommended for MCP and text-only agents | [Completions →](api_openai_completions.md) |
| | `POST /v1/responses` | Conversational AI with tool calling, streaming, and server-side storage | [Responses →](api_openai_responses.md) |
| | `POST /v1/responses/input_tokens` | Count input tokens without generating a response | [Responses →](api_openai_responses.md) |
| | `POST /v1/responses/compact` | Compact a conversation into a reusable summary item | [Responses →](api_openai_responses.md) |
| | `GET/DELETE /v1/responses/{id}` | Retrieve or delete stored responses | [Responses →](api_openai_responses.md) |
| | `POST /v1/responses/{id}/cancel` | Cancel a background response | [Responses →](api_openai_responses.md) |
| | `GET /v1/responses/{id}/input_items` | List the input items of a stored response | [Responses →](api_openai_responses.md) |
| **💬 Conversations** | `POST /v1/conversations` | Create a conversation holding multi-turn state | [Conversations →](api_openai_conversations.md) |
| | `GET/POST/DELETE /v1/conversations/{id}` | Retrieve, update the metadata of, or delete a conversation | [Conversations →](api_openai_conversations.md) |
| | `GET/POST /v1/conversations/{id}/items` | List or add conversation items | [Conversations →](api_openai_conversations.md) |
| | `GET/DELETE /v1/conversations/{id}/items/{item_id}` | Retrieve or delete one conversation item | [Conversations →](api_openai_conversations.md) |
| **🎨 Images** | `POST /v1/images/generations` | Text-to-image generation | [Generations →](api_openai_images_generations.md) |
| | `POST /v1/images/edits` | Image editing and transformations | [Edits →](api_openai_images_edits.md) |
| | `POST /v1/images/variations` | Generate image variations | [Variations →](api_openai_images_variations.md) |
| **🎬 Videos** | `POST/GET/DELETE /v1/videos` | Asynchronous text/image-to-video generation jobs | [Videos →](api_openai_videos.md) |
| | `GET /v1/videos/{id}/content` | Download generated video content | [Videos →](api_openai_videos.md) |
| **🔊 Audio** | `POST /v1/audio/speech` | Text-to-speech synthesis | [Text to Speech →](api_openai_audio_speech.md) |
| | `POST /v1/audio/transcriptions` | Speech-to-text transcription | [Transcriptions →](api_openai_audio_transcriptions.md) |
| | `POST /v1/audio/translations` | Speech-to-English translation | [Translations →](api_openai_audio_translations.md) |
| **🎙️ Realtime** | `POST /v1/realtime/client_secrets` | Mint a short-lived client secret carrying a session configuration | [Realtime →](api_openai_realtime.md) |
| | `WS /v1/realtime` | Live, bidirectional speech-to-speech session | [Realtime →](api_openai_realtime.md) |
| **🧠 Embeddings** | `POST /v1/embeddings` | Vector embeddings for semantic search | [Embeddings →](api_openai_embeddings.md) |
| **🛡️ Moderations** | `POST /v1/moderations` | Content safety classification via Bedrock Guardrails or Amazon Comprehend | [Moderations →](api_openai_moderations.md) |
| **📋 Models** | `GET /v1/models` | List available models | [Models →](api_openai_models.md) |
| | `GET /v1/models/{model}` | Retrieve details for one model | [Models →](api_openai_models.md) |
| **📁 Files** | `POST/GET/DELETE /v1/files` | Upload, list, retrieve, download, delete files | [Files →](api_openai_files.md) |
| | `POST /v1/uploads` | Multipart upload sessions for large files | [Files →](api_openai_files.md) |
| | `POST /v1/uploads/{id}/parts`, `…/complete`, `…/cancel` | Add parts to, complete, or cancel an upload session | [Files →](api_openai_files.md) |
| **🔎 Vector Stores** | `POST/GET/DELETE /v1/vector_stores` | Create, list, retrieve, update, delete a searchable file collection | [Vector Stores →](api_openai_vector_stores.md) |
| | `POST /v1/vector_stores/{id}/search` | Search the indexed files by meaning | [Vector Stores →](api_openai_vector_stores.md) |
| | `POST/GET/DELETE /v1/vector_stores/{id}/files`, `…/file_batches` | Attach, list, read and detach the indexed files | [Vector Stores →](api_openai_vector_stores.md) |
| **📦 Batches** | `POST/GET /v1/batches` | Run a file of requests asynchronously at the batch price | [Batches →](api_openai_batches.md) |
| | `POST /v1/batches/{id}/cancel` | Cancel a running batch | [Batches →](api_openai_batches.md) |
### :material-magnify: stdapi.ai Native Extensions
| Category | Endpoint | Capability | Documentation |
|----------|----------|------------|---------------|
| **🔍 Models** | `GET /search_models` | Search models by capability: modality, route, MCP tool, region, streaming, batch, legacy status | [Search Models →](api_search_models.md) |
| **💰 Pricing** | `GET /model_pricing` | Exact AWS unit prices per model: tokens, tiers, cache TTLs, routing, media specs | [Model Pricing →](api_model_pricing.md) |
### { style="height: 1.2em; vertical-align: text-bottom;" } Anthropic-Compatible API
| Category | Endpoint | Capability | Documentation |
|-----------------|--------------------------------------------|---------------------------------------------------------------|-----------------------------------------|
| **💬 Messages** | `POST /anthropic/v1/messages` | Multi-modal conversations with text, images, video, documents | [Messages →](api_anthropic_messages.md) |
| | `POST /anthropic/v1/messages/count_tokens` | Count tokens without sending a message | [Messages →](api_anthropic_messages.md) |
| **📋 Models** | `GET /anthropic/v1/models` | List available models | [Models →](api_anthropic_models.md) |
| | `GET /anthropic/v1/models/{model_id}` | Retrieve model details | [Models →](api_anthropic_models.md) |
| **📁 Files** | `POST/GET/DELETE /anthropic/v1/files` | Upload, list, retrieve, download, delete files | [Files →](api_anthropic_files.md) |
| **📦 Batches** | `POST/GET/DELETE /anthropic/v1/messages/batches` | Run many message requests asynchronously at the batch price | [Message Batches →](api_anthropic_batches.md) |
| | `GET /anthropic/v1/messages/batches/{id}/results` | Stream a finished batch's results as JSONL | [Message Batches →](api_anthropic_batches.md) |
| | `POST /anthropic/v1/messages/batches/{id}/cancel` | Cancel a processing batch | [Message Batches →](api_anthropic_batches.md) |
### { style="height: 1.2em; vertical-align: text-bottom;" } Cohere-Compatible API
| Category | Endpoint | Capability | Documentation |
|-------------------|--------------------------|-------------------------------------------------|-----------------------------------|
| **🔀 Rerank** | `POST /cohere/v2/rerank` | Rank documents by semantic relevance to a query | [Rerank →](api_cohere_rerank.md) |
| | `POST /cohere/v1/rerank` | Legacy v1 rerank for older SDKs and tools | [Rerank →](api_cohere_rerank.md#cohere-v1-rerank-api-legacy) |
| **🧠 Embeddings** | `POST /cohere/v2/embed` | Vector embeddings for semantic search | [Embed →](api_cohere_embed.md) |
| | `POST /cohere/v1/embed` | Legacy v1 embed for older SDKs and tools | [Embed →](api_cohere_embed.md#cohere-v1-embed-api-legacy) |
## :material-tools: MCP (Model Context Protocol)
When `ENABLE_MCP_STREAMABLE_HTTP=true` or `ENABLE_MCP_SSE=true` is configured, stdapi.ai exposes all its endpoints as MCP tools. OpenAI-, Anthropic-, and Cohere-compatible tool names follow the pattern `provider_action`; the native extension tools use their bare names (`search_models`, `model_pricing`).
!!! tip "JSON body support for file and audio tools"
MCP tools send JSON bodies — they cannot construct `multipart/form-data`. All file upload, audio, and upload-part tools therefore accept the file or audio content as a base64 string, data URI (`data:;base64,`), HTTPS URL, or S3 URI in the `file` / `data` field instead of a binary attachment — as do the video generation tool's `input_reference` image, the moderation tool's `image_url` input, and the `openai_image_edit`/`openai_image_variation` tools' image inputs (also accepting a bare string in any of these forms, plus a Files API file ID). The full multipart upload workflow (`openai_upload` → `openai_upload_part` → `openai_upload_complete`) is fully MCP-compatible this way.
!!! tip "What the file, video, and audio tools return"
An MCP tool result carries text, an image, or audio — never an arbitrary binary stream. Endpoints that answer with bytes therefore adapt to what the protocol can hold: text content comes back as text, an image as an image, and generated speech as audio when `stream_format` is set to `audio`. Anything else — a video, a PDF or archive read back through `openai_file_content` or `anthropic_file_content`, and any payload above 3 MB — comes back as a small JSON object holding the media type and the `url` to download it from over HTTP, so an agent is told where the result is rather than handed bytes it cannot use.
| MCP Tool | Endpoint |
|----------------------------------|---------------------------------------------|
| **OpenAI Tools** | |
| `openai_chat_completion` | `POST /v1/chat/completions` |
| `openai_chat_completion_list` | `GET /v1/chat/completions` |
| `openai_chat_completion_get` | `GET /v1/chat/completions/{completion_id}` |
| `openai_chat_completion_update` | `POST /v1/chat/completions/{completion_id}` |
| `openai_chat_completion_delete` | `DELETE /v1/chat/completions/{completion_id}` |
| `openai_chat_completion_messages` | `GET /v1/chat/completions/{completion_id}/messages` |
| `openai_completion` | `POST /v1/completions` |
| `openai_response` | `POST /v1/responses` |
| `openai_response_input_tokens` | `POST /v1/responses/input_tokens` |
| `openai_response_compact` | `POST /v1/responses/compact` |
| `openai_response_get` | `GET /v1/responses/{response_id}` |
| `openai_response_delete` | `DELETE /v1/responses/{response_id}` |
| `openai_response_cancel` | `POST /v1/responses/{response_id}/cancel` |
| `openai_response_input_items` | `GET /v1/responses/{response_id}/input_items` |
| `openai_conversation` | `POST /v1/conversations` |
| `openai_conversation_get` | `GET /v1/conversations/{conversation_id}` |
| `openai_conversation_update` | `POST /v1/conversations/{conversation_id}` |
| `openai_conversation_delete` | `DELETE /v1/conversations/{conversation_id}` |
| `openai_conversation_items` | `POST /v1/conversations/{conversation_id}/items` |
| `openai_conversation_items_list` | `GET /v1/conversations/{conversation_id}/items` |
| `openai_conversation_item_get` | `GET /v1/conversations/{conversation_id}/items/{item_id}` |
| `openai_conversation_item_delete` | `DELETE /v1/conversations/{conversation_id}/items/{item_id}` |
| `openai_image_generation` | `POST /v1/images/generations` |
| `openai_image_edit` | `POST /v1/images/edits` |
| `openai_image_variation` | `POST /v1/images/variations` |
| `openai_video_generation` | `POST /v1/videos` |
| `openai_video_list` | `GET /v1/videos` |
| `openai_video_get` | `GET /v1/videos/{video_id}` |
| `openai_video_content` | `GET /v1/videos/{video_id}/content` |
| `openai_video_delete` | `DELETE /v1/videos/{video_id}` |
| `openai_audio_speech` | `POST /v1/audio/speech` |
| `openai_audio_transcription` | `POST /v1/audio/transcriptions` |
| `openai_audio_translation` | `POST /v1/audio/translations` |
| `openai_realtime_client_secret` | `POST /v1/realtime/client_secrets` |
| `openai_embedding` | `POST /v1/embeddings` |
| `openai_moderation` | `POST /v1/moderations` |
| `openai_model_list` | `GET /v1/models` |
| `openai_model_get` | `GET /v1/models/{model}` |
| `openai_file` | `POST /v1/files` |
| `openai_file_list` | `GET /v1/files` |
| `openai_files_get` | `GET /v1/files/{file_id}` |
| `openai_files_delete` | `DELETE /v1/files/{file_id}` |
| `openai_file_content` | `GET /v1/files/{file_id}/content` |
| `openai_vector_store_create` | `POST /v1/vector_stores` |
| `openai_vector_store_list` | `GET /v1/vector_stores` |
| `openai_vector_store_get` | `GET /v1/vector_stores/{vector_store_id}` |
| `openai_vector_store_update` | `POST /v1/vector_stores/{vector_store_id}` |
| `openai_vector_store_delete` | `DELETE /v1/vector_stores/{vector_store_id}` |
| `openai_vector_store_search` | `POST /v1/vector_stores/{vector_store_id}/search` |
| `openai_vector_store_file_create` | `POST /v1/vector_stores/{vector_store_id}/files` |
| `openai_vector_store_file_list` | `GET /v1/vector_stores/{vector_store_id}/files` |
| `openai_vector_store_file_get` | `GET /v1/vector_stores/{vector_store_id}/files/{file_id}` |
| `openai_vector_store_file_update` | `POST /v1/vector_stores/{vector_store_id}/files/{file_id}` |
| `openai_vector_store_file_delete` | `DELETE /v1/vector_stores/{vector_store_id}/files/{file_id}` |
| `openai_vector_store_file_content` | `GET /v1/vector_stores/{vector_store_id}/files/{file_id}/content` |
| `openai_vector_store_file_batch_create` | `POST /v1/vector_stores/{vector_store_id}/file_batches` |
| `openai_vector_store_file_batch_get` | `GET /v1/vector_stores/{vector_store_id}/file_batches/{batch_id}` |
| `openai_vector_store_file_batch_cancel` | `POST /v1/vector_stores/{vector_store_id}/file_batches/{batch_id}/cancel` |
| `openai_vector_store_file_batch_file_list` | `GET /v1/vector_stores/{vector_store_id}/file_batches/{batch_id}/files` |
| `openai_batch` | `POST /v1/batches` |
| `openai_batch_list` | `GET /v1/batches` |
| `openai_batch_get` | `GET /v1/batches/{batch_id}` |
| `openai_batch_cancel` | `POST /v1/batches/{batch_id}/cancel` |
| `openai_upload` | `POST /v1/uploads` |
| `openai_upload_part` | `POST /v1/uploads/{upload_id}/parts` |
| `openai_upload_complete` | `POST /v1/uploads/{upload_id}/complete` |
| `openai_upload_cancel` | `POST /v1/uploads/{upload_id}/cancel` |
| **Anthropic Tools** | |
| `anthropic_message` | `POST /anthropic/v1/messages` |
| `anthropic_message_count_tokens` | `POST /anthropic/v1/messages/count_tokens` |
| `anthropic_model_list` | `GET /anthropic/v1/models` |
| `anthropic_model_get` | `GET /anthropic/v1/models/{model_id}` |
| `anthropic_file` | `POST /anthropic/v1/files` |
| `anthropic_file_list` | `GET /anthropic/v1/files` |
| `anthropic_files_get` | `GET /anthropic/v1/files/{file_id}` |
| `anthropic_files_delete` | `DELETE /anthropic/v1/files/{file_id}` |
| `anthropic_file_content` | `GET /anthropic/v1/files/{file_id}/content` |
| `anthropic_message_batch` | `POST /anthropic/v1/messages/batches` |
| `anthropic_message_batch_list` | `GET /anthropic/v1/messages/batches` |
| `anthropic_message_batch_get` | `GET /anthropic/v1/messages/batches/{message_batch_id}` |
| `anthropic_message_batch_results` | `GET /anthropic/v1/messages/batches/{message_batch_id}/results` |
| `anthropic_message_batch_cancel` | `POST /anthropic/v1/messages/batches/{message_batch_id}/cancel` |
| `anthropic_message_batch_delete` | `DELETE /anthropic/v1/messages/batches/{message_batch_id}` |
| **Cohere Tools** | |
| `cohere_rerank` | `POST /cohere/v2/rerank` |
| `cohere_rerank_v1` | `POST /cohere/v1/rerank` |
| `cohere_embed` | `POST /cohere/v2/embed` |
| `cohere_embed_v1` | `POST /cohere/v1/embed` |
| **Native Extension Tools** | |
| `search_models` | `GET /search_models` |
| `model_pricing` | `GET /model_pricing` |
!!! tip "Filtering MCP Tools"
Use `MCP_INCLUDE_TOOLS` or `MCP_EXCLUDE_TOOLS` environment variables to control which tools are exposed. Always include `search_models` so agents can discover the right model ID dynamically. See [Operations Configuration →](operations_configuration.md#mcp-model-context-protocol) for details.
!!! warning "Token Usage for Complex API Tools"
`anthropic_message`, `openai_chat_completion`, and `openai_response` map to large, complex APIs that may use many tokens (prompt, completion, and tool definitions). Select these tools only if your workflow requires the full API capabilities.
## :material-connection: Using stdapi.ai
stdapi.ai speaks the OpenAI, Anthropic, and Cohere APIs unchanged. Any application built on one of them—chatbots, coding assistants, automation tools, custom scripts—runs against stdapi.ai once you point it at your deployment's base URL and give it that deployment's API key. The model name usually stays as it is, and changes only where it differs.
That is because the Anthropic, OpenAI and Cohere models Bedrock serves are also published under the names their providers use, derived mechanically from the Bedrock identifier rather than curated by hand: `anthropic.claude-opus-5` answers to `claude-opus-5`, `openai.gpt-5.6-sol` to `gpt-5.6-sol`, `openai.gpt-oss-120b-1:0` to `gpt-oss-120b`, `cohere.embed-english-v3` to `embed-english-v3.0`, `cohere.rerank-v3-5:0` to `rerank-v3.5`. A client already asking for one of those names needs no model change at all. Where a name *does* differ — a model from another provider, or one named for a provider this deployment does not serve — [`MODEL_ALIASES`](operations_configuration.md#model-aliases) publishes a served model under the name your application already sends.
What the base URL buys is the catalogue behind it. A model name is resolved against the catalogue your deployment actually serves — Amazon Bedrock, Bedrock Mantle, Polly, Transcribe and Comprehend, across every region you enable — so the choice spans providers instead of one vendor's list. A name the catalogue does not contain is answered with `404`: it is never mapped onto another vendor's model of roughly similar class, because that would serve you a different model than the one you asked for. Use [`GET /search_models`](api_search_models.md) to find one.
### { style="height: 1.2em; vertical-align: text-bottom;" } Using the OpenAI-Compatible API
**To connect your OpenAI application:**
1. **Replace the OpenAI API URL** with your stdapi.ai deployment URL
2. **Use the same authentication mechanism** (Bearer token in the `Authorization` header)
3. **Check the model name against what this deployment serves** — OpenAI's own names for the models Bedrock offers (e.g., `gpt-5.6-sol`, `gpt-oss-120b`) resolve as they stand, as do Bedrock model IDs (e.g., `amazon.nova-micro-v1:0`) and any configured alias. A name Bedrock does not serve, such as `gpt-4o` or `dall-e-3`, returns `404` until you [alias](operations_configuration.md#model-aliases) it onto one it does
That's it: the rest of the OpenAI SDK call is unchanged.
### { style="height: 1.2em; vertical-align: text-bottom;" } Using the Anthropic-Compatible API
**To connect your Anthropic application:**
1. **Replace the Anthropic API URL** (`https://api.anthropic.com`) with your stdapi.ai deployment URL + `/anthropic` (e.g., `https://your-endpoint.com/anthropic`)
2. **Use the same authentication mechanism** (`x-api-key` header and `anthropic-version` header)
3. **Check the model name against what this deployment serves** — official Anthropic names (e.g., `claude-opus-5`) resolve to their Bedrock IDs automatically, or use Bedrock model IDs directly
Anthropic names resolving on their own makes the base URL the only change for most applications — the same mechanism that resolves OpenAI's names on the surface above. A Claude version Bedrock no longer serves returns `404` rather than a substitute, so name a current one.
### { style="height: 1.2em; vertical-align: text-bottom;" } Using the Cohere-Compatible API
**To connect your Cohere application:**
1. **Replace the Cohere API URL** (`https://api.cohere.com`) with your stdapi.ai deployment URL + `/cohere` (e.g., `https://your-endpoint.com/cohere`)
2. **Use the same authentication mechanism** (Bearer token in the `Authorization` header)
3. **Check the model name against what this deployment serves** — Cohere's own names for the models Bedrock offers (e.g., `embed-english-v3.0`, `embed-v4.0`, `rerank-v3.5`) resolve as they stand, as do Bedrock model IDs (e.g., `cohere.rerank-v3-5:0`, `cohere.embed-v4:0`) and any configured alias. A Cohere model Bedrock does not serve, such as `embed-english-light-v3.0`, returns `404` until you [alias](operations_configuration.md#model-aliases) it onto one it does
That's it: your Cohere rerank and embed integrations are otherwise unchanged.
## :material-arrow-right: Next Steps
- :material-chat: [**Chat Completions**](api_openai_chat_completions.md) — Conversational AI with multi-modal support
- :material-image: [**Images**](api_openai_images_generations.md) — Generation, edits, and variations
- :material-movie-open: [**Videos**](api_openai_videos.md) — Asynchronous text/image-to-video generation
- :material-music: [**Audio**](api_openai_audio_speech.md) — Text-to-speech, transcription, and translation
- :material-vector-polyline: [**Embeddings**](api_openai_embeddings.md) — Vector embeddings for search and RAG
- :material-sort: [**Rerank**](api_cohere_rerank.md) — Cohere-compatible document reranking for search and RAG
- :material-format-list-bulleted: [**Models**](api_openai_models.md) — List and discover available models
- :material-view-list: [**Models**](models.md) — Every model served, with modalities, regions, AWS prices and leaderboard scores
- :material-magnify: [**Search Models**](api_search_models.md) — Filter models by capability, modality, route, or MCP tool
- :material-currency-usd: [**Model Pricing**](api_model_pricing.md) — Exact AWS unit prices for cost-aware model selection
- :material-message: [**Messages**](api_anthropic_messages.md) — Anthropic-compatible conversational AI with tool calling
- :material-check-all: [**Features**](features.md) — Full capabilities and AWS integrations
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy to AWS with Terraform
- :material-puzzle: [**Use Cases**](use_cases.md) — Integration examples with popular tools
- :material-cash-multiple: [**Cost Management**](operations_cost_management.md) — Model, infrastructure, and license costs, and per-request cost estimation
- :material-email-outline: [**Contact**](contact.md) — Technical questions, sales, and private offers
---
## Responses
Source: https://stdapi.ai/md/api_openai_responses.md
# Responses API
Generate model responses with Amazon Bedrock foundation models through an OpenAI Responses API-compatible interface. Supports text, images, tool calling, and streaming.
## Why Choose the Responses API?
- :material-tools: __Tool Calling__
Define function tools and get structured tool calls back. Full round-trip support with `function_call_output`.
- :material-code-json: __Structured Output__
Request JSON object or JSON schema output via `text.format` to get machine-readable responses.
- :material-lightning-bolt: __Streaming__
Real-time token streaming with granular events for text deltas, tool calls, and lifecycle milestones.
- :material-brain: __Extended Reasoning__
Enable chain-of-thought reasoning on supported models via `reasoning.effort`.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|-------------------------------------------|----------|--------------------------------------------------------------------------|-------------------------------------------|--------------------------------|
| `/v1/responses` | `POST` | Create a model response | Amazon Bedrock Converse API · Amazon Bedrock Mantle | `openai_response` |
| `/v1/responses/input_tokens` | `POST` | Count input tokens without generating a response | Amazon Bedrock CountTokens API | `openai_response_input_tokens` |
| `/v1/responses/compact` | `POST` | Compact a conversation into a reusable summary | Amazon Bedrock Converse API | `openai_response_compact` |
| `/v1/responses/{response_id}` | `GET` | Retrieve a stored response | Amazon Bedrock Sessions · Bedrock Mantle | `openai_response_get` |
| `/v1/responses/{response_id}` | `DELETE` | Delete a stored response | Amazon Bedrock Sessions · Bedrock Mantle | `openai_response_delete` |
| `/v1/responses/{response_id}/cancel` | `POST` | Cancel a background response — see [Stored Responses](#stored-responses) | Amazon Bedrock Sessions · Bedrock Mantle | `openai_response_cancel` |
| `/v1/responses/{response_id}/input_items` | `GET` | List the input items of a stored response | Amazon Bedrock Sessions | `openai_response_input_items` |
## Feature Compatibility
| Feature | Status | Notes |
|-----------------------------------------------------------------------|:---------------------------------------:|------------------------------------------------------------------------------|
| **Input** | | |
| Plain text (`input` as string) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Simple string shorthand for a single user message |
| Structured message array | :material-check-circle:{ .success role="img" aria-label="Supported" } | Array of `EasyInputMessage` / `InputMessage` items |
| `instructions` (system prompt) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Injected as a Bedrock system block |
| `system` / `developer` role | :material-check-circle:{ .success role="img" aria-label="Supported" } | Treated as a system instruction |
| Image input (`input_image`) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | HTTP URLs and base64 data URIs supported |
| File input (`input_file`) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | File URLs and base64 data supported |
| `function_call_output` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Submit tool results as input; supports text, image, and file parts |
| Echoed output items (message, reasoning, refusal) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Replayed to the model; refusal parts preserved; unknown upstream fields tolerated |
| Echoed `custom_tool_call` / `image_generation_call` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Replayed as tool calls (freeform input wrapped as `{"input": ...}`; image results attached) |
| Echoed `file_search_call` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Replayed as a search the model already ran, with the passages it returned; an item carrying no `results` is dropped with its call |
| Hosted-tool call items (`web_search_call`, `code_interpreter_call`, `computer_call`, `tool_search_call`, shell/apply-patch/MCP items, `compaction_trigger`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Input-history tolerance only: echoed items are accepted and dropped on replay (no Bedrock equivalent). Whether each *tool* can actually be used is listed under Tool Calling below |
| Echoed `program` / `program_output` items | :material-check-circle:{ .success role="img" aria-label="Supported" } | Input-history tolerance only: accepted and dropped on replay (no Bedrock equivalent) |
| `item_reference` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Accepted and dropped on replay |
| **Tool Calling** | | |
| Function tools (`type: "function"`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Full schema mapping to Bedrock toolSpec |
| `tool_choice: "auto"` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Model selects among available tools |
| `tool_choice: "required"` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Model must call at least one tool |
| `tool_choice: "none"` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Prevents tool calls |
| Named `tool_choice` (force) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Force a specific function to be called |
| `tool_choice: allowed_tools` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Approximated: `required` + 1 function → forced tool; `required` + many → any tool; `auto` → auto; type-variants add no constraint |
| `parallel_tool_calls` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Accepted for every model and honored by models able to constrain tool use; echoed in the response, which reports the tool calls actually made |
| Built-in tools (`code_interpreter`, `web_search`, `image_generation`) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | See [OpenAI Integrated Tools](#openai-integrated-tools) |
| `file_search` tool | :material-check-circle:{ .success role="img" aria-label="Supported" } | Served from the vector stores named in `vector_store_ids` (see [File Search](#file-search)); forwarded upstream on Bedrock Mantle native models |
| `web_search` `filters.allowed_domains` / `user_location` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Rejected with a `400` where the search cannot be restricted; honored on Bedrock Mantle native models |
| `web_search` `search_context_size` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Accepted and ignored — the answer is still searched and cited; honored on Bedrock Mantle native models |
| `computer` / `computer_use_preview` tools | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | No Converse equivalent — accepted and dropped (see [Computer Use Not Supported](#computer-use-not-supported)); forwarded upstream on Bedrock Mantle native models |
| `mcp` tool | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | No Converse equivalent — accepted and dropped; forwarded upstream on Bedrock Mantle native models |
| `local_shell` / `shell` tools | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted and dropped; no Bedrock equivalent |
| `custom` / `namespace` / `tool_search` / `apply_patch` tools | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted and dropped; no Bedrock equivalent |
| `programmatic_tool_calling` tool / `tool_choice` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | No Converse equivalent — accepted and dropped, the model calls the declared tools directly (the `tool_choice` degrades to the model's default choice); forwarded upstream on Bedrock Mantle native models |
| **Generation Control** | | |
| `max_output_tokens` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Maps to Bedrock `maxTokens` |
| `temperature` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | 0–2 range; mapped to Bedrock inference config |
| `top_p` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | 0–1 range; nucleus sampling |
| `top_logprobs` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | 0–20 range accepted and echoed; log probabilities are never returned on Converse-served models; forwarded upstream on Bedrock Mantle native models |
| `reasoning` (effort) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Configures reasoning; without `effort` defaults to `medium`; `effort: "none"` disables; chain of thought returned as `reasoning` output items |
| `reasoning.summary` / `generate_summary` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Accepted but ignored on Converse-served models — no summary is generated; forwarded upstream on Bedrock Mantle native models |
| `reasoning.context` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Accepted but ignored on Converse-served models — context scoping is not applied; forwarded upstream on Bedrock Mantle native models |
| `reasoning.mode` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Accepted but ignored on Converse-served models — pro-mode reasoning selection is not applied; forwarded upstream on Bedrock Mantle native models |
| `text.verbosity` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Accepted but ignored on Converse-served models; forwarded upstream on Bedrock Mantle native models |
| `include` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | `reasoning.encrypted_content` is honored; other values are accepted and ignored (forwarded upstream on Bedrock Mantle native models) |
| `metadata` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Forwarded to Bedrock `requestMetadata` |
| `prompt_cache_key` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Cache prompts to reduce costs and latency |
| `prompt_cache_options` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | `mode: "explicit"` caches only the parts marked with `prompt_cache_breakpoint`; `ttl: "30m"` mapped to a 1 hour Amazon Bedrock retention on Anthropic models (other models use the default 5 minute TTL) when `prompt_cache_retention` is unset; echoed on the response; forwarded upstream on Bedrock Mantle native models |
| `prompt_cache_breakpoint` (input content part) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Explicit cache boundary mapped to an Amazon Bedrock `cachePoint` (max. 4 per request) |
| `prompt_cache_retention` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Cache TTL: `in_memory`, `24h`, `1h`, or `5m` |
| `service_tier` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Maps to Bedrock service tier header |
| Extra model-specific params | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Extra model-specific parameters not supported by the OpenAI API |
| `truncation` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | `disabled` (the OpenAI default) is the behavior served, and is accepted; `auto` returns `400` |
| `max_tool_calls` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Returns `400`; not supported |
| `context_management` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Returns `400`; not supported |
| `background` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Accepted but ignored on Converse-served models — execution is synchronous; forwarded upstream on Bedrock Mantle native models, where background responses can be cancelled — see [Stored Responses](#stored-responses) |
| `store` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Persists the response — Amazon Bedrock session storage (non-streaming) or Mantle native storage for Mantle models (streaming supported) |
| `stream_options` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Accepted but ignored on Converse-served models; forwarded upstream on Bedrock Mantle native models |
| `conversation` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Prepends the conversation's items to `input` and appends the turn to it unless `store` is false; rejected (`400`) with `previous_response_id` — see [Conversations](api_openai_conversations.md) |
| `prompt` (template reference) | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Amazon Bedrock Prompt Management prompt ARN only, when enabled server-side — see [Managed Prompt Templates](#managed-prompt-templates) |
| `safety_identifier` / `user` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Does not affect generation; identifies the end user in the request log and in [per-user cost attribution](operations_cost_management.md#per-user-attribution) |
| `client_metadata` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Accepted but ignored on Converse-served models (sent by newer OpenAI clients such as Codex); forwarded upstream on Bedrock Mantle native models |
| `moderation` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Applies an Amazon Bedrock guardrail; results in the response `moderation` field (on the terminal event when streaming) — rejected (`400`) on Mantle-served models |
| **Output Format** | | |
| `text.format: "text"` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Plain text output |
| `text.format: "json_object"` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Accepted for all models; syntactically valid JSON is not guaranteed for every model |
| `text.format: "json_schema"` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Structured JSON output with schema validation |
| **Multi-Turn** | | |
| `previous_response_id` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Continues a response stored with `store=true` |
| Compaction (`POST /v1/responses/compact`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Stateless summary item; send it back in `input` to continue |
| **Streaming** | | |
| `stream: true` | :material-check-circle:{ .success role="img" aria-label="Supported" } | SSE stream with full lifecycle events |
| `response.created` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Emitted at stream start |
| `response.in_progress` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Emitted after created |
| `response.output_text.delta` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Text token deltas |
| `response.output_text.done` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Final text for each content part |
| `response.function_call_arguments.delta` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Tool call argument deltas |
| `response.function_call_arguments.done` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Finalized tool call arguments |
| `response.reasoning_text.delta` / `.done` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Reasoning text deltas on reasoning models |
| `response.output_text.annotation.added` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | `url_citation` annotations as web-search citations arrive |
| `response.web_search_call.in_progress` / `.searching` / `.completed` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Bracket each `web_search_call` item, in that order |
| `response.completed` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Final event when generation finishes normally |
| `response.incomplete` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Final event when output is truncated or filtered (no `response.completed`) |
| `response.failed` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Final event when generation fails; the response carries `error` |
| `error` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Spec error event on mid-stream failures, followed by `response.failed` |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Model-Dependent** — Behavior depends on the model or backend; check the Notes column
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
!!! note "Bedrock Mantle passthrough"
On [Mantle](features.md#bedrock-mantle-models) models served natively by the upstream Responses API, the parameters that the Converse path accepts but ignores — `background`, `include` (values other than `reasoning.encrypted_content`), `stream_options`, `reasoning.summary`, `text.verbosity`, `client_metadata`, `top_logprobs` — the hosted tools (`file_search`, `code_interpreter`, `computer`, `mcp`, `image_generation`) and the `web_search` search options (`filters`, `search_context_size`, `user_location`) are forwarded verbatim upstream: the upstream API decides whether they take effect or return a clean error. Web access is the exception: on every Mantle model the server decides whether a search may reach the external web, and unless the deployment allows the override, a request asking for a different value is rejected with a `400` — see [OpenAI GPT web search](#openai-gpt-web-search), where the tool is served natively.
## Model Support
All models supported by the Amazon Bedrock Converse and Converse Stream API are supported, plus every model served by [Bedrock Mantle](features.md#bedrock-mantle-models) when enabled — including OpenAI GPT-5.x, xAI Grok, and Google Gemma 4. Requests to Mantle models are passed through natively or converted automatically depending on the model's upstream API support.
!!! note "Project attribution (`OpenAI-Project`)"
Mantle requests can be attributed to a Bedrock Project for cost tracking and observability with the `OpenAI-Project: ` header (a bare project ID such as `proj_abc123`, not an ARN). It is honored per-request only when [`AWS_BEDROCK_ALLOW_MANTLE_PROJECT_OVERRIDE`](operations_configuration.md#bedrock-allow-mantle-project-override) is `true`; otherwise the server default ([`AWS_BEDROCK_MANTLE_PROJECT`](operations_configuration.md#bedrock-mantle-project)) applies. This applies **only** to models served by the Bedrock Mantle endpoint — classic `bedrock-runtime` models ignore the header.
## Advanced Features
### System Prompt (`instructions`)
Use `instructions` to define the assistant's behavior — it is injected as a Bedrock system block.
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"instructions": "You are a helpful assistant that answers in French.",
"input": "Say hello."
}'
```
### Function Tool Calling
Define function tools and submit results in a round-trip conversation.
!!! note "Multi-Turn Conversations"
For multi-turn conversations, pass the full message history in the `input` array, or store a response with `store=true` and continue it via `previous_response_id` (see [Stored Responses](#stored-responses)).
!!! warning "Unsupported Built-In Tools"
`computer`, `computer_use_preview`, `mcp`, `local_shell`, `shell`, `custom`,
`namespace`, `tool_search`, and `apply_patch` tools have no backend
equivalent: they are **accepted for compatibility and dropped** from the tool
configuration, so the model cannot call them.
`file_search` is the exception: it is **served** from the vector stores the
request names — see [File Search](#file-search).
!!! warning "Programmatic Tool Calling"
The `programmatic_tool_calling` tool — and `tool_choice: {"type": "programmatic_tool_calling"}` —
has no Bedrock Converse equivalent: the tool is **accepted and dropped** from
the tool configuration, and the `tool_choice` degrades to the model's default
choice. The request still succeeds and the model calls the declared tools
**directly**, one round trip at a time, instead of orchestrating them from
generated code, so no `program` or `program_output` items are returned.
Tools restricted to `allowed_callers: ["programmatic"]` remain exposed as
regular directly-callable tools. Bedrock Mantle native models receive the
parameters unchanged and serve programmatic tool calling themselves when the
model supports it.
Echoed `program` and `program_output` history items are also accepted (and
dropped on replay) so conversations recorded against the real API can be
replayed; the paired `function_call` items carry the actual tool traffic.
**Step 1 — Request a tool call:**
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"input": "What'\''s the weather in Paris?",
"tool_choice": "required",
"tools": [
{
"type": "function",
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]
}
}
]
}'
```
**Step 2 — Submit the tool result:**
Without `previous_response_id`, each request is stateless: replay the full
history, including the `function_call` item from step 1's output alongside
its matching `function_call_output`.
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"input": [
{
"role": "user",
"content": "What'\''s the weather in Paris?"
},
{
"type": "function_call",
"call_id": "",
"name": "get_weather",
"arguments": "{\"city\": \"Paris\"}"
},
{
"type": "function_call_output",
"call_id": "",
"output": "{\"temperature\": \"18°C\", \"condition\": \"cloudy\"}"
}
],
"tools": [
{
"type": "function",
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]
}
}
]
}'
```
### Streaming
Real-time token streaming with granular SSE lifecycle events.
```bash
curl -N -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"input": "Tell me a short story.",
"stream": true
}'
```
The stream emits events in order: `response.created` → `response.in_progress` → `response.output_item.added` → `response.content_part.added` → `response.output_text.delta` (repeated) → `response.output_text.done` → `response.content_part.done` → `response.output_item.done` → a terminal event.
The terminal event matches the outcome, exactly like the OpenAI API:
- `response.completed` — generation finished normally.
- `response.incomplete` — output was cut short (e.g. `max_output_tokens` reached or content filtered); `response.completed` is **not** emitted, and the SDK's `get_final_response()` raises accordingly.
- `response.failed` — generation failed; the embedded response carries an `error` object.
If an error occurs mid-stream, a spec `error` event (with `code`, `message`, `param`, and `sequence_number`) is emitted, followed by a terminal `response.failed` snapshot.
!!! note "Non-streaming failures"
Without `stream: true` there is no terminal event to carry the failure, so a
generation that ends in `status: "failed"` is reported as an HTTP `502` whose
`error.message` is the failure reason — never a `200` with an empty `output`.
Only `background: true` requests keep the `200` with the `failed` response
body, so the client can poll the stored response.
### Structured JSON Output
Request machine-readable output using `text.format`.
**JSON object:**
`json_object` is not schema-validated: the gateway asks the model to reply with a
JSON object on your behalf, but nothing constrains its decoding, so this is a
best-effort guarantee rather than a hard one. Including the word "JSON" in the
input, as below, still helps the model comply. Use `json_schema` when the
response must conform to a specific shape.
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"input": "Return the current date and day of week as JSON.",
"text": {"format": {"type": "json_object"}}
}'
```
**JSON schema:**
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"input": "What is 2 + 2? Reply with answer and confidence.",
"text": {
"format": {
"type": "json_schema",
"name": "MathResult",
"schema": {
"type": "object",
"properties": {
"answer": {"type": "number"},
"confidence": {"type": "number"}
},
"required": ["answer", "confidence"]
}
}
}
}'
```
### Extended Reasoning
Enable chain-of-thought reasoning on supported models (e.g. Amazon Nova 2, Anthropic Claude 3.7+) via `reasoning.effort`.
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openai.gpt-5.6-sol",
"input": "Solve: if a train travels 120 km in 90 minutes, what is its speed?",
"reasoning": {"effort": "low"},
"max_output_tokens": 4096
}'
```
The chain of thought is returned as a `reasoning` output item preceding the assistant message, with the text in `content` parts of type `reasoning_text`. Streaming shape and token accounting depend on the serving path:
- **Converse-served models**: the item streams through `response.output_item.added`, `response.reasoning_text.delta` / `.done`, and `response.output_item.done` events before the message events. Bedrock does not split reasoning tokens out of `outputTokens`, so `usage.output_tokens_details.reasoning_tokens` is always `0`; reasoning tokens are still billed inside `output_tokens`.
- **Mantle-native models**: the reasoning is returned as an encrypted-content item with no plaintext `reasoning_text.delta` events — only `response.output_item.added` / `.done` bracket it. The reasoning-token split reported in `usage` is whatever the upstream API returns.
- **Mantle models converted to another API**: the chain of thought comes back like the Converse path — `content` parts of type `reasoning_text`, opened with `response.content_part.added`, streamed through `response.reasoning_text.delta` / `.done`, and closed with `response.content_part.done` — not as `summary` events.
Add `"include": ["reasoning.encrypted_content"]` to attach an `encrypted_content` envelope to each reasoning item. Echo the item back in the `input` of the next request to carry the model's reasoning state (including signatures and redacted content) across turns with no server-side storage — reasoning items from the official OpenAI API are accepted too, with their encrypted content safely ignored. Echoing a reasoning item back **without** its `encrypted_content` replays it unsigned: Anthropic Claude models only continue from a thinking passage they can recognise as their own, so that reasoning is left out of the turn — the request still succeeds, only the earlier chain of thought is no longer visible to the model. Every other model family receives the unsigned reasoning as-is.
### Prompt Caching
!!! warning "Cache Creation Costs"
Cache creation incurs a higher cost than regular token processing. Only use prompt caching when you expect a high cache hit ratio across multiple requests with similar prompts.
Prompt caching reduces latency and costs by caching repetitive prompt components. Set the `prompt_cache_key` parameter to enable:
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openai.gpt-5.6-sol",
"instructions": "You are a helpful assistant.",
"input": "What is Python?",
"prompt_cache_key": "default"
}'
```
**Granular Cache Control:**
Use dot-separated values to cache specific components:
- `"system"` — Cache system messages only
- `"messages"` — Cache conversation history
- `"tools"` — Cache tool/function definitions (Anthropic Claude only)
- `"system.messages"` — Cache both system and messages
- `"system.tools"` — Cache system and tools
- `"messages.tools"` — Cache messages and tools
- `"system.messages.tools"` — Cache all components
- Any other non-empty value — Cache all components
!!! note "Custom Cache Keys Not Supported"
Custom cache hash keys are not supported. The parameter is used only to control which sections are cached, not as a cache identifier.
**Example — Cache system and tools:**
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-sonnet-5",
"instructions": "You are a data analysis assistant.",
"input": "Analyze this dataset: ...",
"tools": [{"type": "function", "name": "run_sql", ...}],
"prompt_cache_key": "system.tools"
}'
```
**Benefits:**
- **Cost Reduction**: Cached tokens are billed at a lower rate than regular input tokens
- **Lower Latency**: Cached prompts eliminate reprocessing time
- **Automatic Management**: The API handles cache invalidation and updates
**Cache Retention (TTL):**
Control how long cached prompts persist using the `prompt_cache_retention` parameter:
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openai.gpt-5.6-sol",
"input": "Hello",
"prompt_cache_key": "default",
"prompt_cache_retention": "24h"
}'
```
Valid values: `in_memory` (default), `24h`, `1h`, or `5m`. The `1h` and `5m` values are Amazon Bedrock-specific. On Amazon Bedrock, `in_memory` maps to 5 minutes and `24h` maps to 1 hour. The `prompt_cache_options.ttl` value `"30m"` is mapped to a 1 hour retention on Anthropic models (other models use the default 5 minute TTL) when `prompt_cache_retention` is unset.
**Explicit Cache Breakpoints:**
Instead of relying on the `prompt_cache_key` section heuristics, mark the exact cache boundaries with `prompt_cache_breakpoint` on any input content part (`input_text`, `input_image`, `input_file`). Each marked part is followed by an Amazon Bedrock `cachePoint`, so the prompt prefix ending with that part is cached:
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-sonnet-5",
"prompt_cache_options": {"mode": "explicit"},
"input": [
{
"role": "user",
"content": [
{
"type": "input_text",
"text": "Long reusable context...",
"prompt_cache_breakpoint": {"mode": "explicit"}
},
{"type": "input_text", "text": "Summarize it."}
]
}
]
}'
```
- `"mode": "explicit"` caches **only** the marked parts: the `prompt_cache_key` heuristics are disabled for that request.
- `"mode": "implicit"` (default) keeps the `prompt_cache_key` heuristics **and** honors the marked parts.
- At most 4 cache points are sent per request (Amazon Bedrock limit); the oldest ones are dropped when more are requested.
- Breakpoints on models without prompt caching support are accepted and ignored, as are breakpoints on tool output items — those never become a cache point, whatever the model.
- `prompt_cache_options` is echoed back on the response object.
!!! note "Model Support"
Cache retention configuration is only available on select models. See [Amazon Bedrock Prompt Caching - Supported Models](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html#prompt-caching-models) for details on which models support configurable TTL.
Cached token usage is reported in the response:
```json
{
"usage": {
"input_tokens": 1500,
"input_tokens_details": {
"cached_tokens": 1200
},
"output_tokens": 300,
"total_tokens": 1800
}
}
```
Following OpenAI semantics, `input_tokens` covers the **full** prompt: tokens read from and written to the cache are included, and `cached_tokens` (the tokens read from cache) is a subset of `input_tokens`. In this example, 1,200 of the 1,500 input tokens were retrieved from cache.
### Managed Prompt Templates { #managed-prompt-templates }
The `prompt` parameter references a prompt template stored in [Amazon Bedrock Prompt Management](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-management.html). Amazon Bedrock renders the template server-side, so the request body carries only the variable values.
!!! warning "Disabled by Default"
`prompt` returns `400` unless the server operator sets [`AWS_BEDROCK_ALLOW_PROMPT_ARN`](operations_configuration.md#bedrock-allow-prompt-arn) to `true`.
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-sonnet-5",
"prompt": {
"id": "arn:aws:bedrock:us-east-1:123456789012:prompt/ABCDE12345",
"version": "1",
"variables": {"genre": "pop", "number": "3"}
}
}'
```
:octicons-key-24: **Requirements**
- `prompt.id` must be an Amazon Bedrock prompt ARN. OpenAI-hosted prompt template IDs (`pmpt_…`) do not exist on this gateway and return `400`.
- `prompt.version` is an Amazon Bedrock version number. It is appended to the ARN, and must not disagree with a version already present in `prompt.id`. Omit it to run the working draft.
- The prompt must be a **TEXT** prompt bound to a model that this server can serve, and `model` must be that exact model: it is the model used for response formatting and cost attribution. The error message names the model the prompt uses.
- `prompt.variables` values must be plain strings — Amazon Bedrock prompt variables only carry text, so image, file and structured content parts return `400`.
- The prompt's region is derived from its ARN and must be a configured Amazon Bedrock region; cross-region failover is disabled for the request.
- The model must be served by the Converse API: [Mantle](features.md#bedrock-mantle-models) native models return `400`, as they have no Prompt Management equivalent.
:octicons-x-circle-24: **Rejected Alongside `prompt`**
The stored prompt version already provides the conversation, the system prompt, the tools and the inference parameters, so combining `prompt` with `input`, `instructions`, `tools`, `tool_choice`, `text`, `temperature`, `top_p`, `max_output_tokens`, `reasoning` or `previous_response_id` returns `400` instead of silently dropping them.
Streaming, `store`, `moderation` and guardrail headers remain available. The remaining request-level parameters (`metadata`, `service_tier`, `prompt_cache_*`, …) are accepted and echoed on the response, but not applied: the Bedrock call carries only the prompt resource and its variables.
### File Search { #file-search }
`file_search` lets any chat model answer from the files you indexed in a
[vector store](api_openai_vector_stores.md), with no change to your client code:
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-2-lite-v1:0",
"input": "How many vacation days do I get?",
"tools": [{
"type": "file_search",
"vector_store_ids": ["vs_abc123"]
}]
}'
```
The model decides when to search and with which query. Each search it runs is
reported as a `file_search_call` output item — carrying the `queries` used —
ahead of the `message` it grounded. When streaming, that item is framed by
`response.file_search_call.in_progress`, `.searching` and `.completed`, and the
answer streams after it.
The retrieved passages are **not** returned unless you ask for them:
```json
{
"model": "amazon.nova-2-lite-v1:0",
"input": "How many vacation days do I get?",
"include": ["file_search_call.results"],
"tools": [{"type": "file_search", "vector_store_ids": ["vs_abc123"]}]
}
```
Each result then carries its `file_id`, `filename`, `text`, `score` and the
`attributes` stored with the file.
!!! note "Citations"
The grounded answer carries a `file_citation` annotation on its
`output_text` content for every file the passages were read from, each
naming the `file_id` and `filename`, so a client can attribute the answer
without asking for the passages themselves.
A model reports the passages it was given, never which sentence came from
which one, so there is one citation per file rather than one per passage,
and `index` is the end of the answer text rather than the position of a
cited span. When streaming, they arrive on the message item itself
(`response.output_item.done` and the terminal event) rather than as
separate `response.output_text.annotation.added` events.
**Narrowing the search:**
| Field | Behavior |
|---|---|
| `vector_store_ids` | Every store listed is searched and the best passages across all of them are kept. At least one is required, and a store this deployment does not serve answers `404` before the model is called |
| `max_num_results` | Passages kept per search, `1`–`50`; defaults to `20` |
| `filters` | Restricts the search to files carrying given `attributes`. A comparison operator the store cannot apply is refused with a `400` naming the ones it accepts |
| `ranking_options.score_threshold` | Drops passages below the score. Refused with a `400` on a store whose relevance scores are not comparable between searches |
| `ranking_options.ranker`, `ranking_options.hybrid_search` | Accepted and ignored — the passages are still ranked by relevance |
!!! note "Rounds per response"
The model may refine its query and search again; after two searches the
tool is withdrawn and the model answers with what it has, so one response
never loops indefinitely. Each round is a further model invocation and is
billed as such.
### OpenAI Integrated Tools
The Responses API supports OpenAI's built-in tool types, automatically mapped to the target model's native tools.
#### { style="height: 1.2em; vertical-align: text-bottom;" } Nova Tools
Nova models support web search and code execution as integrated tools.
**Web Search** (`web_search`, `web_search_preview`):
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-2-lite-v1:0",
"input": "What is the current version of Python?",
"tools": [{"type": "web_search"}]
}'
```
**Code Interpreter** (`code_interpreter`):
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-2-lite-v1:0",
"input": "Calculate the first 10 Fibonacci numbers",
"tools": [{"type": "code_interpreter"}]
}'
```
!!! note "Citations: sources and annotations"
Web-search citations surface in two places:
- `action.sources` on the `web_search_call` item (in streaming mode it is
`null` on intermediate events and populated on the terminal event, since
citations arrive after the tool call closes).
- `url_citation` annotations on the assistant message's `output_text`
content. When streaming, each citation also emits a
`response.output_text.annotation.added` event as it arrives.
On Nova models, Amazon Bedrock does not report character positions, so
`start_index` and `end_index` are approximated to the length of the
generated text at the time the citation arrived. The OpenAI GPT-5.x family
reports its own spans — see
[OpenAI GPT Web Search](#openai-gpt-web-search).
!!! warning "Search options"
`filters.allowed_domains` and `user_location` restrict which sources a
search may use, and Nova's grounding cannot apply either: a request
carrying one is **rejected with a `400`** rather than searched
unrestricted. `search_context_size` is accepted and ignored — the answer is
still searched and cited. All three are honored on the models that serve
web search natively, listed below.
The `external_web_access` extra model parameter is refused the same way: a
Nova search runs with the web access the server is configured for, so a
request asking for a different one is **rejected with a `400`** instead of
being searched under the server's value. Sending the configured value, or
omitting the parameter, always works.
!!! warning "Region Compatibility"
`web_search` is available on Amazon Nova 2 and Nova Premier models, in US regions only. Not available on EU inference profiles.
#### { style="height: 1.2em; vertical-align: text-bottom;" } OpenAI GPT Web Search
The OpenAI GPT-5.x family answers `web_search` with the search tool built into
Amazon Bedrock. The model decides when a question needs current information,
runs one or more queries, and grounds its answer in what it finds.
!!! warning "Amazon Bedrock Mantle only"
Amazon Bedrock serves this tool on the Mantle endpoint alone; it is refused
on the `bedrock-runtime` endpoint. Models offered on both — the GPT-5.6
family among them — resolve to their runtime twin by default, which cannot
answer `web_search`: the request is **rejected with a `400`** naming both
ways to reach Mantle, rather than answered without a search. Send it to
Mantle explicitly, with the `x-stdapi-service` header below or by naming the
model in
[`AWS_BEDROCK_MANTLE_PREFERRED_MODELS`](operations_configuration.md#bedrock-mantle-preferred-models).
Available in `us-east-1`, `us-east-2` and `us-west-2`, and billed per query.
`code_interpreter` is refused the same way and for the same reason: no
OpenAI GPT server tool is served on `bedrock-runtime`.
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-stdapi-service: bedrock-mantle" \
-d '{
"model": "openai.gpt-5.6-luna",
"input": "What are the most significant AWS launches announced this month?",
"tools": [{"type": "web_search"}]
}'
```
Each search appears as a `web_search_call` output item, and every grounded
statement carries a `url_citation` annotation on the assistant message's
`output_text` content, whose `start_index` and `end_index` delimit the answer
text it supports. When streaming, the lifecycle arrives as
`response.web_search_call.in_progress` / `.searching` / `.completed`, and each
citation as a `response.output_text.annotation.added` event.
!!! info "External web access"
Searches are answered from the Amazon Bedrock web index and cache, and
results are current and cited either way.
[`AWS_BEDROCK_EXTERNAL_WEB_ACCESS`](operations_configuration.md#bedrock-external-web-access)
controls whether a search *may* reach the external web, and it takes the
`bedrock-websearch:ExternalWebAccess` permission as well — see
[Web Search IAM](operations_iam_permissions.md#web-search-iam).
AWS
[documents](https://docs.aws.amazon.com/bedrock/latest/userguide/web-search.html)
that retrieval is served entirely from that index and cache today, so no
request data leaves the AWS boundary even when the permission is granted, and
that a future release may allow live external retrieval — at which point
request data may leave it. Enabling this is therefore an advance decision
about behaviour that can change: leave it off unless you intend that.
A request may choose its own web access by sending `external_web_access` as
an extra model parameter (a top-level field, or `extra_body` in the OpenAI
SDK), and only when
[`AWS_BEDROCK_ALLOW_EXTERNAL_WEB_ACCESS_OVERRIDE`](operations_configuration.md#bedrock-allow-external-web-access-override)
is enabled; otherwise a value differing from the configured one is rejected
with a `400`. These are the models whose search takes the choice per
request: everywhere else the parameter must match the server's value.
```json
{
"model": "openai.gpt-5.6-luna",
"input": "What shipped this week?",
"tools": [{"type": "web_search"}],
"external_web_access": true
}
```
!!! warning "Availability and cost"
Web search is available on the OpenAI GPT-5.x models in `us-east-1`,
`us-east-2` and `us-west-2`. Each search runs in the Region that served the
model call and is never routed to another Region, so keep one of the three
in [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions):
a model served from anywhere else cannot search.
It is billed per query on top of the model's
tokens — see [Built-in tool pricing](operations_cost_management.md#built-in-tool-pricing).
It is offered on this endpoint only: requesting the equivalent server tool
on `/v1/messages` returns a `400`.
#### :material-image: Image Generation
The `image_generation` integrated tool works with **all text models** — Claude, Nova, and any future model. The gateway intercepts the tool, lets the LLM compose the image prompt and parameters via a synthetic function call, then generates the image against a configured Bedrock image model and returns an `image_generation_call` output item to the client. Intermediate `function_call` items are suppressed.
!!! warning "Configuration Required"
Set the [`IMAGE_GENERATION_MODEL`](operations_configuration.md#image-generation-model) environment variable to a Bedrock image model ID (e.g. `amazon.nova-canvas-v1:0`). The tool definition may also specify a `model` field to override the default per request.
**Example — Generate an image:**
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"input": "Generate a photorealistic image of a red panda sitting on a tree branch.",
"tools": [{"type": "image_generation"}],
"tool_choice": "required"
}'
```
The response contains an `image_generation_call` output item:
```json
{
"output": [
{
"type": "image_generation_call",
"id": "img_abc123",
"status": "completed",
"result": ""
}
]
}
```
You can also specify image parameters in the tool definition:
```json
{
"type": "image_generation",
"size": "1024x1024",
"quality": "high",
"output_format": "png"
}
```
!!! info "`quality` is set by you, not by the model"
The model chooses the prompt, size and output format for each call, but `quality` is read from the tool definition only. Most image models have no quality control and reject the parameter, so a value the model volunteered would fail the generation.
!!! info "`partial_images` is accepted and ignored"
`partial_images` (0-3) is accepted for OpenAI API compatibility but never acts: no available model streams partial images, so no `response.image_generation_call.partial_image` event is emitted and the finished image always arrives in a single `response.image_generation_call.completed` event. The same applies to [image generations](api_openai_images_generations.md) and [image edits](api_openai_images_edits.md).
#### Computer Use Not Supported
!!! warning "Computer Use Not Supported"
The `computer` and `computer_use_preview` integrated tools are **not supported**: requests succeed, but the tools are accepted and dropped from the tool configuration, so the model can never call them.
### Provider-Specific Parameters
A top-level field this API does not declare is forwarded to the model as a
provider-specific inference parameter, as on
[Chat Completions](api_openai_chat_completions.md#provider-specific-parameters)
and [Messages](api_anthropic_messages.md#provider-specific-parameters). A
capability Amazon Bedrock exposes and the OpenAI API has no field for is
therefore reachable without leaving this endpoint — the OpenAI SDK sends these
through `extra_body`:
```json
{
"model": "anthropic.claude-sonnet-5",
"input": "Write a poem about the sea",
"top_k": 50
}
```
Server-wide defaults per model come from `DEFAULT_MODEL_PARAMS`, and a
per-request value wins over them.
**Behavior:**
- :material-check-circle:{ .success role="img" aria-label="Supported" } **Compatible parameters**: forwarded to the model and applied
- :material-alert-circle:{ .warning } **Unsupported parameters**: the backend refuses the request, returned as a `400`
- :material-alert-circle:{ .warning } **Reserved names**: `additional_request_fields`, `max_tokens`, `model_id`, `stop_sequences`, `temperature`, `top_logprobs` and `top_p` are the argument names the gateway binds when it builds the Bedrock call, so sending one as an extra is rejected with a `400` naming it instead of binding twice — use the declared `max_output_tokens`, `temperature`, `top_p` and `top_logprobs` fields
- :material-alert-circle:{ .warning } **Client-side control fields**: names no provider treats as inference parameters (LiteLLM's `drop_params` among them) are dropped before the call. [`EXTRA_MODEL_PARAMS_DENYLIST`](operations_configuration.md#extra-model-params-denylist) extends that list, and [`EXTRA_MODEL_PARAMS_DROP_ALL`](operations_configuration.md#extra-model-params-drop-all) disables the passthrough entirely
## Input Token Counting
Count input tokens without generating a response. Useful for estimating costs or checking context-window fit before making a full response call.
**Basic usage:**
```bash
curl -X POST "$BASE/v1/responses/input_tokens" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"input": "Hello, how are you?"
}'
```
**Response:**
```json
{
"object": "response.input_tokens",
"input_tokens": 142
}
```
**With instructions and tools:**
```bash
curl -X POST "$BASE/v1/responses/input_tokens" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"input": "What is the weather?",
"instructions": "You are a helpful assistant.",
"tools": [{"type": "function", "name": "get_weather", "description": "Get weather for a location", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}]
}'
```
!!! note "Limitations"
The `previous_response_id` and `conversation` parameters are not supported for token counting (they would change the count); `personality` (a token-counting-only schema field) and `reasoning.context` are accepted and ignored. Token counting is not available for models served by [Amazon Bedrock Mantle](features.md#bedrock-mantle-models) (the request is rejected with a `400` error).
## Stored Responses
Set `store: true` to persist a response in [Amazon Bedrock session storage](https://docs.aws.amazon.com/bedrock/latest/userguide/sessions.html): one AWS-managed session per stored response, encrypted at rest (optionally with [your own KMS key](operations_configuration.md#aws-bedrock-session-encryption-key-arn)), with no state on the server itself.
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "amazon.nova-micro-v1:0", "input": "Hello!", "store": true}'
```
The returned `id` then works with:
- `GET /v1/responses/{response_id}` — retrieve the stored response.
- `GET /v1/responses/{response_id}/input_items` — list the input items that produced it. Bedrock Mantle native storage does not serve input item listings: for Mantle-stored responses this returns `404` with an explanatory message.
- `DELETE /v1/responses/{response_id}` — delete it (and its Amazon Bedrock session).
- `POST /v1/responses/{response_id}/cancel` — for Mantle region-tagged IDs, proxied to Bedrock Mantle (background responses are cancellable upstream); for Bedrock-session-stored responses it fails with the OpenAI synchronous-response error since execution is synchronous.
- `previous_response_id` on a new request — continue the conversation: the stored input and output are automatically prepended to the new input (instructions are not carried over, per the OpenAI API).
!!! warning "Response IDs are stdapi.ai-specific"
Response IDs embed the serving AWS region, so conversation turns chained with `previous_response_id` stay region-local. These IDs **cannot** be used directly against the Bedrock Mantle API, and raw [Mantle](features.md#bedrock-mantle-models) response IDs are not accepted by stdapi.ai.
!!! note "Behavior notes"
- `store` defaults to **false** on this implementation (the OpenAI API defaults to true).
- `POST /v1/responses/input_tokens` and `POST /v1/responses/compact` cannot reference a Mantle-stored response via `previous_response_id` — like input-item listings, Mantle native storage does not serve the stored items back.
- On Amazon Bedrock session storage, `store=true` is ignored with `stream=true` (a warning is recorded in the request log). [Mantle](features.md#bedrock-mantle-models) models persist responses in Mantle native storage instead, where `store` works with streaming too.
- Mantle models without native Responses storage (Messages- or Chat-Completions-bound) use Amazon Bedrock session storage like classic models. Only a `store=true` request answered through a mid-request API fallback (away from the upstream Responses API) is served without storage, with a warning recorded in the request log; its ID cannot be retrieved later. `previous_response_id` on such a fallback returns `400` instead — conversation history is never silently dropped.
- Sessions are created in the primary Bedrock region and persist until deleted through the API — see [operator guidance on cleaning up stale sessions](operations_configuration.md#bedrock-session-storage-optional).
- `GET /v1/responses/{response_id}` rejects `stream=true` with `400`; `include` and `starting_after` are accepted and ignored.
- Amazon Bedrock session storage is offered in fewer regions than model inference. Where the primary Bedrock region does not provide it, `store=true` is ignored and a warning naming the region as the cause is recorded in the request log — the response itself is still returned. Configure a primary region that provides session storage to avoid this entirely.
- Requires the Amazon Bedrock session management IAM permissions (`bedrock:CreateSession`, `bedrock:CreateInvocation`, `bedrock:PutInvocationStep`, `bedrock:GetInvocationStep`, `bedrock:ListInvocationSteps`, `bedrock:ListInvocations`, `bedrock:ListSessions`, `bedrock:ListTagsForResource`, `bedrock:EndSession`, `bedrock:DeleteSession`, `bedrock:TagResource`). Without them, `store=true` is ignored (with a request-log warning) and the response is not persisted.
## Conversation Compaction
Compact a long conversation into a single `compaction` item to keep multi-turn sessions within the context window. The model summarizes the provided `input`; the summary comes back as an opaque item that you include in the `input` of later requests instead of the full history.
```bash
curl -X POST "$BASE/v1/responses/compact" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-pro-v1:0",
"input": [
{"role": "user", "content": "..."},
{"role": "assistant", "content": "..."}
]
}'
```
**Response** (trimmed to the compaction item; `output` also echoes the conversation's message items before it):
```json
{
"id": "resp-...",
"object": "response.compaction",
"created_at": 1752000000,
"output": [
{"id": "ci-...", "type": "compaction", "encrypted_content": "..."}
],
"usage": {"input_tokens": 1500, "output_tokens": 220, "total_tokens": 1720}
}
```
Continue the conversation by sending the compaction item back, followed by new messages:
```json
{
"model": "amazon.nova-pro-v1:0",
"input": [
{"id": "ci-...", "type": "compaction", "encrypted_content": "..."},
{"role": "user", "content": "Next question..."}
]
}
```
!!! note "Stateless compaction"
The compaction content is fully self-contained (marker-prefixed and encoded, not encrypted): no conversation state is needed, and any server instance can expand it. Only compaction items produced by this server can be expanded — items encrypted by the upstream OpenAI API are rejected with `400`, and locally-produced items cannot be continued on a [Mantle](features.md#bedrock-mantle-models)-served model. `previous_response_id` may reference a [stored response](#stored-responses) to include its conversation in the compaction.
---
**Ready to build with AI?** Check out the [Models API](api_openai_models.md) to see all available foundation models!
## Available Request Headers
This endpoint supports standard Bedrock headers for enhanced control over your requests. All headers are optional and can be combined as needed.
### Content Safety (Guardrails)
| Header | Purpose | Valid Values |
|--------------------------------------|------------------------------------|---------------------------------------|
| `X-Amzn-Bedrock-GuardrailIdentifier` | Guardrail ID for content filtering | Your guardrail identifier |
| `X-Amzn-Bedrock-GuardrailVersion` | Guardrail version | Version number (e.g., `1`) |
| `X-Amzn-Bedrock-Trace` | Guardrail trace level | `disabled`, `enabled`, `enabled_full` |
### Performance Optimization
| Header | Purpose | Valid Values |
|--------------------------------------------|------------------------|-------------------------------|
| `X-Amzn-Bedrock-Service-Tier` | Service tier selection | `default`, `flex`, `priority`, `reserved` |
| `X-Amzn-Bedrock-PerformanceConfig-Latency` | Latency optimization | `standard`, `optimized` |
**Example with all headers:**
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Amzn-Bedrock-GuardrailIdentifier: your-guardrail-id" \
-H "X-Amzn-Bedrock-GuardrailVersion: 1" \
-H "X-Amzn-Bedrock-Trace: enabled" \
-H "X-Amzn-Bedrock-Service-Tier: priority" \
-H "X-Amzn-Bedrock-PerformanceConfig-Latency: optimized" \
-d '{
"model": "amazon.nova-micro-v1:0",
"input": "Hello!"
}'
```
!!! info "Detailed Documentation"
For complete information about these headers, configuration options, and use cases, see:
- [Bedrock Guardrails Configuration](operations_configuration.md#bedrock-guardrails)
- [Service Tier and Performance Configuration](operations_configuration.md#bedrock-service-tier-and-performance-configuration)
## Model-Specific Features
### { style="height: 1.2em; vertical-align: text-bottom;" } TwelveLabs Pegasus
`twelvelabs.pegasus-1-2-v1:0` is a video-understanding model. Because Pegasus accepts exactly one video and one text prompt per call, this API adapts the conversation automatically:
- The **latest video** found anywhere in the conversation (any role, any position) is forwarded as the video input.
- The **latest contiguous run of user text** (back to the previous assistant or tool turn) is concatenated and forwarded as the text prompt.
- `temperature` and `max_output_tokens` are forwarded.
- `text.format: json_schema` is forwarded as Pegasus's structured output.
**Silently ignored** (no error): system prompt, tools, `top_p`, stop sequences, and prompt caching.
**Upstream format limitation:** The OpenAI Responses API has no `input_video` content type in its stable spec. To stay fully compatible with standard OpenAI clients, pass the video as an **`input_image`** content item — the server detects the video MIME type automatically and routes it to Pegasus correctly.
**Video input formats**: `data:video/mp4;base64,…`, `https://…`, `s3://bucket/key`, or `file-id:…`. Videos above 18.75 MB are automatically uploaded to S3.
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "twelvelabs.pegasus-1-2-v1:0",
"input": [
{
"type": "message",
"role": "user",
"content": [
{"type": "input_image", "image_url": "s3://my-bucket/video.mp4"},
{"type": "input_text", "text": "Describe what happens in this video."}
]
}
]
}'
```
## Try It Now
**Basic response:**
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"input": "Say hello world"
}'
```
**Streaming response:**
```bash
curl -N -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"input": "Write a haiku about the sea.",
"stream": true
}'
```
**Multi-modal with image:**
```json
{
"model": "amazon.nova-micro-v1:0",
"input": [
{
"role": "user",
"content": [
{"type": "input_text", "text": "Describe this image"},
{"type": "input_image", "image_url": "https://example.com/photo.jpg"}
]
}
]
}
```
**With reasoning:**
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openai.gpt-5.6-sol",
"input": "Solve 12 × 13",
"reasoning": {"effort": "low"},
"max_output_tokens": 4096
}'
```
---
## Conversations
Source: https://stdapi.ai/md/api_openai_conversations.md
# Conversations API
Keep multi-turn state on the server. A conversation holds the items of an
exchange — user messages, model output, reasoning and tool calls — so each new
turn only has to send the new message. Pass a conversation ID as `conversation`
on a [Responses](api_openai_responses.md) request and both the request input and
the response output are added to it automatically.
## Why Choose the Conversations API?
- :material-chat-processing: __Send Only the New Turn__
The conversation's items become the input prefix of the next request, so a long exchange stays a one-message request.
- :material-playlist-plus: __Explicit Item Control__
Add, list, retrieve and delete items yourself, independently of any model call.
- :material-tag-multiple: __Attached Metadata__
Up to 16 key-value pairs per conversation, merged on update and removable key by key.
- :material-format-list-bulleted: __Cursor Pagination__
List items newest- or oldest-first with `limit` and the `after` cursor.
- :material-server-network: __No Gateway State__
The thread lives in your AWS account, not in a gateway instance, so any instance behind a load balancer serves any conversation.
## Available Endpoints
| Endpoint | Method | What It Does | MCP Tool |
|---------------------------------------------------|----------|-------------------------------------|-------------------------------------|
| `/v1/conversations` | `POST` | Create a conversation | `openai_conversation` |
| `/v1/conversations/{conversation_id}` | `GET` | Retrieve a conversation | `openai_conversation_get` |
| `/v1/conversations/{conversation_id}` | `POST` | Update the conversation's metadata | `openai_conversation_update` |
| `/v1/conversations/{conversation_id}` | `DELETE` | Delete a conversation and its items | `openai_conversation_delete` |
| `/v1/conversations/{conversation_id}/items` | `POST` | Add items to a conversation | `openai_conversation_items` |
| `/v1/conversations/{conversation_id}/items` | `GET` | List a conversation's items | `openai_conversation_items_list` |
| `/v1/conversations/{conversation_id}/items/{item_id}` | `GET` | Retrieve one item | `openai_conversation_item_get` |
| `/v1/conversations/{conversation_id}/items/{item_id}` | `DELETE` | Delete one item | `openai_conversation_item_delete` |
## Feature Compatibility
| Feature | Status | Notes |
|-------------------------------------------|:----------------------------------------:|-----------------------------------------------------------------------------|
| **Conversation** | | |
| `items` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Up to 20 initial items, prepared before the conversation is created, so a rejected item leaves no empty conversation behind |
| `metadata` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | The limits in [Metadata](#metadata). A key given a `null` value is accepted and dropped rather than stored; on an update that same `null` removes the key |
| A request with no body | :material-check-circle:{ .success role="img" aria-label="Supported" } | Both fields are optional, and the body itself may be omitted |
| Retrieve, update, delete | :material-check-circle:{ .success role="img" aria-label="Supported" } | `metadata` is the only field an update takes; a delete removes the conversation and every item it holds |
| **Items** | | |
| `items` on an add | :material-check-circle:{ .success role="img" aria-label="Supported" } | 1 to 20 per request; the response is a `list` envelope of the items added, not the whole conversation |
| Item shapes | :material-check-circle:{ .success role="img" aria-label="Supported" } | The [Responses](api_openai_responses.md) `input` and `output` items: messages, reasoning items, tool calls and their outputs |
| A message `content` sent as a string | :material-check-circle:{ .success role="img" aria-label="Supported" } | Expanded into an `input_text` or `output_text` part according to the message's `role` |
| An `id` sent on a new item | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Accepted and ignored — the server mints the identifier, prefixed by the item's type |
| `item_reference` items | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Resolved against the conversation and then dropped rather than stored again, since the item it names is already there; one naming an item that is not answers `404` |
| Retrieve and delete one item | :material-check-circle:{ .success role="img" aria-label="Supported" } | A delete returns the conversation, and the item leaves both the listing and the prefix of the next Responses turn |
| **Listing items** | | |
| `order`, `limit`, `after` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Bounds and cursor semantics in [Listing](#listing) |
| `include` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Accepted on the listing, on a retrieval and on an add. Only `reasoning.encrypted_content` changes the response; every other value is accepted and ignored |
| `first_id` / `last_id` / `has_more` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Populated on every page; the server never auto-paginates |
| **Lifecycle** | | |
| Conversation lifetime | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | 30 days from creation, after which every route on it answers `404`. Amazon Bedrock session storage sets the window and it cannot be extended |
| Adding and deleting items | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | 1,000 requests per conversation — see [Limits](#limits) |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
!!! note "No model is involved"
Every endpoint on this page is state management: items are stored and read
back as they were sent. Nothing here embeds, summarises or re-runs an item,
so no endpoint takes a model identifier and a conversation behaves the same
whatever the deployment's catalog holds. Summarising a long exchange is
[`POST /v1/responses/compact`](api_openai_responses.md#conversation-compaction),
on the Responses API.
## Quick Start
```python
from openai import OpenAI
client = OpenAI(base_url="https://your-gateway/v1", api_key="YOUR_API_KEY")
conversation = client.conversations.create(metadata={"topic": "travel"})
first = client.responses.create(
model="amazon.nova-micro-v1:0",
input="My favourite city is Lisbon.",
conversation=conversation.id,
)
second = client.responses.create(
model="amazon.nova-micro-v1:0",
input="Which city did I name?",
conversation=conversation.id,
)
print(second.output_text)
```
The second request carries no history: the conversation supplies it.
## Using a Conversation with the Responses API
| Request | Effect |
|--------------------------------------------|------------------------------------------------------------------------|
| `conversation="conv-..."` | The conversation's items are prepended to `input`; the request input and the response output are appended to the conversation. |
| `conversation={"id": "conv-..."}` | Same; both forms are accepted. |
| `conversation=...`, `store=false` | The conversation is still used as the input prefix, but nothing is added to it. |
| `conversation=...`, `stream=true` | Items are added once the stream has ended, and the terminal event carries the conversation. |
| `conversation=...` and `previous_response_id=...` | Rejected with `400` (`mutually_exclusive_parameters`) — pick one way of continuing the exchange. |
The response echoes the conversation it belongs to as `"conversation": {"id": "conv-..."}`.
A response that fails before generating anything adds nothing to the conversation.
`conversation` is also accepted on `/v1/responses/input_tokens`, where the
conversation's items are counted ahead of `input`.
## Items
Items use the same shapes as the Responses API `input` and `output`: messages,
reasoning items, function calls and their outputs.
- **Item IDs are assigned by the server.** An `id` sent on a new item is ignored.
- **Adding items returns the items that were added**, as a `list` envelope — not
the whole conversation.
- **`item_reference` items** point at an item already in the conversation; a
reference to an item that is not there returns `404`.
- **Deleting an item returns the conversation**, and the item disappears from
the listing.
- `include=reasoning.encrypted_content` returns the encrypted content of
reasoning items; other `include` values are accepted and ignored.
### Listing
| Parameter | Default | Notes |
|-----------|----------|--------------------------------------------------------------|
| `order` | `desc` | `asc` is conversation order. |
| `limit` | `20` | Up to 100 items per page; `0` returns an empty page. |
| `after` | — | An item ID; only items strictly after it are returned. An ID that is not in the conversation returns `404`. |
| `include` | — | Extra item fields to return. |
Each page carries `first_id`, `last_id` and `has_more`. Pass the page's
`last_id` as `after` to read the next page; the server never auto-paginates.
## Metadata
| Limit | Value |
|------------------------------|-------|
| Key-value pairs | 16 |
| Key length | 64 |
| Value length | 512 |
Updating **merges**: keys that are not sent keep their value, and a key sent as
`null` is removed. `metadata` is required on update — omitting it returns `400`
(`missing_required_parameter`), and sending `null` returns `400`
(`invalid_type`).
## Limits
| Limit | Value |
|---------------------------------------------------|------------------------|
| Items per add request | 20 |
| Invocation steps read when listing a conversation | 1,000 — a large item spans several, so a listing can stop early |
| Conversation lifetime | 30 days after creation |
A response bound to a conversation counts as one adding request, whatever its
number of output items. Start a new conversation to continue an exchange that
has reached the limit, and see
[Troubleshooting](operations_troubleshooting.md)
if a conversation stops accepting items earlier than expected.
## Errors
| Status | When |
|--------|----------------------------------------------------------------------|
| `400` | A malformed `conversation_id` or `item_id`, a metadata limit, an empty or oversized `items` list, or `conversation` combined with `previous_response_id`. |
| `404` | A well-formed identifier that names no conversation or item, including one created on another provider. |
## Prerequisites
Conversations are stored in Amazon Bedrock session storage in your own account.
The gateway's IAM role needs the
[Bedrock Session Storage permissions](operations_iam_permissions.md#bedrock-session-storage-optional);
without them, conversation requests fail with `503`. Set
[`AWS_BEDROCK_SESSION_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-bedrock-session-encryption-key-arn)
to encrypt conversation content with your own AWS KMS key.
## See Also
- [Responses API](api_openai_responses.md) — the `conversation` parameter and stored responses.
- [Configuration](operations_configuration.md#bedrock-session-storage-optional) — session storage settings.
- [IAM Permissions](operations_iam_permissions.md#bedrock-session-storage-optional) — the policy statement.
---
## Chat Completions
Source: https://stdapi.ai/md/api_openai_chat_completions.md
# Chat Completions API
Generate conversational AI responses with Amazon Bedrock foundation models—including Claude, Nova, Llama, and more—through an OpenAI-compatible interface.
## Why Choose the Chat Completions API?
- :material-brain: __Multiple Models__
Access models from Anthropic, Amazon, Meta, and more through one API. Choose the best model for your task without vendor lock-in.
- :material-image-multiple: __Multi-Modal__
Process text, images, videos, and documents together. Support for URLs, data URIs, and direct S3 references.
- :material-shield-check: __Built-In Safety__
Amazon Bedrock Guardrails provide content filtering and safety policies.
- :material-aws: __AWS Scale & Reliability__
Run on AWS infrastructure with service tiers for optimized latency. Multi-region model access for availability and performance.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|------------------------|--------|--------------------------------------------|--------------------------|---------------------------|
| `/v1/chat/completions` | `POST` | Conversational AI with multi-modal support | Amazon Bedrock Converse API · Amazon Bedrock Mantle | `openai_chat_completion` |
| `/v1/chat/completions` | `GET` | List stored chat completions | Amazon Bedrock Sessions | `openai_chat_completion_list` |
| `/v1/chat/completions/{completion_id}` | `GET` | Retrieve a stored chat completion | Amazon Bedrock Sessions | `openai_chat_completion_get` |
| `/v1/chat/completions/{completion_id}` | `POST` | Update a stored chat completion's metadata | Amazon Bedrock Sessions | `openai_chat_completion_update` |
| `/v1/chat/completions/{completion_id}` | `DELETE` | Delete a stored chat completion | Amazon Bedrock Sessions | `openai_chat_completion_delete` |
| `/v1/chat/completions/{completion_id}/messages` | `GET` | List the input messages of a stored chat completion | Amazon Bedrock Sessions | `openai_chat_completion_messages` |
## Feature Compatibility
Two outcomes are possible for a parameter no model behind this API can honor, and the Notes column below says which one applies. A parameter that only tunes an answer you can still use — a quality or latency hint, a telemetry opt-in — is **accepted and ignored**, so a client setting it on every request keeps working. A parameter that *is* the request — an output format, a modality, or a safety restriction that would silently disappear — is **rejected with a `400`** naming it, because the alternative is returning something you did not ask for.
| Feature | Status | Notes |
|------------------------------------------|:----------------------------------------:|-----------------------------------------------------------------|
| **Messages & Roles** | | |
| Text messages | :material-check-circle:{ .success role="img" aria-label="Supported" } | Full support for all text content |
| Image input (`image_url`) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | HTTP, data URIs |
| Image input from S3 | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | S3 URLs |
| Video input | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Supported by select models |
| Audio input | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Supported by select models |
| Document input (`file`) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | PDF and document support varies by model |
| Assistant `audio` reference | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | An assistant turn carrying only `audio: {"id": …}` is dropped (past audio is not replayable); resend the `transcript` as text to keep it in context |
| Files API (`file_id`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Reference uploaded files via `type: "file"` — see [Files API](api_openai_files.md) |
| System messages | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Includes `developer` role |
| **Tool Calling** | | |
| Function calling (`tools`) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Full OpenAI-compatible schema |
| Legacy `function_call` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Backward compatibility maintained |
| Parallel tool calls | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Multiple tools in one turn |
| Disable parallel tool calls | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | `parallel_tool_calls: false` is accepted for every model and honored by models able to constrain tool use; the response reports the tool calls actually made |
| Server tools | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Provider system tools and Claude server tools |
| `tool_choice` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | `auto`, `none`, `required`, and named-function choice are supported; `tool_choice: {"type": "allowed_tools"}` is rejected with `400` — supported on the [Responses API](api_openai_responses.md) |
| **Generation Control** | | |
| `max_tokens` / `max_completion_tokens` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Output length limits |
| `temperature` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Mapped to Bedrock inference params |
| `top_p` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Nucleus sampling control |
| `stop` sequences | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Custom stop strings. Whitespace-only sequences are rejected with `400` (Amazon Bedrock limitation) |
| `frequency_penalty` / `presence_penalty` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Repetition control |
| `seed` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Deterministic generation |
| `logit_bias` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Not all models support biasing |
| `top_logprobs` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Forwarded to the model as a provider-specific field; honored only by models that support it. Usable even though `logprobs` is rejected |
| `top_k` (From Qwen API) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Candidate token set size for sampling |
| `reasoning_effort` (OpenAI API-compatible) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Reasoning control: none/minimal/low/medium/high/xhigh/max (accepted for all models) |
| `enable_thinking` (Qwen API-compatible) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Enable/disable thinking mode (accepted for all reasoning models) |
| `thinking_budget` (Qwen API-compatible) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Thinking token budget (accepted for all reasoning models) |
| `thinking` (Moonshot API-compatible) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Thinking config: {"type": "enabled"/"disabled"} (accepted for all models) |
| `reasoning` (OpenRouter API-compatible) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Reasoning object: `effort`, `max_tokens`, `enabled`, `exclude`. Equivalent to `reasoning_effort`, `thinking_budget` and `enable_thinking`; conflicting values are rejected with `400` |
| `include_reasoning` (OpenRouter API-compatible) | :material-check-circle:{ .success role="img" aria-label="Supported" } | `false` omits the reasoning text from the response, like `reasoning: {"exclude": true}`; the reasoning tokens are still generated and billed |
| `n` (multiple choices) | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Generate multiple responses, not supported with streaming |
| `logprobs` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Rejected with `400` when enabled (`false`/`null` accepted, as they request the default behavior); `top_logprobs` (above) remains usable |
| `prediction` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Static predicted output content. Accepted and ignored — a latency hint the Amazon Bedrock Converse API has no equivalent for, and the answer is unchanged without it; forwarded upstream on [Mantle](#bedrock-mantle) passthrough models |
| `response_format: "json_object"` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Accepted for all models; syntactically valid JSON is not guaranteed for every model |
| `response_format: "json_schema"` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Structured JSON output validated against the supplied schema |
| `verbosity` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Model verbosity. Accepted and ignored — steer the answer length from the prompt; forwarded upstream on [Mantle](#bedrock-mantle) passthrough models |
| `web_search_options` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Web search tool. Rejected with `400` when set |
| `translation_options` (Qwen API-compatible) | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Translation tuning options. Rejected with `400` when set |
| `prompt_cache_key` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Cache prompts to reduce costs and latency |
| `prompt_cache_options` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | `mode: "explicit"` caches only the parts marked with `prompt_cache_breakpoint`; `ttl: "30m"` mapped to a 1 hour Amazon Bedrock retention on Anthropic models (other models use the default 5 minute TTL) when `prompt_cache_retention` is unset |
| `prompt_cache_breakpoint` (content part) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Explicit cache boundary mapped to an Amazon Bedrock `cachePoint` (max. 4 per request) |
| Extra model-specific params | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Extra model-specific parameters not supported by the OpenAI API |
| **Streaming & Output** | | |
| Text | :material-check-circle:{ .success role="img" aria-label="Supported" } | Text messages |
| Streaming (`stream: true`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Server-Sent Events (SSE) |
| Streaming obfuscation | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Unsupported |
| Audio | :material-check-circle:{ .success role="img" aria-label="Supported" } | Model output or synthesis from text output (synthesis is Converse-only — not performed for Mantle-served requests); non-streaming only — `stream: true` with audio output is rejected with `400` |
| `response_format` (JSON mode) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | `json_object` accepted for all models, without a syntax guarantee on every model; `json_schema` structured output is model-specific |
| `reasoning_content` (From Deepseek API) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Text reasoning messages. The single response field; on an assistant message replayed in `messages`, `reasoning` is accepted as an alias for it, and it is [dropped on models that only accept their own thinking](#replaying-reasoning-in-a-multi-turn-conversation) |
| `annotations` (URL citations) | :material-check-circle:{ .success role="img" aria-label="Supported" } | URL citations from system tools (non-streaming only) |
| **Usage tracking** | | |
| Input text tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Billing unit |
| Output tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Billing unit |
| Reasoning tokens | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Converse-served models do not split them out: `completion_tokens_details` is not populated and reasoning tokens are billed inside `completion_tokens`. Mantle-native models report whatever split their upstream API returns |
| **Other** | | |
| Service tiers | :material-check-circle:{ .success role="img" aria-label="Supported" } | Mapped to Bedrock service tiers and latency options |
| `metadata` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Echoed in the response, updatable on stored completions, and usable to filter the Bedrock invocation log. Also forwarded to Bedrock `requestMetadata`, whose limits apply: max 16 pairs, values ≤256 characters, restricted character set |
| `store` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Persists the completion in Amazon Bedrock session storage (non-streaming) |
| List / update stored completions | :material-check-circle:{ .success role="img" aria-label="Supported" } | List with `model`/`metadata` filters; metadata update |
| `safety_identifier` / `user` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Does not affect generation; identifies the end user in the request log and in [per-user cost attribution](operations_cost_management.md#per-user-attribution) |
| Bedrock Guardrails | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Content safety policies — not applied to Mantle-served requests |
| `moderation` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Applies an Amazon Bedrock guardrail; results in the response (non-streaming) — rejected (`400`) on Mantle-served models |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Model-Dependent** — Behavior depends on the model or backend; check the Notes column
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
## Model Support
All models supported by the Amazon Bedrock Converse and Converse Stream API are supported, plus every model served by [Bedrock Mantle](features.md#bedrock-mantle-models) when enabled — including OpenAI GPT-5.x, xAI Grok, and Google Gemma 4. Requests to Mantle models are passed through natively or converted automatically depending on the model's upstream API support — see [Bedrock Mantle](#bedrock-mantle) below.
### Bedrock Mantle
Mantle-served requests follow one of three paths, each with its own parameter fidelity:
| Serving path | Models | Parameter behavior |
|-----------------------------|---------------------------------------------------------------------------|--------------------|
| **Passthrough** | Chat-native models (xAI Grok, OpenAI gpt-oss, Google Gemma 4, other open-weight models) | All schema-accepted parameters are forwarded; the upstream API may reject unsupported ones per model with a clean `400` (the upstream error code and parameter are propagated) |
| **Converted to Responses** | OpenAI GPT frontier models; unknown models | Dropped silently: `stop`, `seed`, `frequency_penalty`, `presence_penalty`, `logit_bias`, `top_logprobs`, `audio`, `modalities`, `input_audio` content parts, legacy `functions`/`function_call`. Preserved: `metadata`, `safety_identifier`. `n > 1` rejected with `400`. `store` is handled by stdapi.ai only, never forwarded upstream |
| **Converted to Messages** | Mantle-only Anthropic Claude models | Same drops and `n > 1` rejection as the Responses conversion, except `stop` which is forwarded as `stop_sequences`; plus: `temperature` clamped to ≤ 1.0; `max_tokens` defaults to `4096` when unset; `reasoning_effort` mapped to Anthropic effort levels; `response_format` `json_object`/`json_schema` not available; `metadata`, `prompt_cache_key`, and `prompt_cache_retention` dropped; `service_tier` forwarded only when `auto` |
!!! note "Project attribution (`OpenAI-Project`)"
Mantle requests can be attributed to a Bedrock Project for cost tracking and observability with the `OpenAI-Project: ` header (a bare project ID such as `proj_abc123`, not an ARN). It is honored per-request only when [`AWS_BEDROCK_ALLOW_MANTLE_PROJECT_OVERRIDE`](operations_configuration.md#bedrock-allow-mantle-project-override) is `true`; otherwise the server default ([`AWS_BEDROCK_MANTLE_PROJECT`](operations_configuration.md#bedrock-mantle-project)) applies. This applies **only** to models served by the Bedrock Mantle endpoint — classic `bedrock-runtime` models ignore the header.
### Model Name Aliases
This API supports dynamic model name aliases matching official provider APIs. Models like OpenAI and Anthropic provide dynamic aliases in their official APIs—this gateway supports the same model names, automatically resolving them to Amazon Bedrock model identifiers.
**Examples (OpenAI GPT OSS models supported by Bedrock):**
- `gpt-oss-20b` → `openai.gpt-oss-20b-1:0`
## Advanced Features
### Prompt Caching
Reduce costs and improve response times by caching frequently-used prompt components across multiple requests. This feature is particularly effective for applications with consistent system prompts, tool definitions, or conversation contexts.
**Supported Models:**
- **Anthropic Claude**: Full support for system, messages, and tools caching
- **Amazon Nova**: Support for system and messages caching
!!! info "Documentation"
See [Amazon Bedrock Prompt Caching - Supported Models](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html#prompt-caching-models) for the complete list of models supporting prompt caching.
!!! warning "Cache Creation Costs"
Cache creation incurs a higher cost than regular token processing. Only use prompt caching when you expect a high cache hit ratio across multiple requests with similar prompts.
**How to Use:**
Set the `prompt_cache_key` parameter to enable caching:
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-fable-5",
"prompt_cache_key": "default",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant with extensive knowledge..."
},
{"role": "user", "content": "What is 2 + 2?"}
]
}'
```
**Granular Cache Control:**
Enable caching for specific prompt sections using dot-separated values:
- `"system"` - Cache system messages only
- `"messages"` - Cache conversation history
- `"tools"` - Cache tool/function definitions (Anthropic Claude only)
- `"system.messages"` - Cache both system and messages
- `"system.tools"` - Cache system and tools
- `"messages.tools"` - Cache messages and tools
- `"system.messages.tools"` - Cache all components
- Any other non-empty value - Cache all components
!!! note "Custom Cache Keys Not Supported"
Custom cache hash keys are not supported. The parameter is used only to control which sections are cached, not as a cache identifier.
```json
{
"model": "anthropic.claude-fable-5",
"prompt_cache_key": "system.tools",
"messages": [...],
"tools": [...]
}
```
**Benefits:**
- **Cost Reduction**: Cached tokens are billed at a lower rate than regular input tokens
- **Lower Latency**: Cached prompts eliminate reprocessing time
- **Automatic Management**: The API handles cache invalidation and updates
**Cache Retention (TTL):**
Control how long cached prompts persist using the `prompt_cache_retention` parameter:
!!! info "Model Support"
Cache retention configuration is only available on select models. See [Amazon Bedrock Prompt Caching - Supported Models](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html#prompt-caching-models) for details on which models support configurable TTL.
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-fable-5",
"prompt_cache_key": "default",
"prompt_cache_retention": "24h",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant..."
},
{"role": "user", "content": "What is 2 + 2?"}
]
}'
```
**Available Retention Values:**
- `"in_memory"` - Short-term caching (mapped to 5 minutes on Amazon Bedrock)
- `"24h"` - Long-term caching (mapped to 1 hour on Amazon Bedrock)
- Additional Amazon Bedrock values: `"1h"`, `"5m"` (provider-specific)
!!! note "OpenAI to Amazon Bedrock Mapping"
OpenAI retention values are mapped to Amazon Bedrock equivalents for compatibility:
- `"in_memory"` → 5 minutes
- `"24h"` → 1 hour
The OpenAI `prompt_cache_options` object is also accepted: its `ttl` (`"30m"`) is mapped to a 1 hour Amazon Bedrock retention on Anthropic models (other models use the default 5 minute TTL) when `prompt_cache_retention` is not set.
**Explicit Cache Breakpoints:**
Instead of relying on the `prompt_cache_key` section heuristics, mark the exact cache boundaries with `prompt_cache_breakpoint` on any content part (`text`, `image_url`, `input_audio`, `file`, `refusal`). Each marked part is followed by an Amazon Bedrock `cachePoint`, so the prompt prefix ending with that part is cached:
```json
{
"model": "anthropic.claude-fable-5",
"prompt_cache_options": {"mode": "explicit"},
"messages": [
{
"role": "system",
"content": [
{
"type": "text",
"text": "Long reusable instructions...",
"prompt_cache_breakpoint": {"mode": "explicit"}
}
]
},
{"role": "user", "content": "What is 2 + 2?"}
]
}
```
- `"mode": "explicit"` caches **only** the marked parts: the `prompt_cache_key` heuristics are disabled for that request.
- `"mode": "implicit"` (default) keeps the `prompt_cache_key` heuristics **and** honors the marked parts.
- At most 4 cache points are sent per request (Amazon Bedrock limit); the oldest ones are dropped when more are requested.
- Breakpoints on models without prompt caching support are accepted and ignored, as are breakpoints on tool result messages — those never become a cache point, whatever the model.
**Usage Tracking:**
Cached token usage is reported in the response:
```json
{
"usage": {
"prompt_tokens": 1500,
"completion_tokens": 100,
"total_tokens": 1600,
"prompt_tokens_details": {
"cached_tokens": 1200,
"cache_write_tokens": 300
}
}
}
```
In this example, 1,200 tokens were retrieved from cache and the remaining 300 tokens were processed and written to the cache. `cache_write_tokens` (an extra field beyond the OpenAI API) reports the tokens written to the cache when the model reports them, on both non-streaming responses and the trailing `stream_options.include_usage` chunk; it is omitted when no cache write occurred.
### System Prompt
System prompts define the AI assistant's behavior, personality, and instructions (e.g., "You are a helpful assistant"). Most models support system prompts.
!!! warning "Unsupported Models"
Some models don't support system prompts (`mistral.mistral-7b-instruct-v0:2`, `mistral.mixtral-8x7b-instruct-v0:1`). By default, **stdapi.ai silently drops system messages** for these models, allowing cross-model compatibility. To receive errors instead, configure [`DROP_UNSUPPORTED_SYSTEM_PROMPT=false`](operations_configuration.md#drop-unsupported-system-prompt).
### { style="height: 1.2em; vertical-align: text-bottom;" } S3 Image Support
Access images directly from your S3 buckets without generating pre-signed URLs or downloading files locally.
**Supported Formats:**
- **Images**: JPEG, PNG, GIF, WebP
**How to Use:**
Simply reference your S3 images using the `s3://` URI scheme in `image_url` fields:
```json
{
"model": "anthropic.claude-fable-5",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image"},
{
"type": "image_url",
"image_url": {"url": "s3://my-bucket/images/photo.jpg"}
}
]
}
]
}
```
!!! warning "IAM Permissions Required"
Your API service must have IAM permissions to read from the specified S3 buckets. S3 objects must be in the same AWS region as the executed model or accessible via your IAM role. Standard S3 data transfer and request costs apply.
**Benefits:**
- No pre-signed URLs - Direct S3 access without generating temporary URLs
- Security - Images stay in your AWS account with IAM-controlled access
- Performance - Optimized data transfer within AWS infrastructure
- Large images - No size limitations of data URIs or base64 encoding
### :material-file-link: Files API References (`file-id:`)
The string-overloaded `image_url.url`, `file.file_data`, and `input_audio.data` fields also accept the project-local `file-id:` URI scheme to reference files previously uploaded via the [Files API](api_openai_files.md):
```json
{
"model": "anthropic.claude-fable-5",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image"},
{"type": "image_url", "image_url": {"url": "file-id:file-0190c51c7de7455d9b8c2efe27dfbf67"}}
]
}]
}
```
!!! info "Two equivalent ways to reference an uploaded file"
The OpenAI-native typed path `{"type": "file", "file": {"file_id": "file-…"}}` is unchanged and still preferred when a typed object is acceptable. The `file-id:` URI is the equivalent for the *string-overloaded* `image_url.url` / `file.file_data` / `input_audio.data` fields, where today you would otherwise pass an `s3://`, `https://`, or `data:` URI. See [Referencing Uploaded Files via the `file-id:` URI Scheme](api_openai_files.md#referencing-uploaded-files-via-the-file-id-uri-scheme).
### Server Tools
Amazon Bedrock system tools are built-in capabilities that foundation models can use directly, without requiring you to implement backend integrations.
**How to Use:**
Add system tools to your `tools` array as normal. System tools don't require parameter definitions—just specify the tool name and the model will handle the rest.
#### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Nova Tools
| Tool | `function.name` | Amazon Nova 2 | Amazon Nova Premier (legacy) | API Support |
|------|-----------------|:-------------:|:-------------------:|:-----------:|
| Web Grounding | `nova_grounding` | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } |
| Code Interpreter | `nova_code_interpreter` | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } |
!!! danger "Code Interpreter Not Compatible"
**`nova_code_interpreter` cannot be used via this API.** The code execution result cannot be surfaced in the OpenAI Chat Completions response format.
##### Web Grounding
Amazon Nova Web Grounding enables models to search the web for current information, helping answer questions requiring real-time data like news, weather, product availability, or recent events. The model automatically determines when to use web grounding based on the user's query.
!!! info "Learn More"
- [Amazon Nova Web Grounding - User Guide](https://docs.aws.amazon.com/nova/latest/userguide/grounding.html)
- [Build More Accurate AI Applications with Amazon Nova Web Grounding - Blog Post](https://aws.amazon.com/blogs/aws/build-more-accurate-ai-applications-with-amazon-nova-web-grounding/)
**Usage:**
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-2-lite-v1:0",
"messages": [
{
"role": "user",
"content": "What are the current AWS Regions and their locations?"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "nova_grounding"
}
}
]
}'
```
**Response Format:**
When using web grounding, the API response includes `annotations` with URL citations in non-streaming mode:
```json
{
"choices": [{
"message": {
"role": "assistant",
"content": "The AWS Regions include...",
"annotations": [
{
"type": "url_citation",
"url_citation": {
"url": "https://aws.amazon.com/about-aws/global-infrastructure/",
"title": "AWS Global Infrastructure"
}
}
]
}
}]
}
```
!!! note "Streaming Mode"
URL citation `annotations` are only available in non-streaming responses.
**Limitations:**
- **No streaming citations**: URL citation `annotations` are not emitted in streaming responses.
!!! warning "Region Compatibility"
Web Grounding is only available in US Amazon Bedrock regions. To ensure all requests are routed to a US region, restrict the model using [`AWS_BEDROCK_MODEL_REGION_RESTRICT`](operations_configuration.md#bedrock-model-region-restrict):
```bash
export AWS_BEDROCK_MODEL_REGION_RESTRICT='{"amazon.nova-": ["us-east-1"]}'
```
#### { style="height: 1.2em; vertical-align: text-bottom;" } Anthropic Claude Server Tools
Anthropic Claude models support server-side tools (bash, text editor, memory) that are executed by the model provider. Declare them using the standard OpenAI function tool format: set `type` to `"function"` and `function.name` to the tool name.
**Supported Tools by Model:**
| Tool | `function.name` | Claude 3.5 Sonnet v2 | Claude 3.7+ |
|------|-----------------|:--------------------:|:-----------:|
| Bash | `bash` | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } |
| Text Editor | `str_replace_based_edit_tool` | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } |
| Computer | `computer` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } |
| Memory | `memory` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } |
!!! warning "Computer Use Not Supported"
The computer use workflow requires screenshots to be returned as images inside tool results. The OpenAI Chat Completions API does not support image content in `role: "tool"` messages, so the complete agent loop cannot be implemented. **`computer` is not usable via this route.**
**Usage:**
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-sonnet-5",
"messages": [
{"role": "user", "content": "Run a Python script that prints hello world."}
],
"tools": [
{"type": "function", "function": {"name": "bash"}},
{"type": "function", "function": {"name": "str_replace_based_edit_tool"}}
]
}'
```
**Tool Parameters:**
Some Claude server tools accept additional configuration. Pass tool-specific parameters inside `function.parameters`:
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-sonnet-5",
"messages": [
{"role": "user", "content": "Edit the file hello.py to print hello world."}
],
"tools": [
{
"type": "function",
"function": {
"name": "str_replace_based_edit_tool",
"parameters": {"type": "object", "max_characters": 5000}
}
}
]
}'
```
!!! tip "Beta Headers"
Claude server tools require specific `anthropic-beta` flags, which are **automatically injected** — no manual header needed:
- `bash`, `str_replace_based_edit_tool` → `computer-use-2024-10-22` (Claude 3.5) or `computer-use-2025-01-24` (Claude 3.7+)
- `memory` → `context-management-2025-06-27` (Claude 3.7+)
### Reasoning Control
This API supports several approaches to control [Amazon Bedrock reasoning](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-reasoning.html) behavior. Reasoning enables foundation models to break down complex tasks into smaller steps ("chain of thought"), improving accuracy for multi-step analysis, math problems, and complex reasoning tasks. All approaches work with all Amazon Bedrock models that support reasoning capabilities.
!!! info "Model Support for Configurable Reasoning"
Not all reasoning-capable models support configurable reasoning control. Support varies by model:
- **Anthropic Claude 3.7 - 4.5**: Both `reasoning_effort` and `thinking_budget` parameters supported (token budget-based reasoning)
- **Anthropic Claude Sonnet 4.6 / Opus 4.6 and later** (including Fable and Mythos): `reasoning_effort` parameter only (adaptive reasoning)
- **Amazon Nova 2 models**: `reasoning_effort` parameter only
- **DeepSeek V3 models**: `reasoning_effort` parameter only
Models listed as effort-only still accept a token budget: it turns reasoning
on, and the depth comes from their own effort scale.
#### { style="height: 1.2em; vertical-align: text-bottom;" } OpenAI and DeepSeek API-Compatible Reasoning Parameters
Use the `reasoning_effort` parameter with predefined effort levels. This format is shared by the OpenAI and [DeepSeek](https://api-docs.deepseek.com/api/create-chat-completion) Chat Completions APIs and works with all Amazon Bedrock models supporting reasoning.
**Available Levels:**
- `none` - Disable reasoning
- `minimal` - Quick responses with minimal reasoning
- `low` - Light reasoning for straightforward tasks
- `medium` - Balanced reasoning for most use cases
- `high` - Deep reasoning for complex problems
- `xhigh` - Maximum reasoning for complex problems
- `max` - Its own (higher) effort tier on the adaptive Claude models served by the Converse API (Sonnet/Opus 4.6 and later, plus Fable), which forward it unchanged; collapsed onto the model's top reasoning tier on the fixed-scale models (Claude 3.7 - 4.5, Amazon Nova 2, DeepSeek, Kimi). On Amazon Bedrock Mantle, Claude models — Mythos among them — are reached over the Anthropic Messages API, and that conversion maps `max` to `high`; every other Mantle-served model receives the level as sent. Claude 4.6 also maps `xhigh` down to `high`
**What You Get:**
- **`reasoning_content` field** (DeepSeek API-compatible): models include their thinking process in the response. The field name is an operator setting ([`CHAT_COMPLETIONS_REASONING_FIELD`](operations_configuration.md#chat-completions-reasoning-field)): `reasoning_content` by default, `reasoning` for clients written against OpenRouter or vLLM, or `none` to keep responses strictly OpenAI-shaped.
- **Streaming support**: `choices[].delta.reasoning_content` chunks in real time
**Example:**
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-fable-5",
"reasoning_effort": "high",
"messages": [{"role": "user", "content": "Solve this complex problem..."}]
}'
```
!!! note "Compatibility"
This format is accepted for all reasoning-capable models. Models that don't support this parameter will ignore it.
#### { style="height: 1.2em; vertical-align: text-bottom;" } Qwen API-Compatible Reasoning Parameters
Use explicit `enable_thinking` & `thinking_budget` parameters for fine-grained control over thinking mode. This is a Qwen API-compatible format that works with all Amazon Bedrock models supporting reasoning.
**Parameters:**
- `enable_thinking` (boolean): Enable or disable thinking mode
- Default: Model-specific (usually `false`)
- `thinking_budget` (integer): Maximum thinking process length in tokens
- Only effective when `enable_thinking` is `true`
- Passed to the model as `budget_tokens`
- Default: Model's maximum chain-of-thought length
**Example:**
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-sonnet-5",
"enable_thinking": true,
"thinking_budget": 2000,
"messages": [{"role": "user", "content": "Solve this complex problem..."}]
}'
```
!!! note "Compatibility"
This format is accepted for all reasoning-capable models. Models that don't support these parameters will ignore them.
#### { style="height: 1.2em; vertical-align: text-bottom;" } Moonshot API-Compatible Thinking Control
The `thinking` parameter provides a Moonshot API-compatible format for controlling thinking/reasoning on models that support it.
!!! info "Documentation"
See [Moonshot Kimi API](https://platform.kimi.ai/docs/api/chat) for more information.
**Parameters:**
- `thinking={"type": "enabled"}` — Enable thinking mode
- `thinking={"type": "disabled"}` — Disable thinking mode
This format is accepted for all reasoning-capable models. Models that don't support this parameter will ignore it.
#### :material-router-network: OpenRouter API-Compatible Reasoning Object
The `reasoning` object groups the same controls into a single field, and `include_reasoning` toggles whether the reasoning text is returned.
!!! info "Documentation"
See [OpenRouter Reasoning Tokens](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens) for more information.
**Parameters:**
- `reasoning.effort` (string): Effort level, exactly as `reasoning_effort`
- `reasoning.max_tokens` (integer): Reasoning token budget, exactly as `thinking_budget` (implies `enabled`)
- `reasoning.enabled` (boolean): Enable reasoning, exactly as `enable_thinking`
- `reasoning.exclude` (boolean): Omit the reasoning text from the response
- `include_reasoning` (boolean): `false` is equivalent to `reasoning: {"exclude": true}`
**Example:**
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-sonnet-5",
"reasoning": {"effort": "high", "exclude": true},
"messages": [{"role": "user", "content": "Solve this complex problem..."}]
}'
```
!!! note "Conflicting Values"
`reasoning.effort` and `reasoning.max_tokens` are mutually exclusive, and a sub-field disagreeing with its flat equivalent (for example `reasoning: {"effort": "high"}` with `reasoning_effort: "low"`) is rejected with `400`.
!!! note "Excluded Reasoning"
Excluding the reasoning text does not disable reasoning: the model still thinks, and the reasoning tokens are still counted in `usage` and billed. Use `reasoning_effort: "none"` or `reasoning: {"enabled": false}` to turn reasoning off.
#### :material-history: Replaying Reasoning in a Multi-Turn Conversation
Appending the assistant message you just received to `messages` — the standard
multi-turn idiom, and what the [DeepSeek API](https://api-docs.deepseek.com/api/create-chat-completion)
asks for after a tool call — is always accepted, `reasoning_content` included.
Anthropic Claude models only continue from a thinking passage they can recognise
as their own, which a replayed text field is not, so their reasoning is left out
of that turn instead. The message content, tool calls and refusal are sent
unchanged and the conversation continues normally; only the earlier chain of
thought is no longer visible to the model. Every other model family receives the
replayed reasoning as-is.
What this does **not** affect, measured on Claude Haiku 4.5:
- **Reasoning on the new turn.** The model still thinks, and still returns
`reasoning_content` — the reasoning setting for the turn being generated is
independent of the history. It re-derives rather than continuing the earlier
chain.
- **Tool-call continuations.** A turn that carried a tool call is answered
correctly with the earlier reasoning left out.
- **[Prompt caching](#prompt-caching).** Cache hits are unaffected, including
when the cache point sits inside the conversation immediately after the turn
whose reasoning is left out — measured across three turns, each one reading
the previous turn's cache in full and extending it. The omission is the same
on every turn, so the cached prefix stays identical and keeps growing.
!!! tip "Keeping the reasoning in context on Claude"
Use the [Responses API](api_openai_responses.md), which carries the thinking
passage in a form Claude accepts. Request
`include: ["reasoning.encrypted_content"]` and echo the reasoning items back,
and the model continues from its own earlier reasoning.
### Provider-Specific Parameters
Unlock advanced model capabilities by passing provider-specific parameters directly in your requests. These parameters are forwarded to Amazon Bedrock and allow you to access features unique to each foundation model provider.
!!! info "Documentation"
See [Bedrock Model Parameters](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html) for the complete list of available parameters per model.
**How It Works:**
Add provider-specific fields at the top level of your request body alongside standard OpenAI parameters. The API automatically forwards these to the appropriate model provider via Amazon Bedrock.
**Examples:**
**Top K Sampling:**
```json
{
"model": "anthropic.claude-fable-5",
"messages": [{"role": "user", "content": "Write a poem"}],
"top_k": 50,
"temperature": 0.7
}
```
**Configuration Options:**
**Option 1: Per-Request**
Add provider-specific parameters directly in your request body (as shown in examples above).
**Option 2: Server-Wide Defaults**
Configure default parameters for specific models via the `DEFAULT_MODEL_PARAMS` environment variable:
```bash
export DEFAULT_MODEL_PARAMS='{
"anthropic.claude-sonnet-5": {
"anthropic_beta": ["context-management-2025-06-27"]
}
}'
```
!!! tip "Parameter Priority"
Per-request parameters override server-wide defaults.
**Behavior:**
- :material-check-circle:{ .success role="img" aria-label="Supported" } **Compatible parameters**: Forwarded to the model and applied
- :material-alert-circle:{ .warning } **Unsupported parameters**: Return HTTP 400 with an error message
- :material-alert-circle:{ .warning } **Reserved names**: `model_id` and `additional_request_fields` collide with the gateway's own request-building parameters and are rejected with a `400 invalid_request_error` naming the key, instead of being forwarded
#### { style="height: 1.2em; vertical-align: text-bottom;" } Anthropic Claude Features
Enable cutting-edge Claude capabilities through Anthropic beta feature flags.
##### Beta Feature Flags
Enable experimental Claude features like interleaved thinking by adding the `anthropic_beta` array to your request (extended thinking itself is controlled through the [reasoning parameters](#reasoning-control), not a beta flag):
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-sonnet-5",
"messages": [{"role":"user","content":"Summarize the news headline."}],
"anthropic_beta": ["Interleaved-thinking-2025-05-14"]
}'
```
!!! tip "Server-Wide Configuration"
You can also configure beta flags server-wide using the `DEFAULT_MODEL_PARAMS` environment variable (see [Provider-Specific Parameters](#provider-specific-parameters)).
!!! warning "Unsupported Beta Flags"
Unsupported flags that would change output return HTTP 400 errors.
!!! info "Documentation"
See [Using Claude on Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages-request-response.html) for more details on Claude-specific parameters.
## Stored Chat Completions
Set `store: true` to persist a chat completion in [Amazon Bedrock session storage](https://docs.aws.amazon.com/bedrock/latest/userguide/sessions.html) — same mechanism, region, and [KMS setting](operations_configuration.md#aws-bedrock-session-encryption-key-arn) as [stored responses](api_openai_responses.md#stored-responses). The returned `id` then works with the full stored-completion surface:
- `GET /v1/chat/completions` — list stored completions, sorted by creation time (`order`, `after`, `limit`), filterable by `model` and by metadata pairs. Metadata filters accept either one `metadata[key]=value` parameter per key (what the OpenAI SDKs send) or a single `metadata={"key": "value"}` JSON object of string values, for clients that can only send a whole object in one query parameter. A bare `metadata` in any other shape is rejected with `400` naming both accepted forms.
- `GET /v1/chat/completions/{completion_id}` — retrieve the stored completion.
- `POST /v1/chat/completions/{completion_id}` — replace its `metadata` (`null` clears it).
- `GET /v1/chat/completions/{completion_id}/messages` — list its input messages.
- `DELETE /v1/chat/completions/{completion_id}` — delete it and its backing session.
`store` defaults to **false** on this implementation and is ignored with `stream=true` or when the server lacks the [session storage IAM permissions](operations_configuration.md#bedrock-session-storage-optional) (a warning is recorded in the request log). Listings scan a capped number of sessions (1,000) in the primary Bedrock region; accounts beyond the cap may see incomplete listings.
## Available Request Headers
This endpoint supports standard Bedrock headers for enhanced control over your requests. All headers are optional and can be combined as needed.
### Content Safety (Guardrails)
| Header | Purpose | Valid Values |
|--------------------------------------|------------------------------------|---------------------------------------|
| `X-Amzn-Bedrock-GuardrailIdentifier` | Guardrail ID for content filtering | Your guardrail identifier |
| `X-Amzn-Bedrock-GuardrailVersion` | Guardrail version | Version number (e.g., `1`) |
| `X-Amzn-Bedrock-Trace` | Guardrail trace level | `disabled`, `enabled`, `enabled_full` |
### Performance Optimization
| Header | Purpose | Valid Values |
|--------------------------------------------|------------------------|-------------------------------------------|
| `X-Amzn-Bedrock-Service-Tier` | Service tier selection | `default`, `flex`, `priority`, `reserved` |
| `X-Amzn-Bedrock-PerformanceConfig-Latency` | Latency optimization | `standard`, `optimized` |
**Example with all headers:**
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Amzn-Bedrock-GuardrailIdentifier: your-guardrail-id" \
-H "X-Amzn-Bedrock-GuardrailVersion: 1" \
-H "X-Amzn-Bedrock-Trace: enabled" \
-H "X-Amzn-Bedrock-Service-Tier: priority" \
-H "X-Amzn-Bedrock-PerformanceConfig-Latency: optimized" \
-d '{
"model": "anthropic.claude-fable-5",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
!!! info "Detailed Documentation"
For complete information about these headers, configuration options, and use cases, see:
- [Bedrock Guardrails Configuration](operations_configuration.md#bedrock-guardrails)
- [Service Tier and Performance Configuration](operations_configuration.md#bedrock-service-tier-and-performance-configuration)
## Model-Specific Features
### { style="height: 1.2em; vertical-align: text-bottom;" } TwelveLabs Pegasus
`twelvelabs.pegasus-1-2-v1:0` is a video-understanding model. Because Pegasus accepts exactly one video and one text prompt per call, this API adapts the conversation automatically:
- The **latest video** found anywhere in the conversation (any role, any position) is forwarded as the video input.
- The **latest contiguous run of user text** (back to the previous assistant or tool turn) is concatenated and forwarded as the text prompt.
- `temperature` and `max_tokens` are forwarded.
- `response_format: json_schema` is forwarded as Pegasus's structured output.
**Silently ignored** (no error): system prompts, tools, `top_p`, stop sequences, and prompt caching.
**Upstream format limitation:** The OpenAI Chat Completions API has no `video_url` content part type. To stay fully compatible with standard OpenAI clients, pass the video as an **`image_url`** content part — the server detects the video MIME type automatically and routes it to Pegasus correctly.
**Video input formats**: `data:video/mp4;base64,…`, `https://…`, `s3://bucket/key`, or `file-id:…`. Videos above 18.75 MB are automatically uploaded to S3.
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "twelvelabs.pegasus-1-2-v1:0",
"messages": [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {"url": "s3://my-bucket/video.mp4"}
},
{"type": "text", "text": "Describe what happens in this video."}
]
}
]
}'
```
## Try It Now
**Basic chat completion:**
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"messages": [{"role": "user", "content": "Say hello world"}]
}'
```
**Streaming response:**
```bash
curl -N -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"stream": true,
"messages": [{"role": "user", "content": "Write a haiku about the sea."}]
}'
```
**Multi-modal with image:**
```json
{
"model": "amazon.nova-micro-v1:0",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image"},
{"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
]
}
]
}
```
**With reasoning:**
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-sonnet-5",
"reasoning_effort": "low",
"messages": [{"role": "user", "content": "Solve 12*13"}]
}'
```
**Response with reasoning:**
```json
{
"choices": [{
"message": {
"role": "assistant",
"reasoning_content": "12 × 10 = 120, plus 12 × 3 = 36 → 156",
"content": "156"
}
}]
}
```
---
**Ready to build with AI?** Check out the [Models API](api_openai_models.md) to see all available foundation models!
---
## Completions
Source: https://stdapi.ai/md/api_openai_completions.md
# Completions API
Generate text completions with Amazon Bedrock foundation models—including Claude, Nova, Llama, and more—through an OpenAI-compatible interface using the simple completions format.
!!! info "Legacy upstream, first-class here"
OpenAI labels `/v1/completions` as **legacy** in their platform documentation and recommends new OpenAI projects migrate to `/v1/chat/completions` or `/v1/responses` for vendor compatibility. On stdapi.ai, this endpoint is a **first-class route** with the same quality guarantees as the others — its compact schema and small token footprint make it an excellent pick for MCP-based text agents and simple prompt-to-text workloads.
## Why Choose the Completions API?
- :material-feather: __Smallest Token Footprint__
The most compact request/response schema of the text APIs — ideal for MCP-based text agents and high-volume prompt-to-text workloads.
- :material-format-list-group: __Batch Prompts__
Send multiple independent prompts in one request and get one choice back per prompt, with streaming support.
- :material-file-link: __Multimodal Prompt Inputs__
Reference prompts and files via `https://`, `s3://`, `data:`, or `file-id:` URIs — including images, documents, audio, and video.
- :material-aws: __AWS Scale & Reliability__
Run on AWS infrastructure with service tiers and multi-region model access for availability and performance.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|-------------------|--------|----------------------------------|-------------------------------------------|---------------------|
| `/v1/completions` | `POST` | Simple prompt-to-text completion | Amazon Bedrock Converse API · Amazon Bedrock Mantle | `openai_completion` |
## Feature Compatibility
| Feature | Status | Notes |
|------------------------------------|:----------------------------------------:|------------------------------------------------------------------|
| **Prompt Input** | | |
| Single text prompt | :material-check-circle:{ .success role="img" aria-label="Supported" } | Full support for string prompts |
| Multiple prompts (batch) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Returns one choice per prompt; rejected with `400` on Mantle-served models |
| Text + files collapse (multimodal) | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | `[text, file, …]` sent as one multimodal request with one choice |
| Prompt from URL (`https://`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | HTTP URL reference |
| Prompt from S3 (`s3://`) | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | S3 URI reference |
| Prompt from data URI (`data:`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Base64-encoded data URI |
| Prompt from Files API (`file-id:`) | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Reference uploaded files |
| Token array prompts | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Not supported — use string prompts; rejected with `400` |
| **Generation Control** | | |
| `max_tokens` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Output length limits |
| `temperature` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Mapped to Bedrock inference params |
| `top_p` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Nucleus sampling control |
| `stop` sequences | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Custom stop strings; dropped when a Mantle request is converted to the Responses API |
| `n` (multiple choices) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Supported with and without streaming; `n > 1` rejected with `400` when a Mantle request is converted to the Responses or Messages API |
| `best_of` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored |
| `echo` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored; rejected with `400` on Mantle-served models |
| `frequency_penalty` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored |
| `presence_penalty` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored |
| `logit_bias` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored |
| `logprobs` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored; rejected with `400` on Mantle-served models |
| `seed` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored |
| `suffix` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored; rejected with `400` on Mantle-served models |
| **Streaming** | | |
| Streaming (`stream: true`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Server-Sent Events (SSE) |
| `stream_options.include_usage` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Usage in final chunk |
| Streaming with multiple prompts | :material-check-circle:{ .success role="img" aria-label="Supported" } | Deltas interleave; `choices[0].index` identifies the prompt |
| Streaming with n>1 | :material-check-circle:{ .success role="img" aria-label="Supported" } | Deltas interleave; `choices[0].index` identifies each choice |
| **Other** | | |
| Service tiers | :material-check-circle:{ .success role="img" aria-label="Supported" } | Mapped to Bedrock service tiers; `service_tier` and `prompt_cache_*` are not forwarded when a Mantle request is converted |
| `user` / `safety_identifier` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Forwarded to Amazon Bedrock as `requestMetadata`; on Mantle, `user` is forwarded as the OpenAI `user` field (as `metadata.user_id` when served via the Anthropic API) and `safety_identifier` is not forwarded |
| Extra model-specific params | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Unrecognized top-level keys are forwarded to the model as provider-specific fields; `model_id`, `additional_request_fields` and `stop_sequences` are reserved by the gateway and rejected with a `400 invalid_request_error` naming the key (use `stop` for stop sequences) |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Model-Dependent** — Behavior depends on the model or backend; check the Notes column
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
## Model Support
Every model the [Chat Completions API](api_openai_chat_completions.md#model-support) serves also answers here — the same model classes back both routes, with the prompt adapted into a single Converse turn. That means all models supported by the Amazon Bedrock Converse and Converse Stream API, plus every model served by [Bedrock Mantle](features.md#bedrock-mantle-models) when enabled. The Mantle conversion behavior tabled above is the same three serving paths described on the Chat Completions page.
To list the models this deployment serves on this route, call [`search_models`](api_search_models.md) with `route=openai_completion`.
### Model Name Aliases
The dynamic aliases published by the official provider APIs resolve here too, exactly as they do for chat — `gpt-oss-20b` reaches `openai.gpt-oss-20b-1:0`.
## Prompt Input Types
_stdapi.ai extends the standard completions interface with multiple input modes:_
### String Prompt (Standard)
The simplest input — a single text prompt:
```bash
curl -X POST "$BASE/v1/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"prompt": "The capital of France is",
"max_tokens": 20
}'
```
### Batch Prompts
Send multiple independent prompts in a single request — the server returns one `choices[]` entry per prompt:
```bash
curl -X POST "$BASE/v1/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"prompt": ["One plus one is", "Two plus two is", "Three plus three is"],
"max_tokens": 10
}'
```
### Text + Files: Single-Request Multimodal Collapse
When the prompt list contains **exactly one text string and one or more file references**, stdapi.ai packs them in input order into a **single multimodal request** and returns **one choice**. The text becomes a `text` block and each file becomes an `image` / `document` / `audio` / `video` Bedrock block (content type auto-detected) — the natural "ask once using these files as context" pattern.
- **Trigger**: list with exactly one `str` element and ≥1 URL elements (`https://`, `s3://`, `data:`, `file-id:`).
- **Effect**: elements are resolved concurrently, packed as Bedrock content blocks preserving input order, and sent as a single request. You get one `Completion` choice back.
- **Requires**: a model that supports the target modalities (e.g. Claude, Nova for image / document input).
- **Unchanged otherwise**: any other shape returns one `Completion` choice per list element. Each `str` becomes a `text` block; each `InputFileUrl` becomes the content block matching its detected MIME type (image, video, audio, document). See [Files-only prompts](#files-only-prompts) below.
Image analysis with a base64 data URI (Claude handles image input):
```bash
IMAGE_B64=$(base64 < chart.png | tr -d '\n')
curl -X POST "$BASE/v1/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d @- <"` → one request, one choice, with a single media block.
- `prompt: [, , …]` → one request per file (batch), one choice per file, each carrying its own media block.
The request reaches the model as-is; the model returns output or an error depending on whether it supports that modality (for example, Claude handles images and documents, Nova handles images, video, and documents). Use this shape for quick "what is this?" queries where the model's default behavior is enough; for tighter control over the response, prefer the collapse pattern above with an explicit instruction.
```bash
IMAGE_B64=$(base64 < screenshot.png | tr -d '\n')
curl -X POST "$BASE/v1/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d @- <
- :material-upload: __Simple Upload__
Upload any file with a single `multipart/form-data` request. Files are immediately available for use in inference.
- :material-upload-multiple: __Multipart Upload__
Stream large files in parts via the Uploads API. Parts are assembled by S3 native multipart upload.
- :material-clock-outline: __Optional Expiry__
Set `expires_after` to automatically expire files after a configurable number of seconds (1 hour to 30 days).
- :material-format-list-bulleted: __Paginated Listing__
List files with ascending or descending order and cursor-based pagination using the `after` parameter.
- :material-file-document-multiple: __Chat Integration__
Reference uploaded files directly in Chat Completions messages using `"type": "file"` content parts.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|------------------------------------|----------|----------------------------------------|------------|--------------------------|
| `/v1/files` | `POST` | Upload a file | Amazon S3 | `openai_file` |
| `/v1/files` | `GET` | List files with pagination | Amazon S3 | `openai_file_list` |
| `/v1/files/{file_id}` | `GET` | Retrieve file metadata | Amazon S3 | `openai_files_get` |
| `/v1/files/{file_id}` | `DELETE` | Delete a file | Amazon S3 | `openai_files_delete` |
| `/v1/files/{file_id}/content` | `GET` | Download raw file bytes | Amazon S3 | `openai_file_content` |
| `/v1/uploads` | `POST` | Create a multipart upload session | Amazon S3 Multipart Upload | `openai_upload` |
| `/v1/uploads/{upload_id}/parts` | `POST` | Add a part to an upload session | Amazon S3 Multipart Upload | `openai_upload_part` |
| `/v1/uploads/{upload_id}/complete` | `POST` | Complete the upload and produce a file | Amazon S3 Multipart Upload | `openai_upload_complete` |
| `/v1/uploads/{upload_id}/cancel` | `POST` | Cancel a pending upload session | Amazon S3 Multipart Upload | `openai_upload_cancel` |
## Feature Compatibility
| Feature | Status | Notes |
|----------------------------|:----------------------------------------:|-----------------------------------------------------------------------------------------------|
| **Upload** | | |
| `file` (multipart) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Required binary form field |
| `file` (JSON body) | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Base64, data URI, HTTPS URL, or S3 URI — for MCP / AI agents |
| `purpose` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Strictly validated against `assistants`, `batch`, `fine-tune`, `vision`, `user_data`, `evals` (others rejected); `batch` defaults to a 30-day expiry unless `expires_after[seconds]` is set, other purposes have no behavioral effect |
| `expires_after[anchor]` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Only `"created_at"` is accepted; expiry is computed from `expires_after[seconds]` |
| `expires_after[seconds]` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Range: 3 600 – 2 592 000 (1 hour – 30 days) |
| **Listing** | | |
| `order=asc` / `order=desc` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Ascending and descending supported; default `desc` |
| `after` cursor | :material-check-circle:{ .success role="img" aria-label="Supported" } | Forward cursor pagination |
| `limit` | :material-check-circle:{ .success role="img" aria-label="Supported" } | 1 – 10 000; default 10 000 |
| `purpose` filter | :material-check-circle:{ .success role="img" aria-label="Supported" } | Filter results by uploaded purpose |
| **File size cap** | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | No limit imposed by stdapi.ai; a direct upload streams in fixed 8 MiB parts, so S3's 10,000-part ceiling caps it at ~78 GiB |
| **Expiry enforcement** | :material-check-circle:{ .success role="img" aria-label="Supported" } | Expired files return 404 at read time and are omitted from listings; S3 Lifecycle as backstop |
| **Chat integration** | :material-check-circle:{ .success role="img" aria-label="Supported" } | Use `file_id` in `type: "file"` content parts |
| `status` field | :material-check-circle:{ .success role="img" aria-label="Supported" } | Always `"processed"` — no async processing pipeline |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations or differences
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
## Quick Start
### Upload a File
```bash
curl -X POST "$BASE/v1/files" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F "file=@document.pdf;type=application/pdf" \
-F "purpose=assistants"
```
**Response:**
```json
{
"id": "file-0190c51c7de7455d9b8c2efe27dfbf67",
"object": "file",
"bytes": 102400,
"created_at": 1745000000,
"filename": "document.pdf",
"purpose": "assistants",
"status": "processed"
}
```
### Upload via JSON Body (MCP and AI Agents)
When using MCP tools or HTTP clients that cannot construct `multipart/form-data` requests, pass the file as a base64 string, data URI, HTTPS URL, or S3 URI in a JSON body instead.
**Data URI (inline content):**
```bash
curl -X POST "$BASE/v1/files" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"file": "data:text/plain;base64,SGVsbG8gV29ybGQ=",
"purpose": "user_data"
}'
```
**HTTPS URL (server fetches the file):**
```bash
curl -X POST "$BASE/v1/files" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"file": "https://example.com/document.pdf",
"purpose": "assistants"
}'
```
**Raw base64:**
```bash
curl -X POST "$BASE/v1/files" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"file": "SGVsbG8gV29ybGQ=",
"purpose": "user_data"
}'
```
All three variants return the same `FileObject` response as a multipart upload.
### Upload with Expiry
```bash
curl -X POST "$BASE/v1/files" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F "file=@temp.txt;type=text/plain" \
-F "purpose=assistants" \
-F "expires_after[anchor]=created_at" \
-F "expires_after[seconds]=3600"
```
!!! info "Expiry Semantics"
Expiry is enforced lazily on every access: calls to retrieve metadata, download content, or reference the file in inference return HTTP 404 once the expiry time has passed, and listings skip the file rather than returning an entry that can no longer be retrieved. Each expired object encountered is scheduled for deletion; S3 Lifecycle rules clean up the rest as a background backstop.
### Retrieve Metadata
```bash
curl "$BASE/v1/files/file-0190c51c7de7455d9b8c2efe27dfbf67" \
-H "Authorization: Bearer $OPENAI_API_KEY"
```
### List Files
```bash
# Default (newest first, up to 10 000 files)
curl "$BASE/v1/files" \
-H "Authorization: Bearer $OPENAI_API_KEY"
# Ascending, limit 20
curl "$BASE/v1/files?order=asc&limit=20" \
-H "Authorization: Bearer $OPENAI_API_KEY"
# Next page using after cursor
curl "$BASE/v1/files?order=asc&limit=20&after=file-0190c51c7de7455d9b8c2efe27dfbf67" \
-H "Authorization: Bearer $OPENAI_API_KEY"
# Filter by purpose
curl "$BASE/v1/files?purpose=fine-tune" \
-H "Authorization: Bearer $OPENAI_API_KEY"
```
### Download Content
```bash
curl "$BASE/v1/files/file-0190c51c7de7455d9b8c2efe27dfbf67/content" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-o downloaded.pdf
```
### Delete a File
```bash
curl -X DELETE "$BASE/v1/files/file-0190c51c7de7455d9b8c2efe27dfbf67" \
-H "Authorization: Bearer $OPENAI_API_KEY"
```
**Response:**
```json
{
"id": "file-0190c51c7de7455d9b8c2efe27dfbf67",
"object": "file",
"deleted": true
}
```
## Uploads API
The Uploads API lets you stream large files to S3 in parts without buffering the entire file in memory. Each upload session is backed by an S3 native multipart upload.
### Upload ID Format
An upload ID and the file it produces share the same identifier — only the prefix changes (`upload_` → `file-`) when the upload is completed, so the final file ID is known upfront.
### Create an Upload Session
The declared `bytes` must be between 1 byte and 8 GiB; a larger declared size is rejected when the session is created.
Set the optional `expires_after` object to give the resulting file a TTL (same behavior as `expires_after` on `/v1/files`, 1 hour to 30 days). The pending upload's own `expires_at` always reflects the upload session's expiry (1 day); the requested file TTL appears on the resulting file object once the upload is completed.
```bash
curl -X POST "$BASE/v1/uploads" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"bytes": 6291456,
"filename": "large_dataset.bin",
"mime_type": "application/octet-stream",
"purpose": "assistants",
"expires_after": {"anchor": "created_at", "seconds": 3600}
}'
```
**Response:**
```json
{
"id": "upload_0190c51c7de7455d9b8c2efe27dfbf67",
"object": "upload",
"status": "pending",
"bytes": 6291456,
"filename": "large_dataset.bin",
"purpose": "assistants",
"created_at": 1745000000,
"expires_at": 1745086400
}
```
### Add Parts
Each part except the last must be at least 5 MiB (S3 minimum part size); the last part may be any size. An upload accepts at most 10,000 parts (S3's own ceiling).
**Binary upload (multipart/form-data):**
```bash
curl -X POST "$BASE/v1/uploads/upload_0190c51c7de7455d9b8c2efe27dfbf67/parts" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F "data=@part1.bin"
```
**JSON body (MCP and AI agents):**
When using MCP tools or HTTP clients that cannot construct `multipart/form-data`, pass the chunk as a base64 string, data URI, HTTPS URL, or S3 URI:
```bash
curl -X POST "$BASE/v1/uploads/upload_0190c51c7de7455d9b8c2efe27dfbf67/parts" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"data": "data:application/octet-stream;base64,AAEC..."
}'
```
**Response:**
```json
{
"id": "part_a3f5c81d2b6e49070001abcdef012345",
"object": "upload.part",
"upload_id": "upload_0190c51c7de7455d9b8c2efe27dfbf67",
"created_at": 1745000001
}
```
### Complete the Upload
```bash
curl -X POST "$BASE/v1/uploads/upload_0190c51c7de7455d9b8c2efe27dfbf67/complete" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"part_ids": [
"part_a3f5c81d2b6e49070001abcdef012345",
"part_a3f5c81d2b6e49070002fedcba987654"
],
"md5": "9e107d9d372bb6826bd81d3542a419d6"
}'
```
`part_ids` must be listed in ascending upload order (part 1, part 2, ...); S3 cannot reassemble multipart uploads out of order, so a reordered list is rejected with a 400 error rather than silently reordered.
The optional `md5` is the hex-encoded MD5 digest of the **whole file** — the parts concatenated in `part_ids` order, not a digest per part. When it is supplied the completed file is verified against it, and a mismatch is refused with a 400 error and leaves no file behind. Omit it and the upload completes unverified.
The assembled file reports the `created_at` of the upload session rather than of its completion, and takes the matching position in [the listing](#list-files) — so a large file uploaded part by part sorts where its own timestamp says it belongs.
**Response:** A completed `Upload` object with the `file` field populated.
```json
{
"id": "upload_0190c51c7de7455d9b8c2efe27dfbf67",
"object": "upload",
"status": "completed",
"bytes": 6291456,
"filename": "large_dataset.bin",
"purpose": "assistants",
"created_at": 1745000000,
"expires_at": 1745086400,
"file": {
"id": "file-0190c51c7de7455d9b8c2efe27dfbf67",
"object": "file",
"bytes": 6291456,
"created_at": 1745000000,
"expires_at": 1745003600,
"filename": "large_dataset.bin",
"purpose": "assistants",
"status": "processed"
}
}
```
### Cancel an Upload
```bash
curl -X POST "$BASE/v1/uploads/upload_0190c51c7de7455d9b8c2efe27dfbf67/cancel" \
-H "Authorization: Bearer $OPENAI_API_KEY"
```
!!! info "Upload sessions expire after 1 day"
If an upload is not completed within 1 day of creation it is automatically aborted. Parts uploaded to an expired session are discarded by S3.
### Uploads Feature Compatibility
| Feature | Status | Notes |
|--------------------------|:----------------------------------------:|--------------------------------------------------------------|
| `bytes` (declared size) | :material-check-circle:{ .success role="img" aria-label="Supported" } | 1 byte – 8 GiB; validated at completion against actual assembled size |
| `filename` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Carried through to the final file object |
| `mime_type` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Set as the S3 `ContentType` for the assembled object |
| `purpose` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Echoed to the final file object |
| Part data (binary) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Standard `multipart/form-data` binary upload via the `data` field |
| Part data (JSON body) | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Base64, data URI, HTTPS URL, or S3 URI — for MCP / AI agents |
| Part ordering | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | `part_ids` must be listed in ascending upload order; S3 cannot reassemble out of order |
| Part count / size limits | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Max 10,000 parts; every part except the last must be at least 5 MiB |
| `md5` checksum | :material-check-circle:{ .success role="img" aria-label="Supported" } | Hex MD5 of the whole file; a mismatch is refused with a 400 error |
| Session TTL | :material-check-circle:{ .success role="img" aria-label="Supported" } | 1 day from creation |
### End-to-End Example (Uploads)
```bash
# 1. Create an upload session
UPLOAD_ID=$(curl -s -X POST "$BASE/v1/uploads" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"bytes": 6291456,
"filename": "large_file.bin",
"mime_type": "application/octet-stream",
"purpose": "assistants"
}' | jq -r .id)
# 2. Upload parts (first part >= 5 MiB, last part any size)
PART_A_ID=$(curl -s -X POST "$BASE/v1/uploads/$UPLOAD_ID/parts" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F "data=@part1.bin" | jq -r .id)
PART_B_ID=$(curl -s -X POST "$BASE/v1/uploads/$UPLOAD_ID/parts" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F "data=@part2.bin" | jq -r .id)
# 3. Complete — the checksum covers the parts concatenated, in order
MD5=$(cat part1.bin part2.bin | md5sum | cut -d' ' -f1)
FILE_ID=$(curl -s -X POST "$BASE/v1/uploads/$UPLOAD_ID/complete" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d "{\"part_ids\": [\"$PART_A_ID\", \"$PART_B_ID\"], \"md5\": \"$MD5\"}" | jq -r .file.id)
echo "File ready: $FILE_ID"
# Cleanup
curl -X DELETE "$BASE/v1/files/$FILE_ID" \
-H "Authorization: Bearer $OPENAI_API_KEY"
```
## Chat Completions Integration
Reference an uploaded file inside a `POST /v1/chat/completions` message using `"type": "file"`:
```bash
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-haiku-4-5-20251001-v1:0",
"max_tokens": 512,
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Summarize this document."
},
{
"type": "file",
"file": {
"file_id": "file-0190c51c7de7455d9b8c2efe27dfbf67"
}
}
]
}
]
}'
```
!!! info "Model Support"
stdapi.ai picks the Amazon Bedrock content block from the file's media type — `document` for PDF, DOC/DOCX, XLS/XLSX, CSV, HTML, Markdown and plain text, `image` for images — and forwards it unchanged. No per-model gate is applied, so whether the file is accepted depends on the model's own input modalities: use a multimodal model (e.g. Claude or Amazon Nova) when passing PDFs or images via `file_id`, otherwise the model itself rejects the request.
## End-to-End Example
```bash
# 1. Upload the file
FILE_ID=$(curl -s -X POST "$BASE/v1/files" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F "file=@document.pdf;type=application/pdf" \
-F "purpose=assistants" | jq -r .id)
# 2. Reference in a chat completion
curl -X POST "$BASE/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"model\": \"anthropic.claude-haiku-4-5-20251001-v1:0\",
\"max_tokens\": 512,
\"messages\": [{
\"role\": \"user\",
\"content\": [
{\"type\": \"text\", \"text\": \"What is the key finding in this document?\"},
{\"type\": \"file\", \"file\": {\"file_id\": \"$FILE_ID\"}}
]
}]
}"
# 3. Cleanup
curl -X DELETE "$BASE/v1/files/$FILE_ID" \
-H "Authorization: Bearer $OPENAI_API_KEY"
```
## Referencing Uploaded Files via the `file-id:` URI Scheme
The native `file_id` JSON field shown above is the OpenAI-compatible way to reference an uploaded file in routes that have a typed `file_id` slot (chat completions content parts, responses content parts, image edits, …). For **string-overloaded** file fields that already accept URI schemes like `s3://`, `https://`, or `data:` — and therefore do **not** have a typed `file_id` slot — this implementation defines an additional project-local URI scheme:
```text
file-id:
```
!!! tip "Project-local URI scheme — `file-id:`"
`file-id:` is an **extension beyond the original OpenAI API**, parallel to the existing `s3://`, `https://`, and `data:` schemes already accepted on the same fields. It lets a client upload a file once via `/v1/files` and reuse it across embeddings, transcriptions, image edits, chat completions and other routes — without re-uploading or exposing an S3 URL.
* **Where accepted:** any string-overloaded file field (e.g. `image_url.url`, `input_audio.data`, `file.file_data` in chat completions; `input` on `/v1/embeddings`; `file` on audio transcription/translation; `image_url`/`mask` on image edits/variations; Anthropic image/document `source.url` and `source.data`).
* **Where rejected:** the ingest endpoints (`POST /v1/files`, `POST /v1/uploads/{id}/parts`) return **400** for `file-id:` inputs, because resolving them there would silently clone an existing file.
* **Where unchanged:** JSON fields that already accept a typed `file_id` (e.g. chat content parts `{"type":"file","file":{"file_id":"file-…"}}`) keep working exactly as in the OpenAI API; do not wrap those bare IDs in the `file-id:` prefix.
* **Detection:** match is **case-sensitive** (`file-id:`, lowercase) with no whitespace stripping; the payload after the prefix must be a valid Files API ID, otherwise the request fails with `400 invalid_request_error`.
* **Resolution:** the file is fetched from its underlying S3 object using the same code path as `s3://` URIs. A missing or expired file returns `404 not_found`. Content-type validation is delegated to each route, so an audio file passed to an image-only field is rejected the same way an `https://…/foo.mp3` URL would be.
### Worked Example — Embed an Uploaded File
```bash
# 1. Upload the file once.
FILE_ID=$(curl -s -X POST "$BASE/v1/files" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F "file=@research_paper.pdf;type=application/pdf" \
-F "purpose=assistants" | jq -r .id)
# 2. Reference it via file-id: in another route.
curl -X POST "$BASE/v1/embeddings" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"model\": \"amazon.nova-2-multimodal-embeddings-v1:0\",
\"input\": \"file-id:${FILE_ID}\"
}"
# 3. (Optional) delete the file when no longer needed.
curl -X DELETE "$BASE/v1/files/${FILE_ID}" \
-H "Authorization: Bearer $OPENAI_API_KEY"
```
## Errors
| HTTP | Cause |
|------|--------------------------------------------------------------------|
| 400 | Invalid `expires_after` range, bad filename, size mismatch, or unknown part ID |
| 400 | `part_ids` not listed in ascending upload order on `/v1/uploads/{upload_id}/complete` |
| 400 | A non-last part under 5 MiB, or an upload past its 10,000-part limit |
| 400 | `file-id:` URI passed to an ingest endpoint (`POST /v1/files`, `POST /v1/uploads/{upload_id}/parts`) |
| 400 | Upload session is no longer pending — it was already completed or cancelled |
| 404 | File or upload not found, already deleted, or expired |
| 503 | `AWS_S3_BUCKET` is not configured |
## Configuration
Files are stored in S3 under the prefix configured by [`AWS_S3_FILES_PREFIX`](operations_configuration.md#aws-s3-files-prefix) (default: `files/`). Configure S3 Lifecycle rules on this prefix to automatically delete expired objects and apply Intelligent-Tiering.
---
**Store files once, use them across requests.** See [Anthropic Files API](api_anthropic_files.md) for the Anthropic-compatible equivalent.
---
## Vector Stores
Source: https://stdapi.ai/md/api_openai_vector_stores.md
# Vector Stores API
Index your files once and search them by meaning. A vector store holds the
passages of the files attached to it; a search returns the passages closest to a
query, with the file they came from and a similarity score. There is no
embedding pipeline, chunker or vector database to run — attach a file and
search it.
## Why Choose the Vector Stores API?
- :material-magnify: __Search by Meaning__
A query finds the passages that answer it, not the ones sharing its words.
- :material-file-upload: __Attach and Forget__
Upload a file with the [Files API](api_openai_files.md), attach it, and it becomes searchable — indexing runs in the background.
- :material-filter-variant: __Attribute Filters__
Tag files with up to 16 attributes and restrict a search to the ones that match.
- :material-package-variant-closed: __File Batches__
Attach many files in one request and follow their progress with a single identifier.
- :material-timer-sand: __Expiration Policies__
Expire a store after a number of days without a search, so scratch stores do not accumulate.
- :material-shield-lock: __Your Own Account__
Documents, passages and vectors are stored in your AWS account, and never leave it.
- :material-database-arrow-right: __Bring Your Own Knowledge Base__
Address an [Amazon Bedrock knowledge base](#knowledge-base-stores) you already run as a vector store, through the same endpoints.
## Available Endpoints
| Endpoint | Method | What It Does | MCP Tool |
|----------------------------------------------------------------------|----------|-------------------------------------|----------------------------------------------|
| `/v1/vector_stores` | `POST` | Create a vector store | `openai_vector_store_create` |
| `/v1/vector_stores` | `GET` | List vector stores | `openai_vector_store_list` |
| `/v1/vector_stores/{vector_store_id}` | `GET` | Retrieve a vector store | `openai_vector_store_get` |
| `/v1/vector_stores/{vector_store_id}` | `POST` | Update name, metadata or expiration | `openai_vector_store_update` |
| `/v1/vector_stores/{vector_store_id}` | `DELETE` | Delete a vector store | `openai_vector_store_delete` |
| `/v1/vector_stores/{vector_store_id}/search` | `POST` | Search the indexed passages | `openai_vector_store_search` |
| `/v1/vector_stores/{vector_store_id}/files` | `POST` | Attach a file | `openai_vector_store_file_create` |
| `/v1/vector_stores/{vector_store_id}/files` | `GET` | List the attached files | `openai_vector_store_file_list` |
| `/v1/vector_stores/{vector_store_id}/files/{file_id}` | `GET` | Retrieve an attached file | `openai_vector_store_file_get` |
| `/v1/vector_stores/{vector_store_id}/files/{file_id}` | `POST` | Replace a file's attributes | `openai_vector_store_file_update` |
| `/v1/vector_stores/{vector_store_id}/files/{file_id}` | `DELETE` | Detach a file | `openai_vector_store_file_delete` |
| `/v1/vector_stores/{vector_store_id}/files/{file_id}/content` | `GET` | Read a file's indexed passages | `openai_vector_store_file_content` |
| `/v1/vector_stores/{vector_store_id}/file_batches` | `POST` | Attach several files at once | `openai_vector_store_file_batch_create` |
| `/v1/vector_stores/{vector_store_id}/file_batches/{batch_id}` | `GET` | Retrieve a file batch | `openai_vector_store_file_batch_get` |
| `/v1/vector_stores/{vector_store_id}/file_batches/{batch_id}/cancel` | `POST` | Cancel a file batch | `openai_vector_store_file_batch_cancel` |
| `/v1/vector_stores/{vector_store_id}/file_batches/{batch_id}/files` | `GET` | List a file batch's files | `openai_vector_store_file_batch_file_list` |
## Feature Compatibility
A store this server owns and a [knowledge base store](#knowledge-base-stores)
answer the same endpoints, but not always the same way. Where the two differ,
the row says which one it is talking about.
| Feature | Status | Notes |
|--------------------------------------------|:----------------------------------------:|-----------------------------------------------------------------------------|
| **Creating a store** | | |
| `name`, `description` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Up to 256 and 512 characters |
| `file_ids` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Up to 2000, indexed in the background; a rejected file leaves no store behind |
| `chunking_strategy` | :material-check-circle:{ .success role="img" aria-label="Supported" } | `auto`, or `static` with the bounds in [Chunking](#chunking); becomes the store's own strategy |
| `expires_after` | :material-check-circle:{ .success role="img" aria-label="Supported" } | `anchor: "last_active_at"` and a day count — see [Expiration](#expiration) |
| `metadata` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Up to 16 pairs, 64-character keys, 512-character values |
| [Knowledge base stores](#knowledge-base-stores) | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | An Amazon Bedrock knowledge base you already run, served through these same endpoints as `vs_kb_...` |
| Creating one of those | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | A knowledge base is addressed, never created: no request field names one |
| **Managing a store** | | |
| Retrieve | :material-check-circle:{ .success role="img" aria-label="Supported" } | On a knowledge base store, name, description, creation time and status are read from it |
| Update `name`, `metadata`, `expires_after` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | `expires_after: null` clears the policy. `400` on a knowledge base store — the three are read from it |
| Delete | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | `400` on a knowledge base store, which is managed outside this server |
| `limit`, `order`, `after`, `before` | :material-check-circle:{ .success role="img" aria-label="Supported" } | On every listing: 1 to 100, default 20, newest first — see [Listing Order](#listing-order) |
| `filter` on a file listing | :material-check-circle:{ .success role="img" aria-label="Supported" } | Returns only the files with the named status |
| **Attaching a file** | | |
| `file_id` | :material-check-circle:{ .success role="img" aria-label="Supported" } | A file uploaded with the [Files API](api_openai_files.md) |
| `attributes` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Up to 16 pairs within a 2048-byte budget. On a knowledge base store they are indexed and searchable but never reported back on the file, and the key `stdapi-filename` is reserved — `400` |
| `chunking_strategy` on an attach | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | `400` on a knowledge base store, which chooses its own passage boundaries |
| Replace a file's `attributes` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Replaces the whole set. `400` on a knowledge base store — attach the file again with the attributes it should carry |
| Retrieve, detach | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Detaching a document of the corpus behind a knowledge base store is `400`; it is removed where that corpus comes from |
| Read a file's indexed passages | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | `400` on a knowledge base store — download the file itself instead |
| **File batches** | | |
| `file_ids` or `files` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Exactly one of the two, up to 2000 entries; `files` carries per-file `attributes` and `chunking_strategy`. `400` on a knowledge base store |
| Batch-level `attributes`, `chunking_strategy` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Applied to every `file_ids` entry |
| Retrieve, cancel, list a batch's files | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | A cancel stops the files that have not started; the ones already indexed stay. `400` on a knowledge base store |
| **Searching** | | |
| `query` | :material-check-circle:{ .success role="img" aria-label="Supported" } | A string, or up to 16 strings searched together, on either kind of store |
| `max_num_results` | :material-check-circle:{ .success role="img" aria-label="Supported" } | 1 to 50, default 10 |
| `filters` | :material-check-circle:{ .success role="img" aria-label="Supported" } | All eight comparison operators and both combinators, nestable, on either kind of store — see [Filters](#filters) |
| `ranking_options.score_threshold` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | `400` on a knowledge base store, whose relevance value is not comparable between searches; use `max_num_results` there |
| `ranking_options.ranker` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Accepted and ignored — results are always ranked by the store's own relevance, and any value is taken |
| `rewrite_query` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Accepted and ignored — the query is searched as written |
| `has_more` / `next_page` on a results page | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Always `false` and `null`: a search returns one complete page |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
## Model Support
No request field names a model: a store embeds with the model it was built with,
and a search embeds its query with that same one. Which model that is depends on
the kind of store.
| Store | Embeds with |
|------------------------------------------------|-------------------------------------------------------------------------------|
| One this server owns | [`VECTOR_STORE_EMBEDDING_MODEL`](operations_configuration.md#vector-store-embedding-model), recorded on the store when it is created |
| A [knowledge base store](#knowledge-base-stores) | The embedding model of the knowledge base itself — the setting above is not used, and the knowledge base embeds the query on its side |
Any model this deployment serves that produces embeddings is eligible. To
shortlist them, call [`search_models`](api_search_models.md) with
`route=openai_embedding`. A model that produces something else is refused with
`400`, naming it, when a store is created — never silently, and never at search
time.
!!! note "No reranking stage"
A search is a single pass: the passages closest to the query, in that
order. Nothing re-scores them afterwards, so
`ranking_options.ranker` names no model and selects nothing — see the
[Rerank API](api_cohere_rerank.md) to re-order results yourself. A Bedrock
managed knowledge base may rerank inside its own retrieval, which is its
behaviour rather than this server's.
## Listing Order
Every listing — stores, a store's files, a batch's files — is ordered by the
`created_at` it reports for each object, **most recent first**; `order=asc`
reverses it. Objects sharing a second are ordered by identifier, so the sequence
is stable from one page to the next.
| Object | `created_at` is |
|--------|-----------------------------------------------------------------------------------|
| Store | When the store was created. |
| File | When the file was **attached to that store** — not when it was uploaded. Attaching the same file again moves it to the newest end. |
The `after` and `before` cursors are positions in that order: `after` returns the
objects that follow the named one, `before` the page that ends just before it,
both in the direction `order` asks for. Neither restarts the listing from the
top when the object it names has since been deleted.
## Quick Start
```python
import time
from openai import OpenAI
client = OpenAI(base_url="https://your-gateway/v1", api_key="YOUR_API_KEY")
uploaded = client.files.create(file=open("handbook.txt", "rb"), purpose="assistants")
store = client.vector_stores.create(name="handbook", file_ids=[uploaded.id])
# Indexing is asynchronous: wait until the store reports it finished.
deadline = time.monotonic() + 300
while client.vector_stores.retrieve(store.id).status == "in_progress":
assert time.monotonic() < deadline, "indexing did not finish"
time.sleep(2)
for result in client.vector_stores.search(
store.id, query="How much parental leave do I get?"
):
print(result.score, result.filename, result.content[0].text)
```
## Indexing Is Asynchronous
Attaching a file returns immediately with `status="in_progress"`. Poll the file,
or the store, until it settles:
| Object | Field | Settled when |
|--------|--------------------------|--------------------------------------------------------------------|
| Store | `status` | `completed` — no attached file is still being indexed. |
| Store | `file_counts` | `in_progress` reaches `0`; the other counters sum to `total`. |
| File | `status` | `completed`, `failed` or `cancelled`. |
| Batch | `status` / `file_counts` | Same, for the files of that batch only. |
The counters summarise the files, so they can trail a file that has just
settled — a store still reporting `in_progress` for a file already `completed`
converges within a moment. Poll the file itself when you need the earliest
possible answer.
Indexing is bounded server-wide: attaching many files at once never indexes
more than a couple at a time, so a large attach queues rather than being
refused. A poll always terminates: a file whose indexing was interrupted and
cannot be resumed settles as `failed` with `last_error.code="server_error"`,
and attaching it again indexes it.
### Durable Indexing { #durable-indexing }
Whether an interruption costs you anything depends on one deployment setting,
[`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url):
| Setting | What happens when the server indexing a file is replaced |
|---|---|
| Unset (default) | The file settles as `failed`. Attach it again. |
| Set | Another server picks the work up and finishes it. The file stays `in_progress` a little longer, then settles as `completed`. |
Nothing about the API changes: the same fields, the same statuses, the same
polling. A file only takes longer to settle. Indexing stays at-least-once and
never bills twice — work that already completed is not redone.
Ask your administrator which of the two your deployment runs before designing a
client around it.
## Supported Files
Files must be **text**: plain text, Markdown, source code, CSV, JSON, XML,
YAML and anything else whose bytes decode as UTF-8 and whose content type is
not a known binary one — a text file uploaded as `application/octet-stream` or
`application/pdf` is refused on its content type, before its bytes are read.
A file that is not text settles as `status="failed"` with
`last_error.code="unsupported_file"`. The message names what **that** store
indexes, and — when another kind of store would take the file as it stands —
where to send it instead:
> This file type cannot be indexed by this vector store. It indexes text only.
> Provide the content as a text file. A knowledge base store indexes this file
> type as it stands.
Convert documents to text otherwise — [RAG
Pipelines](use_cases_rag.md#document-parsing) shows a document-conversion stage
that produces Markdown from PDF and office formats.
| `last_error.code` | Meaning |
|-------------------|----------------------------------------------------------------|
| `unsupported_file`| The file is not one this store indexes. |
| `invalid_file` | The file is text but holds nothing to index, or is too large. |
| `server_error` | Indexing failed, or was interrupted; attach the file again. |
A [knowledge base store](#knowledge-base-stores) indexes more than text — PDF and
office documents as they stand, and media on a Bedrock managed one. Its own
refusals list the formats that store accepts.
## Chunking
A file is split into overlapping passages before it is indexed. Send a
`chunking_strategy` to choose the split:
```json
{
"type": "static",
"static": {"max_chunk_size_tokens": 800, "chunk_overlap_tokens": 400}
}
```
| Field | Range | Default |
|-------------------------|----------------|---------|
| `max_chunk_size_tokens` | 100 to 4096 | 800 |
| `chunk_overlap_tokens` | 0 to half the chunk size | 400 |
`"type": "auto"` inherits the store's own strategy — the one given when it was
created — which also applies to every file attached without a strategy at all.
A store created without one falls back to the server default, which comes from
[`VECTOR_STORE_CHUNK_SIZE_TOKENS`](operations_configuration.md#vector-store-chunk-size-tokens).
!!! note "Chunk sizes are approximate"
The chunk size is applied as a text-length budget, and a cut is moved back
to the nearest line or word boundary, so a passage never ends mid-word. The
resulting passages therefore do not match another provider's split
character for character. A passage is additionally capped by what the
configured embedding model accepts in one input, so a very large
`max_chunk_size_tokens` may produce shorter passages than asked for.
`GET /v1/vector_stores/{id}/files/{file_id}/content` returns the passages a file
was indexed as, in document order — the fastest way to see what a chunking
strategy actually produced.
## Searching
```python
page = client.vector_stores.search(
store.id,
query="parental leave",
max_num_results=5,
filters={"key": "department", "type": "eq", "value": "hr"},
ranking_options={"score_threshold": 0.4},
)
```
| Parameter | Default | Notes |
|----------------------------------|---------|-----------------------------------------------------------------|
| `query` | — | A string, or an array of strings searched together. |
| `max_num_results` | `10` | 1 to 50. |
| `filters` | — | A comparison or compound filter over the files' `attributes`. |
| `ranking_options.score_threshold`| — | Drops results scoring below it. |
| `ranking_options.ranker` | — | Accepted and ignored: results are always ranked by similarity. |
| `rewrite_query` | `false` | Accepted and ignored: the query is searched as written. |
Each result carries `file_id`, `filename`, the file's `attributes`, the matching
`content`, and a `score` between `0` and `1` where `1` is an exact match. Results
are ordered best first, and the page is complete — search is never paginated.
!!! note "Scores on a knowledge base store"
On a [`vs_kb_...` store](#knowledge-base-stores) the `score` is the relevance
value the knowledge base measured, reported unchanged. It orders the results
within one response and is **not** the `0`-to-`1` similarity above, so never
compare it across stores or against a fixed threshold —
`ranking_options.score_threshold` is refused on those stores for that reason.
### Filters
| `type` | Matches |
|-----------------------------------|--------------------------------------------|
| `eq`, `ne` | Equal / not equal — string, number or boolean |
| `gt`, `gte`, `lt`, `lte` | Numeric comparison — `value` must be a number |
| `in`, `nin` | Value in / not in an array |
| `and`, `or` | Combine filters, nestable |
Filters apply to the `attributes` of the attached file, never to its content. A
filter that matches nothing returns an empty page rather than an error, while an
ordering operator given a non-numeric `value` is rejected with `400`: store a
date as a number (for example `20260115`) to compare ranges of it.
## Attributes
Up to 16 key-value pairs per attached file, with string, number or boolean
values. They are returned on the file and on every search result, and are what
`filters` matches against.
| Limit | Value |
|------------------------------------|--------------|
| Key-value pairs | 16 |
| Key length | 64 |
| String value length | 512 |
| Total size of all attributes | 2048 bytes |
Attributes larger than the total budget are rejected with `400`, naming the
limit. `POST /v1/vector_stores/{id}/files/{file_id}` **replaces** the whole set;
the new values apply to later searches once the file is `completed`. Replacing
the attributes of a file that is still `in_progress` is accepted, but the
indexing in flight writes the attributes it started with — wait for the file to
settle, then replace them.
## Expiration
```json
{"expires_after": {"anchor": "last_active_at", "days": 7}}
```
`expires_at` is `last_active_at` plus `days`. **Only a search refreshes
`last_active_at`** — attaching, reading or updating a store does not — so a
store that is written to but never searched still expires. Once past its
expiration a store reads back with `status="expired"` and returns no search
result; its indexed content is released and a search never brings it back. Send
`"expires_after": null` on an update to remove the policy.
## Knowledge Base Stores { #knowledge-base-stores }
A vector store can also be served by an
[Amazon Bedrock knowledge base](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html)
you already created. Allowlist it in
[`AWS_BEDROCK_KNOWLEDGE_BASE_IDS`](operations_configuration.md#aws-bedrock-knowledge-base-ids)
and it is addressed as the vector store `vs_kb_` — the knowledge
base identifier is ten alphanumeric characters — on every `/v1/vector_stores`
endpoint, and returned by `GET /v1/vector_stores` next to the stores the server
owns.
Amazon Bedrock builds a knowledge base over documents in two ways, and **both
are served**:
| Knowledge base | API type |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|
| [Bedrock managed](https://docs.aws.amazon.com/bedrock/latest/userguide/kb-build-managed.html) — Amazon Bedrock owns the datastore, the embedding model, the parser and the reranker | `MANAGED` |
| [Customer-managed](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build.html) — you provision the vector store and choose the embedding model | `VECTOR` |
They differ in [two things](#two-kinds) a client can see; everything else on this
page holds for either.
!!! warning "A structured-data knowledge base is not one of them"
A knowledge base [connected to a structured data store](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build-structured.html)
(`SQL`) answers a query with database rows rather than passages, and one
backed by an Amazon Kendra GenAI index (`KENDRA`) is a search index rather
than a vector store. Neither is served here — allowlist only the two above.
**The knowledge base stays yours.** It is addressed, never created and never
deleted; the server searches it and manages the documents of its data source.
Its name, description, creation time and status are read from the knowledge base
itself.
!!! warning "Attaching a file needs a custom data source"
A file attached through this API becomes an in-line document, which only a
**custom** data source takes — on either kind. Point the allowlist
entry at one, as `/`; pointed at a data
source that syncs its corpus from a bucket or another service, the store
answers `400` to an attach and keeps serving search, listing and reading.
A knowledge base can hold both kinds.
```python
store = client.vector_stores.retrieve("vs_kb_ABCDE12345")
uploaded = client.files.create(file=open("handbook.pdf", "rb"), purpose="assistants")
client.vector_stores.files.create(
vector_store_id=store.id, file_id=uploaded.id, attributes={"department": "hr"}
)
for result in client.vector_stores.search(
store.id, query="How much parental leave do I get?", max_num_results=5
):
print(result.score, result.filename, result.content[0].text)
```
### What Works
| Request | On a `vs_kb_...` store |
|------------------------------------------------------|--------------------------------------------------------------------------------------------------------------|
| `GET /v1/vector_stores` | Lists it alongside the stores the server owns. |
| `GET /v1/vector_stores/{id}` | Name, description, creation time and status, as the knowledge base reports them. |
| `POST /v1/vector_stores/{id}/files` | Attaches an uploaded file: it becomes a document of the knowledge base's custom data source, with its `attributes` kept searchable. |
| `GET /v1/vector_stores/{id}/files` | Lists the documents of the store's data source, including the ones put there outside this API. |
| `GET /v1/vector_stores/{id}/files/{file_id}` | Retrieves one document, including one a search returned from elsewhere in the knowledge base. |
| `DELETE /v1/vector_stores/{id}/files/{file_id}` | Removes a document this API attached. |
| `POST /v1/vector_stores/{id}/search` | Searches it, with `filters` and `max_num_results`. |
`filters` works in full: all eight comparison operators (`eq`, `ne`, `gt`,
`gte`, `lt`, `lte`, `in`, `nin`) and both combinators (`and`, `or`), over any
metadata key, with no schema to declare beforehand.
### What Is Refused
| Request | Answer |
|------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| `POST /v1/vector_stores` creating one | Not creatable: the store is addressed, never created. No request field names a knowledge base. |
| `DELETE /v1/vector_stores/{id}` | `400` — the store is managed outside the server. |
| `POST /v1/vector_stores/{id}` (`name`, `metadata`, `expires_after`) | `400` — they are read from the knowledge base. |
| `chunking_strategy` on an attach | `400` — the store chooses its own passage boundaries. |
| `POST /v1/vector_stores/{id}/files/{file_id}` (attribute rewrite) | `400` — attach the file again with the attributes it should carry. |
| `GET /v1/vector_stores/{id}/files/{file_id}/content` | `400` — the passages a file was indexed as cannot be listed. Download the file itself with the [Files API](api_openai_files.md). |
| `POST /v1/vector_stores/{id}/files` on a store whose data source syncs its corpus | `400` — that corpus is maintained where it comes from. Use a store this server owns, or ask for one that accepts uploads. |
| `DELETE /v1/vector_stores/{id}/files/{file_id}` on a document of the corpus | `400` — it was not attached here, and is removed where the corpus comes from. |
| `POST /v1/vector_stores/{id}/file_batches` and the other batch routes | `400` — attach and follow files one at a time. |
| `ranking_options.score_threshold` | `400` — the store's relevance scores are not comparable between searches; use `max_num_results` instead. |
| A `query` over the length limit | `400` naming the limit. The query is never truncated. |
`usage_bytes`, `file_counts`, and a file's `chunking_strategy` and `attributes`
are reported as unknown — zero, or absent — rather than invented: the corpus is
yours, and the server does not claim to know what it holds.
A document's `created_at` is the only time the knowledge base reports for it: the
instant it last ingested that document, which for a document ingested once is
when it was created. The document listing is ordered on that same value, so a
page never reports a time that contradicts its own order.
### Identifiers
A file you attach keeps its own `file-...` identifier. A document that was
already in the knowledge base — one of the corpus behind it — is reported under
an opaque `kbdoc_...` identifier, and the per-file routes accept it.
A search covers the whole knowledge base, not only the documents attached here,
so a result may name a document of a corpus this API never wrote to. Its
`kbdoc_...` identifier still reads back: `GET /v1/vector_stores/{id}/files/{file_id}`
answers for it, wherever in the knowledge base it lives. Removing it is the one
thing refused — that corpus is maintained where it comes from.
### Document Formats
Files are indexed as they stand, with no conversion step:
| Formats | Indexed by |
|---------------------------------------------------------------------------|--------------------------------------------------|
| `.txt`, `.md`, `.html`, `.csv`, `.doc`, `.docx`, `.xls`, `.xlsx`, `.pdf` | Every knowledge base |
| `.ppt`, `.pptx`, images, audio and video | A Bedrock managed knowledge base, additionally |
| Anything else whose bytes decode as text | Every knowledge base, indexed as text |
One file is at most **5 MiB**.
A format outside that table is refused when the file is attached — `400`, naming
the formats **this** store takes — rather than accepted and then reported as
`failed`. A file that is attached and then fails is one of a format the store
does take, so it settles with `last_error.code="server_error"`.
### Where the Two Kinds Differ { #two-kinds }
Only two things differ from the client's point of view:
| Difference | Customer-managed | Bedrock managed |
|--------------------------|------------------|--------------------------------------------|
| Document formats indexed | The table above | Also `.ppt`, `.pptx`, images, audio, video |
| `query` length | 1,000 characters | 10,000 characters |
### Cost
A knowledge base search costs more than a search on a store the server owns, and
a knowledge base backed by an always-on vector database bills whether it is
queried or not. This is information for choosing between the two, not a
recommendation.
What the [usage log](operations_cost_management.md#vector-stores) reports differs
per kind, because the retrieval runs inside the knowledge base rather than
through an embedding model of the server's:
| Knowledge base | Search | Attaching a file |
|------------------|----------------------------------------------------------------|------------------------------------------------------|
| Bedrock managed | One `search_units` unit per query, at the published flat rate | Not reported: index storage is billed monthly per GB |
| Customer-managed | Not reported: no per-retrieval rate is published for it | Not reported: billed by its own embedding model |
Everything left unreported is on your AWS bill and readable from AWS Cost
Explorer. Full detail in [Cost Management](operations_cost_management.md#vector-stores).
## Limits
| Limit | Value |
|-----------------------------------|--------------------------------------------|
| Files per file batch | 2000 |
| Queries per search | 16 |
| Results per search | 50 |
| Store `metadata` | 16 pairs, 64-character keys, 512-character values |
| File size | 100 MiB, or [`MAX_INPUT_FILE_SIZE`](operations_configuration.md#max-input-file-size) when it is lower |
A file above the size limit is not rejected at request time: it settles as
`status="failed"` with `last_error.code="invalid_file"`, like any other file
that cannot be indexed.
## Errors
| Status | When |
|--------|-----------------------------------------------------------------------------------|
| `400` | Attributes above the total budget, a chunking strategy outside its bounds, a `gt`/`gte`/`lt`/`lte` filter given a non-numeric value, or a request a [knowledge base store](#knowledge-base-stores) does not accept. |
| `404` | An identifier that names no store, attached file or batch. A `vs_kb_...` identifier that is not allowlisted answers exactly like any unknown store. |
| `409` | The store is being updated concurrently by another request; retry it. |
| `503` | The deployment has no vector storage configured. |
## Prerequisites
Vector stores are stored in your own AWS account, in an
[Amazon S3 vector bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors.html)
you create yourself. Set
[`AWS_S3_VECTORS_BUCKET`](operations_configuration.md#aws-s3-vectors-bucket) to
its name and [`AWS_S3_VECTORS_REGION`](operations_configuration.md#aws-s3-vectors-region)
to its Region; the endpoints answer `503` until both a vector bucket and
[`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) are configured. The
gateway's IAM role needs the
[Vector Stores permissions](operations_iam_permissions.md#vector-stores-optional).
The model that turns text into vectors is
[`VECTOR_STORE_EMBEDDING_MODEL`](operations_configuration.md#vector-store-embedding-model).
It is recorded on each store when the store is created, so changing the setting
only affects stores created afterwards — existing stores keep answering with the
model they were built with.
Indexing survives a server being replaced only when
[`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url)
names an Amazon SQS queue you created, with the
[durable indexing permissions](operations_iam_permissions.md#durable-vector-store-indexing)
on it — see [Durable indexing](#durable-indexing).
[Knowledge base stores](#knowledge-base-stores) need none of the above: they need
[`AWS_BEDROCK_KNOWLEDGE_BASE_IDS`](operations_configuration.md#aws-bedrock-knowledge-base-ids),
the [knowledge base permissions](operations_iam_permissions.md#knowledge-base-vector-stores),
and a knowledge base in the first
[`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) entry.
They bring their own storage and their own embedding model, and they are listed
and served even when no vector bucket is configured.
## See Also
- [Files API](api_openai_files.md) — uploading the files you attach.
- [RAG Pipelines](use_cases_rag.md) — using a vector store as the retrieval stage.
- [Embeddings API](api_openai_embeddings.md) — embedding text yourself instead.
- [Configuration](operations_configuration.md#vector-stores-optional) — the settings above.
- [Cost Management](operations_cost_management.md) — what indexing and searching cost.
---
## Batches
Source: https://stdapi.ai/md/api_openai_batches.md
# Batch API
Run a large set of chat completion or embedding requests asynchronously, at a lower price than the synchronous API, through the OpenAI Batch API shape.
A batch is created from a file of requests, runs without a connection held open, and is read back from the result files it produces — exactly the OpenAI workflow, so the official OpenAI SDKs work by changing the base URL.
## Why Choose the Batch API?
- :material-tag-arrow-down: __Lower Price per Token__
Batched requests are billed at the published batch rate, well below the on-demand rate for the same model.
- :material-swap-horizontal: __Drop-in OpenAI Compatibility__
`client.batches.create(...)` and `client.batches.retrieve(...)` work unchanged, as does the JSONL input format.
- :material-server-off: __No Connection to Hold__
Submit and walk away. Results stay readable through the [Files API](api_openai_files.md) once the batch ends.
- :material-cloud-lock: __Private AWS Backend__
Requests and results are stored in your own S3 buckets — no traffic to third-party endpoints.
## Available Endpoints
| Endpoint | Method | What It Does | MCP Tool |
|-----------------------------|--------|---------------------------------------|-----------------------|
| `/v1/batches` | `POST` | Create a batch from an uploaded file | `openai_batch` |
| `/v1/batches` | `GET` | List batches, newest first | `openai_batch_list` |
| `/v1/batches/{batch_id}` | `GET` | Retrieve a batch and its counters | `openai_batch_get` |
| `/v1/batches/{batch_id}/cancel` | `POST` | Cancel a batch that is still running | `openai_batch_cancel` |
## Feature Compatibility
| Feature | Status | Notes |
|----------------------------------|:----------------------------------------:|-----------------------------------------------------------------------------|
| **Creation** | | |
| `input_file_id` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Must be uploaded with `purpose="batch"` |
| `endpoint` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | `/v1/chat/completions` and `/v1/embeddings` |
| `completion_window` | :material-check-circle:{ .success role="img" aria-label="Supported" } | `24h`, as upstream |
| `metadata` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Up to 16 key-value pairs, returned on every read |
| `output_expires_after` | :material-check-circle:{ .success role="img" aria-label="Supported" } | `anchor: "created_at"` and 1 hour to 30 days, counted from the moment the result files are written; omit it to keep them until deleted |
| **Per-request body** | | |
| `messages`, `max_tokens`, sampling | :material-check-circle:{ .success role="img" aria-label="Supported" } | Same parameters as [Chat Completions](api_openai_chat_completions.md) |
| `response_format` `json_object` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Full support |
| `tools` / `tool_choice` / `functions` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Refused when the batch is created — tool use is not available in a batch |
| `response_format` `json_schema` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Refused when the batch is created |
| `stream` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | A batch has nothing to stream to |
| `n` above 1 | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Send one request per completion instead |
| `prompt_cache_key` / `prompt_cache_breakpoint` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Accepted and ignored — a batch reads and writes no prompt cache, and the request is answered without one |
| `input`, `dimensions` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Same parameters as [Embeddings](api_openai_embeddings.md), one `input` per request |
| `encoding_format` `base64` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Refused when the batch is created — batched vectors come back as numbers |
| **Lifecycle** | | |
| Retrieve / poll | :material-check-circle:{ .success role="img" aria-label="Supported" } | `validating` → `in_progress` → `finalizing` → `completed` |
| Cancel | :material-check-circle:{ .success role="img" aria-label="Supported" } | `cancelling` then `cancelled`; requests already answered stay in `output_file_id`, and a batch that has ended is unchanged |
| List batches | :material-check-circle:{ .success role="img" aria-label="Supported" } | Newest first, with an `after` cursor |
| `output_file_id` / `error_file_id` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Readable through the [Files API](api_openai_files.md) |
| `usage` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Token totals, reported once the batch ends |
| `finalizing` status | :material-check-circle:{ .success role="img" aria-label="Supported" } | Reported with `finalizing_at` while the results of a batch whose requests have run are being assembled; `completed` follows once they are readable |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
!!! note "Content Guardrails and Batches"
A request that a [guardrail](operations_configuration.md#aws-bedrock-guardrail-identifier) would apply to is refused rather than run unguarded. Send those requests without batching.
!!! note "Prompt Caching and Batches"
Batched requests neither read nor write a prompt cache, on any model. A request carrying a cache hint is still accepted and answered — the hint is dropped rather than the request — so a batch reports no cached tokens in `usage.input_tokens_details`. Nothing is lost by leaving the hint in: batched requests are already billed at the batch rate, and the cache discount was never available at that rate.
## Model Support
Any chat or embedding model available for batch inference in your configured Amazon Bedrock regions can be used — the same identifiers as [Chat Completions](api_openai_chat_completions.md) and [Embeddings](api_openai_embeddings.md). To shortlist them, call [`search_models`](api_search_models.md) with `route=openai_chat_completion&batch=true`, or `route=openai_embedding&batch=true` for embeddings; each entry also carries a `batch` field.
!!! warning "The shortlist is a hint, not a rule"
`batch` is advertised on a best-effort basis and never used to reject a request. A model it does not advertise — or says nothing about — may still run a batch, so submit the batch rather than ruling the model out; the answer you get back is the authoritative one.
A model that cannot serve batched requests is refused when the batch is created, naming the model. A model this deployment normally serves through another Amazon Bedrock endpoint is batched under the identifier the batch endpoint knows it by, so it needs nothing from you.
## Workflow
### 1. Upload the requests
One JSON object per line, uploaded with `purpose="batch"`. Every line names the same model, targets the same endpoint, carries a unique `custom_id`, and holds the request itself in `body`.
```json
{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "amazon.nova-micro-v1:0", "messages": [{"role": "user", "content": "Summarize: ..."}]}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "amazon.nova-micro-v1:0", "messages": [{"role": "user", "content": "Summarize: ..."}]}}
```
Embedding requests are written the same way, against `/v1/embeddings`:
```json
{"custom_id": "doc-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "amazon.titan-embed-text-v2:0", "input": "First passage of the corpus"}}
{"custom_id": "doc-2", "method": "POST", "url": "/v1/embeddings", "body": {"model": "amazon.titan-embed-text-v2:0", "input": "Second passage of the corpus"}}
```
Both samples above are abridged: a real file needs at least 100 requests, the minimum a batch carries.
```python
from openai import OpenAI
client = OpenAI(base_url="https://your-host/v1", api_key="...")
requests_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
```
### 2. Create the batch
```python
batch = client.batches.create(
input_file_id=requests_file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
```
**Example request (curl):**
```bash
curl -X POST "https://your-host/v1/batches" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input_file_id": "file-06fvfg3lbdqarbad8kbo55g0sg5h3s4a",
"endpoint": "/v1/chat/completions",
"completion_window": "24h"
}'
```
**Example response:**
```json
{
"id": "batch_06fvfg3lbdqarbad8kbo55g0sg5h3s4a",
"object": "batch",
"endpoint": "/v1/chat/completions",
"input_file_id": "file-06fvfg3lbdqarbad8kbo55g0sg5h3s4a",
"completion_window": "24h",
"status": "validating",
"created_at": 1786568013,
"expires_at": 1786654413,
"request_counts": {"total": 100, "completed": 0, "failed": 0},
"model": "amazon.nova-micro-v1:0"
}
```
### 3. Poll until it ends
```python
batch = client.batches.retrieve(batch.id)
print(batch.status, batch.request_counts)
```
### 4. Read the results
```python
results = client.files.content(batch.output_file_id).text
```
Each line pairs a `custom_id` with the completion it produced:
```json
{"id": "batch_req_9f2c...", "custom_id": "req-1", "response": {"status_code": 200, "request_id": "batch_req_9f2c...", "body": {"id": "chatcmpl-req-1", "object": "chat.completion", "choices": [{"index": 0, "message": {"role": "assistant", "content": "..."}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 22, "completion_tokens": 9, "total_tokens": 31}}}, "error": null}
```
Requests that failed are collected in a separate file, named by `error_file_id`.
!!! tip "Expiring the result files"
Result files are kept until deleted, and are billed as stored objects
meanwhile. Pass `output_expires_after` when creating the batch to have both
files expire on their own — between 1 hour and 30 days, counted from the
moment they are written:
```python
batch = client.batches.create(
input_file_id=requests_file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
output_expires_after={"anchor": "created_at", "seconds": 7 * 24 * 3600},
)
```
!!! warning "Results Are Not in Request Order"
Output lines may come back in any order, as upstream also warns. Match a result to its request with `custom_id`, never with the line number.
## Limits
| Limit | Value |
|--------------------------------|------------------------------|
| Minimum requests per batch | 100 (default quota) |
| Maximum requests per batch | 50,000 |
| Maximum input file size | 200 MB |
| `custom_id` length | 64 characters |
| Distinct models per input file | 1 (upstream rule) |
| Processing window | 24 hours from creation |
A batch below the minimum, or past any of these caps, is refused when it is created and the message names the shortfall, rather than accepted and failed later.
!!! note "The 100-request minimum is a quota default"
100 is the default of the Amazon Bedrock quota *Minimum number of records per batch inference job*, which is set **per model** and adjustable for some of them — see [Amazon Bedrock quotas](https://docs.aws.amazon.com/general/latest/gr/bedrock.html). The gateway checks against that default, not against your account's own value, so a raised quota is enforced by Amazon Bedrock rather than here — a batch of 150 clears this check and is then refused by the backend — and a lowered one is not usable: fewer than 100 requests is still refused here.
## Prerequisites
The Batch API is disabled until the deployment declares an AWS IAM service role that Amazon Bedrock assumes to read the requests and write the results:
- [`AWS_BEDROCK_BATCH_ROLE_ARN`](operations_configuration.md#aws-bedrock-batch-role-arn) — the service role.
- [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) — the bucket holding the batch data.
- [`AWS_S3_BATCHES_PREFIX`](operations_configuration.md#aws-s3-batches-prefix) — the prefix it is stored under.
The permissions the role and the server need are listed in [IAM Permissions](operations_iam_permissions.md#batch-inference).
While the role is unset, every batch endpoint answers `503`.
## Billing
Batched requests are billed at the published batch rate for the model, roughly half the on-demand rate. Usage is recorded once, when the batch ends. See [Cost Management](operations_cost_management.md#batch-inference).
## See Also
- [Files API](api_openai_files.md) — upload the requests, download the results
- [Chat Completions API](api_openai_chat_completions.md) — the per-request body
- [Embeddings API](api_openai_embeddings.md) — the per-request body of an embeddings batch
- [Message Batches API](api_anthropic_batches.md) — the Anthropic-shaped equivalent
- [Configuration](operations_configuration.md#aws-bedrock-batch-role-arn) — enabling batches
---
## Images Generations
Source: https://stdapi.ai/md/api_openai_images_generations.md
# Images API - Image Generation
Generate images with Amazon Bedrock image models like Stability AI and Amazon Nova Canvas through an OpenAI-compatible interface.
## Why Choose the Image Generation API?
- :material-palette: __Quality Output__
Generate photorealistic images, digital art, and illustrations.
- :material-fast-forward: __Real-Time Streaming__
Progressive generation shows partial previews as the model works for interactive applications.
- :material-ruler: __Flexible Control__
Choose dimensions, quality levels, and styles. From quick drafts to high-resolution finals.
- :material-aws: __Scalable Infrastructure__
Generate images at scale with Amazon Bedrock infrastructure. No GPU management required.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|--------------------------|--------|-----------------------------------|-----------------------------|---------------------------|
| `/v1/images/generations` | `POST` | Generate images from text prompts | Amazon Bedrock Image Models | `openai_image_generation` |
## Feature Compatibility
| Feature | Status | Notes |
|--------------------------------|:----------------------------------------:|---------------------------------------------------------------------|
| **Generation** | | |
| Text-to-image (`/generations`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Generate images from prompts |
| **Parameters** | | |
| `prompt` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Text description for generation (required, min 1 char) |
| `model` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Required parameter |
| `n` (number of images) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Multiple images per request; accepted range is 1-10 (default: 1), but the effective maximum is model-dependent (e.g. Amazon Titan and Nova Canvas cap at 5) |
| `size` (WIDTHxHEIGHT) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Output dimensions (default: 1024x1024, format validated; `auto` resolves to the default) |
| `response_format` | :material-check-circle:{ .success role="img" aria-label="Supported" } | `url` or `b64_json` (default: `url`) |
| `quality` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Quality setting (default: `auto`, supports OpenAI & model-specific); accepted and ignored by models with no quality control |
| `style` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Model-specific style parameters; accepted and ignored by models with no style control |
| `output_format` | :material-check-circle:{ .success role="img" aria-label="Supported" } | `png`, `jpeg`, or `webp` (model-specific) |
| `output_compression` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Compression level 1-100% (default: 100) |
| `stream` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Generate images in streaming mode, sending each finished image as an `image_generation.completed` event |
| `partial_images` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted (0-3) but ignored — no available model currently streams partial images; the final image is always sent as a single event |
| `background` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Accepts `auto` (default) and `opaque`; `transparent` is unsupported — responses report `opaque` |
| `moderation` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Only the default `auto` is accepted; other values are rejected with an error |
| Extra model-specific params | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Extra model-specific parameters via JSON body |
| **Output** | | |
| URL response format | :material-check-circle:{ .success role="img" aria-label="Supported" } | Temporary download URLs, valid for 60 minutes (requires AWS_S3_BUCKET) |
| Base64 JSON format | :material-check-circle:{ .success role="img" aria-label="Supported" } | Inline base64-encoded images |
| PNG format | :material-check-circle:{ .success role="img" aria-label="Supported" } | Lossless image output |
| JPEG format | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Lossy compression (model-specific) |
| WebP format | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Modern format with compression (model-specific) |
| Streaming response | :material-check-circle:{ .success role="img" aria-label="Supported" } | Server-sent events with final images (no partial previews) |
| **Usage tracking** | | |
| Input text tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Sourced from AWS billing when available |
| Output image tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Sourced from AWS billing data when available; falls back to the image count (`n`) |
| **Other** | | |
| `user` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Logged but not used for abuse monitoring |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Available on Select Models** — Check your model's capabilities
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
## Model Support
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Models
| Model | Supported Task Types | Notes |
|-----------------------------------|-----------------------------------------|---------------------------------------------------------------------------------------------|
| amazon.nova-canvas-v1:0 (legacy) | `TEXT_IMAGE`, `COLOR_GUIDED_GENERATION` | Supports standard text-to-image generation and color-guided generation with 8 style presets |
| amazon.titan-image-generator-v1 (legacy) | `TEXT_IMAGE` | Basic text-to-image generation |
| amazon.titan-image-generator-v2:0 (legacy) | `TEXT_IMAGE`, `COLOR_GUIDED_GENERATION` | Enhanced text-to-image generation with color-guided generation support |
!!! note "Legacy Amazon Image Models"
AWS has scheduled `amazon.nova-canvas-v1:0` and the Titan image models to reach end of life on September 30, 2026. Deployments with existing access can keep using them until then (legacy models are hidden unless [`AWS_BEDROCK_LEGACY=true`](operations_configuration.md#bedrock-legacy)); the Stability AI Stable Image family is the long-term successor.
### { style="height: 1.2em; vertical-align: text-bottom;" } Stability AI Models
| Model | Supported Task Types | Notes |
|-----------------------------------|----------------------|---------------------------------------------------|
| stability.sd3-5-large-v1:0 | `TEXT_IMAGE` | Stable Diffusion 3.5 Large - high quality output |
| stability.stable-image-core-v1:1 | `TEXT_IMAGE` | Stable Image Core - balanced quality and speed |
| stability.stable-image-ultra-v1:1 | `TEXT_IMAGE` | Stable Image Ultra - premium quality and detail |
!!! info "No Built-In Aliases for OpenAI Image Model Names"
OpenAI's default image model names (`dall-e-2`, `dall-e-3`, `gpt-image-1`) have **no built-in alias**, so requests using them fail with a model-not-found error — the most common first-call issue. Pass one of the model IDs above, or map the OpenAI names to your preferred models with [`MODEL_ALIASES`](operations_configuration.md#model-aliases).
!!! warning "Configuration Required"
You must configure the `AWS_S3_BUCKET` environment variable with a bucket to use the URL response format.
!!! tip "Performance Optimization"
For faster image downloads, especially for high-resolution images or globally distributed users, enable S3 Transfer Acceleration by setting `AWS_S3_ACCELERATE=true`. This uses CloudFront edge locations to accelerate file downloads, providing 50-500% faster speeds for users far from your S3 bucket region. See [S3 Transfer Acceleration configuration](operations_configuration.md#aws-s3-accelerate) for setup details.
## Advanced Features
### Provider-Specific Parameters
Unlock advanced image generation capabilities by passing provider-specific parameters directly in your requests. These parameters are forwarded to Amazon Bedrock and allow you to access features unique to each image model provider.
**Documentation:** [Bedrock Image Model Parameters](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html)
**How It Works:**
Add provider-specific fields at the top level of your request body alongside standard OpenAI parameters. The API automatically forwards these to the appropriate model provider via Amazon Bedrock.
**Configuration Options:**
**Option 1: Per-Request**
Add provider-specific parameters directly in your request body (as shown in the examples below).
**Option 2: Server-Wide Defaults**
Configure default parameters for specific models via the `DEFAULT_MODEL_PARAMS` environment variable:
```bash
export DEFAULT_MODEL_PARAMS='{
"stability.stable-image-core-v1:1": {
"negative_prompt": "blurry, low quality, watermark"
}
}'
```
**Note:** Per-request parameters override server-wide defaults.
**Behavior:**
**Compatible parameters** are forwarded to the model and applied; **unsupported parameters** return HTTP 400 with an error message.
**Examples:**
**Stability AI - Negative Prompts:**
```json
{
"model": "stability.stable-image-core-v1:1",
"prompt": "A serene mountain landscape at sunset",
"negative_prompt": "blurry, distorted, low quality, watermark"
}
```
**Amazon Nova Canvas - Negative Prompts:**
```json
{
"model": "amazon.nova-canvas-v1:0",
"prompt": "An abstract watercolor painting",
"textToImageParams": {"negativeText": "blurry, distorted, low quality, watermark"}
}
```
#### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Nova Canvas
**Basic Usage (Standard OpenAI Parameters):**
```bash
curl -X POST "$BASE/v1/images/generations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-canvas-v1:0",
"prompt": "A futuristic cityscape at night"
}'
```
**Parameter Mapping:**
| OpenAI Parameter | Maps to | Notes |
|------------------|----------------------------------------|-------------------------------------|
| `prompt` | Depends on `taskType` | See taskType-specific mapping below |
| `size` | `imageGenerationConfig.width/height` | Flexible (320-4096) |
| `quality` | `imageGenerationConfig.quality` | "high" → "premium" |
| `style` | `textToImageParams.style` | 8 preset styles |
| `n` | `imageGenerationConfig.numberOfImages` | 1-5 images |
**TaskType-Specific Parameter Mapping:**
| taskType | `prompt` maps to |
|---------------------------|------------------------------------|
| `TEXT_IMAGE` (default) | `textToImageParams.text` |
| `COLOR_GUIDED_GENERATION` | `colorGuidedGenerationParams.text` |
**Advanced Generation Modes:**
Default `taskType` is `"TEXT_IMAGE"`.
Available task types:
- `"TEXT_IMAGE"` - Standard text-to-image generation
- `"COLOR_GUIDED_GENERATION"` - Generate images based on color palette
```bash
# Color-Guided Generation
curl -X POST "$BASE/v1/images/generations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-canvas-v1:0",
"prompt": "A sunset landscape",
"taskType": "COLOR_GUIDED_GENERATION",
"colorGuidedGenerationParams": {
"colors": ["#FF6B6B", "#FFD93D", "#6BCB77"]
}
}'
```
!!! info "Full Parameter Reference"
For all parameters, styles, and task types, see [Amazon Nova Canvas documentation](https://docs.aws.amazon.com/nova/latest/userguide/image-generation.html)
#### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Titan Image Generator
**Basic Usage (Standard OpenAI Parameters):**
```bash
curl -X POST "$BASE/v1/images/generations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.titan-image-generator-v2:0",
"prompt": "A beautiful landscape with mountains"
}'
```
**Parameter Mapping:**
| OpenAI Parameter | Maps to | Notes |
|------------------|----------------------------------------|-----------------------------------------------------------------------------|
| `prompt` | Depends on `taskType` | See taskType-specific mapping below |
| `size` | `imageGenerationConfig.width/height` | Discrete sizes: 512, 768, 1024, 1152, 1216, 1344, 1536, 2048 |
| `quality` | `imageGenerationConfig.quality` | "high" → "premium" |
| `n` | `imageGenerationConfig.numberOfImages` | 1-5 images |
**TaskType-Specific Parameter Mapping:**
| taskType | `prompt` maps to |
|---------------------------|------------------------------------|
| `TEXT_IMAGE` (default) | `textToImageParams.text` |
| `COLOR_GUIDED_GENERATION` | `colorGuidedGenerationParams.text` |
**Advanced Generation Modes:**
Default `taskType` is `"TEXT_IMAGE"`.
Available task types:
- `"TEXT_IMAGE"` - Standard text-to-image generation
- `"COLOR_GUIDED_GENERATION"` - Generate images based on color palette
```bash
# Color-Guided Generation
curl -X POST "$BASE/v1/images/generations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.titan-image-generator-v2:0",
"prompt": "Nature scene with colors",
"taskType": "COLOR_GUIDED_GENERATION",
"colorGuidedGenerationParams": {
"colors": ["#2ECC71", "#3498DB", "#F39C12"]
}
}'
```
!!! info "Full Parameter Reference"
For all parameters and task types, see [Amazon Titan Image Generator documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-titan-image.html)
#### { style="height: 1.2em; vertical-align: text-bottom;" } Stability AI Models
**Basic Usage (Standard OpenAI Parameters):**
```bash
curl -X POST "$BASE/v1/images/generations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "stability.stable-image-ultra-v1:1",
"prompt": "A photorealistic mountain landscape at sunset"
}'
```
**Parameter Mapping:**
| OpenAI Parameter | Maps to | Notes |
|------------------|----------------|----------------------------------------------|
| `prompt` | `prompt` | Text description for generation |
| `size` | `aspect_ratio` | Inferred from size (e.g., 1024x1024 → "1:1") |
| `n` | Multiple calls | Each image is a separate request |
**Model Comparison:**
| Model | Output Formats | Best For |
|-----------------------------------|-----------------|---------------------------------------|
| stability.sd3-5-large-v1:0 | png, jpeg, webp | High quality, versatile compositions |
| stability.stable-image-core-v1:1 | png, jpeg | Balanced quality and speed |
| stability.stable-image-ultra-v1:1 | png, jpeg | Premium quality and detail |
!!! info "Full Parameter Reference"
For all Stability AI parameters, see [Stability AI documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-stability-diffusion.html)
## Available Request Headers
This endpoint supports standard Bedrock headers for enhanced control over your requests. All headers are optional and can be combined as needed.
### Content Safety (Guardrails)
| Header | Purpose | Valid Values |
|--------------------------------------|------------------------------------|---------------------------------------|
| `X-Amzn-Bedrock-GuardrailIdentifier` | Guardrail ID for content filtering | Your guardrail identifier |
| `X-Amzn-Bedrock-GuardrailVersion` | Guardrail version | Version number (e.g., `1`) |
| `X-Amzn-Bedrock-Trace` | Guardrail trace level | `disabled`, `enabled`, `enabled_full` |
### Performance Optimization
| Header | Purpose | Valid Values |
|--------------------------------------------|------------------------|-------------------------------|
| `X-Amzn-Bedrock-Service-Tier` | Service tier selection | `priority`, `default`, `flex` |
| `X-Amzn-Bedrock-PerformanceConfig-Latency` | Latency optimization | `standard`, `optimized` |
**Example with headers:**
```bash
curl -X POST "$BASE/v1/images/generations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Amzn-Bedrock-Service-Tier: priority" \
-H "X-Amzn-Bedrock-PerformanceConfig-Latency: optimized" \
-d '{
"model": "amazon.nova-canvas-v1:0",
"prompt": "A serene mountain landscape at sunset"
}'
```
!!! info "Detailed Documentation"
For complete information about these headers, configuration options, and use cases, see:
- [Bedrock Guardrails Configuration](operations_configuration.md#bedrock-guardrails)
- [Service Tier and Performance Configuration](operations_configuration.md#bedrock-service-tier-and-performance-configuration)
## Try It Now
**Generate image (URL response):**
```bash
curl -X POST "$BASE/v1/images/generations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "A serene mountain landscape at sunset, photorealistic",
"model": "amazon.nova-canvas-v1:0",
"size": "1024x1024",
"quality": "high",
"response_format": "url"
}'
```
**Generate with base64 encoding:**
```bash
curl -X POST "$BASE/v1/images/generations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "A futuristic cityscape with flying cars, digital art style",
"model": "amazon.nova-canvas-v1:0",
"response_format": "b64_json"
}'
```
**Stream generation:**
Each image is sent as a single completed event; `partial_images` is accepted but never produces a preview.
```bash
curl -N -X POST "$BASE/v1/images/generations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "An abstract watercolor painting of emotions",
"model": "amazon.nova-canvas-v1:0",
"stream": true,
"n": 2
}'
```
---
**Unleash your creativity!** Explore available image models in the [Models API](api_openai_models.md).
---
## Images Edits
Source: https://stdapi.ai/md/api_openai_images_edits.md
# Images API - Image Editing
Edit images using inpainting with Amazon Bedrock image models through an OpenAI-compatible interface.
## Why Choose the Image Editing API?
- :material-image-edit: __Precise Control__
Edit specific regions of images while preserving the rest.
- :material-palette-advanced: __Creative Freedom__
Add, remove, or modify elements in existing images with AI assistance.
- :material-layers-triple: __Flexible Masking__
Define edit regions with an explicit mask image, using either alpha transparency or black/white pixels.
- :material-aws: __Scalable Infrastructure__
Edit images at scale with Amazon Bedrock infrastructure.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|--------------------|--------|-----------------------------------------|-----------------------------|--------------------|
| `/v1/images/edits` | `POST` | Edit images using prompts and masks | Amazon Bedrock Image Models | `openai_image_edit` |
## Feature Compatibility
| Feature | Status | Notes |
|--------------------------------|:----------------------------------------:|------------------------------------------------------------------------------------------------------------------------------------------------|
| **Editing** | | |
| Image-to-image (`/edits`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Edit images with prompts and masks |
| **Request Formats** | | |
| Multipart form-data | :material-check-circle:{ .success role="img" aria-label="Supported" } | Binary file uploads via `image` / `image[]` / `mask` fields |
| JSON body | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Structured `images` array with Files API IDs or URLs (the OpenAI edits API is multipart-only) |
| **Parameters** | | |
| `image` / `image[]` | :material-check-circle:{ .success role="img" aria-label="Supported" } | PNG image(s) to edit; every available model accepts exactly one source image and rejects requests providing more with an error |
| `images` (JSON) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Array of `{file_id}` or `{image_url}` references (JSON body, 1-16 entries) |
| `prompt` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Text description of desired changes |
| `mask` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Optional mask defining edit regions; models that do not use a mask reject requests that include one |
| `n` (number of images) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Multiple images per request; accepted range is 1-10, but the effective maximum is model-dependent (e.g. Amazon Titan and Nova Canvas cap at 5) |
| `size` (WIDTHxHEIGHT) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Output dimensions (default: 1024x1024, format validated; `auto` resolves to the default) |
| `model` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Required parameter |
| `response_format` | :material-check-circle:{ .success role="img" aria-label="Supported" } | `url` or `b64_json` (default: `url`) |
| `output_format` | :material-check-circle:{ .success role="img" aria-label="Supported" } | `png`, `jpeg`, or `webp` (model-specific) |
| `output_compression` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Compression level 1-100% (default: 100) |
| `quality` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Quality setting (default: `auto`, supports OpenAI & model-specific); accepted and ignored by models with no quality control |
| `stream` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Generate images in streaming mode, emitting the endpoint's `image_edit.partial_image` and `image_edit.completed` events |
| `partial_images` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted (0-3) but ignored — no available model currently streams partial images; the final image is always sent as a single event |
| `background` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Accepts `auto` (default) and `opaque`; `transparent` is unsupported — responses report `opaque` |
| `input_fidelity` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted for OpenAI API compatibility and ignored (always behaves as `low`) |
| **Output** | | |
| URL response format | :material-check-circle:{ .success role="img" aria-label="Supported" } | Temporary download URLs, valid for 60 minutes (requires AWS_S3_BUCKET) |
| Base64 JSON format | :material-check-circle:{ .success role="img" aria-label="Supported" } | Inline base64-encoded images |
| PNG format | :material-check-circle:{ .success role="img" aria-label="Supported" } | Lossless image output |
| JPEG format | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Lossy compression (model-specific) |
| WebP format | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Modern format with compression (model-specific) |
| Streaming response | :material-check-circle:{ .success role="img" aria-label="Supported" } | Server-sent events with final images (no partial previews) |
| **Usage tracking** | | |
| Input text tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Sourced from AWS billing data when available; remainder after subtracting image tokens |
| Input image tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Count of input images (image files + mask file), capped at the billed input tokens |
| Output image tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Sourced from AWS billing data when available; falls back to the image count (`n`) |
| **Other** | | |
| `user` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Logged but not used for abuse monitoring |
| Extra parameters via form data | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Provider-specific parameters passed through |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Available on Select Models** — Check your model's capabilities
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
## Model Support
!!! info "Model Support"
**Inpainting** (mask-based editing) is supported by **Amazon Nova Canvas**, **Amazon Titan Image Generator**, and **Stability AI** inpaint models.
**Image-to-image** (transformation without masks) is supported by **Stability AI** text-to-image models (SD3.5, Stable Image Core, Stable Image Ultra).
**Upscale** (resolution enhancement) is supported by **Stability AI** upscale models (creative, conservative, fast).
**Style Transfer** (applying reference image style) is supported by **Stability AI** style transfer models.
**Search-based editing** (find & replace/recolor objects) is supported by **Stability AI** search models.
**Background removal** is supported by **Amazon Titan Image Generator v2**, **Amazon Nova Canvas**, and **Stability AI** remove background model.
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Models
| Model | Supported Task Types | Mask Support | Notes |
|-----------------------------------|-----------------------------------------------------------------------------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------|
| amazon.nova-canvas-v1:0 (legacy) | `TEXT_IMAGE`, `INPAINTING`, `OUTPAINTING`, `BACKGROUND_REMOVAL`, `VIRTUAL_TRY_ON` | ✅ Required for inpainting/outpainting ✅ Used as reference for virtual try-on | Supports multiple editing modes including advanced virtual try-on with 3 mask types |
| amazon.titan-image-generator-v1 (legacy) | `INPAINTING`, `OUTPAINTING` | ✅ Required for inpainting/outpainting | Supports text-based mask prompts as alternative to mask images |
| amazon.titan-image-generator-v2:0 (legacy) | `INPAINTING`, `OUTPAINTING`, `BACKGROUND_REMOVAL` | ✅ Required for inpainting/outpainting ❌ Rejected for background removal | Enhanced features including background removal without mask |
!!! note "Legacy Amazon Image Models"
AWS has scheduled `amazon.nova-canvas-v1:0` and the Titan image models to reach end of life on September 30, 2026. Deployments with existing access can keep using them until then (legacy models are hidden unless [`AWS_BEDROCK_LEGACY=true`](operations_configuration.md#bedrock-legacy)); the Stability AI Stable Image family is the long-term successor.
!!! info "Amazon Nova Canvas Default Behavior"
**`amazon.nova-canvas-v1:0`** automatically selects the task type based on the presence of a mask when no `taskType` is explicitly provided:
- **No mask provided** → Uses `TEXT_IMAGE` by default
- **Mask provided** → Uses `INPAINTING` by default
### { style="height: 1.2em; vertical-align: text-bottom;" } Stability AI Models
#### Image-to-Image Models
| Model | Prompt Usage | Mask Usage | Extra Parameters Required | Notes |
|-----------------------------------|------------------------|----------------------|---------------------------|-----------------------------------|
| stability.sd3-5-large-v1:0 | Guides transformation | Rejected if provided | None | Transform images with prompt |
| stability.stable-image-core-v1:1 | Guides transformation | Rejected if provided | None | Balanced quality and speed |
| stability.stable-image-ultra-v1:1 | Guides transformation | Rejected if provided | None | Premium quality and detail |
#### Upscale Models
| Model | Prompt Usage | Mask Usage | Extra Parameters Required | Notes |
|--------------------------------------------|------------------------|----------------------|---------------------------|-----------------------------------------|
| stability.stable-creative-upscale-v1:0 | Guides upscaling | Rejected if provided | None | Prompt-guided upscaling with creativity |
| stability.stable-conservative-upscale-v1:0 | Guides upscaling | Rejected if provided | None | Detail-preserving upscaling |
| stability.stable-fast-upscale-v1:0 | **Not used** | Rejected if provided | None | Fast 4x upscaling without prompt |
#### Edit Models
| Model | Prompt Usage | Mask Usage | Extra Parameters Required | Notes |
|-----------------------------------------------|---------------------------|------------------------------|---------------------------|----------------------------------|
| stability.stable-image-inpaint-v1:0 | Guides inpainting | Optional (marks edit region) | None | Fill masked regions |
| stability.stable-outpaint-v1:0 | Guides outpainting | Rejected if provided | None | Extend image beyond borders |
| stability.stable-image-search-recolor-v1:0 | Describes new color | Rejected if provided | `select_prompt` | Recolor objects by search prompt |
| stability.stable-image-search-replace-v1:0 | Describes replacement | Rejected if provided | `search_prompt` | Replace objects by search prompt |
| stability.stable-image-erase-object-v1:0 | **Not used** | Required (marks object) | None | Remove objects with mask |
| stability.stable-image-remove-background-v1:0 | **Not used** | Rejected if provided | None | Automatic background removal |
#### Control Models
| Model | Prompt Usage | Mask Usage | Extra Parameters Required | Notes |
|-----------------------------------------------|----------------------|----------------------|---------------------------|---------------------------------|
| stability.stable-image-control-sketch-v1:0 | Guides generation | Rejected if provided | None | Generate from sketch |
| stability.stable-image-control-structure-v1:0 | Guides generation | Rejected if provided | None | Structure-preserving generation |
#### Style Models
| Model | Prompt Usage | Mask Usage | Extra Parameters Required | Notes |
|-----------------------------------------|-----------------------|----------------------------------------|---------------------------|-------------------------------|
| stability.stable-image-style-guide-v1:0 | Guides style | Rejected if provided | None | Extract and apply style |
| stability.stable-style-transfer-v1:0 | Guides style transfer | Required (repurposed as `style_image`) | None | Transfer style between images |
!!! note "Output Formats"
All models support standard OpenAI output formats (`png`, `jpeg`, `webp`) via the `output_format` parameter. When a model cannot produce the requested format natively, the gateway re-encodes the result server-side, so the response always carries the format you asked for.
!!! warning "Extra Parameters Required"
Some models require parameters **beyond the standard OpenAI API**:
- **`stability.stable-image-search-recolor-v1:0`**: Requires `select_prompt` form field
- **`stability.stable-image-search-replace-v1:0`**: Requires `search_prompt` form field
**Models that don't use prompt**: `stability.stable-fast-upscale-v1:0`, `stability.stable-image-erase-object-v1:0`, `stability.stable-image-remove-background-v1:0` - provide empty string or omit the `prompt` parameter.
All other Stability models use only standard OpenAI parameters (`image`, `prompt`, and optionally `mask`).
!!! info "No Built-In Aliases for OpenAI Image Model Names"
OpenAI's default image model names (`dall-e-2`, `dall-e-3`, `gpt-image-1`) have **no built-in alias**, so requests using them fail with a model-not-found error — the most common first-call issue. Pass one of the model IDs above, or map the OpenAI names to your preferred models with [`MODEL_ALIASES`](operations_configuration.md#model-aliases).
!!! warning "Configuration Required"
You must configure the `AWS_S3_BUCKET` environment variable with a bucket to use the URL response format.
## Advanced Features
### Request Formats
The `/v1/images/edits` endpoint accepts two request formats:
#### Multipart Form-Data (Binary Uploads)
The classic format — upload image files directly. Use `image` (single) or `image[]` (multiple) for source images and `mask` for the optional edit mask.
```bash
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F image=@source.png \
-F mask=@mask.png \
-F prompt="A red apple on a wooden table" \
-F model="amazon.nova-canvas-v1:0"
```
#### JSON Body (Files API or URL References) :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" }
The modern format — reference images already stored in the Files API or accessible via URL. Send `Content-Type: application/json` with an `images` array (1-16 entries), where each element has either `file_id` or `image_url`:
```bash
# Edit using a Files API file ID
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-canvas-v1:0",
"prompt": "A red apple on a wooden table",
"images": [{"file_id": "file-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}],
"response_format": "b64_json",
"size": "1024x1024"
}'
# Edit using an HTTP URL
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-canvas-v1:0",
"prompt": "Add a dramatic sky",
"images": [{"image_url": "https://example.com/photo.png"}],
"mask": {"file_id": "file-mxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"},
"size": "1024x1024"
}'
```
**`ImageRef` object** (used in `images` array and `mask` field):
| Field | Type | Description |
|-------------|--------|----------------------------------------------------------|
| `file_id` | string | Files API file identifier (`file-*` or `file_*` prefix) |
| `image_url` | string | HTTP/HTTPS URL, data URI (`data:image/png;base64,...`), S3 URI (`s3://bucket/key`), or Files API reference (`file-id:file-` — see [Files API](api_openai_files.md#referencing-uploaded-files-via-the-file-id-uri-scheme)) |
Provide one of `file_id` or `image_url` per `ImageRef`; if both are given,
`file_id` takes precedence. Each array element may also be a plain reference
string (equivalent to `image_url`), and the array is additionally accepted
under the `image` key — the shapes MCP clients derive from the tool schema:
```json
{"model": "amazon.nova-canvas-v1:0", "prompt": "Add a dramatic sky", "image": ["data:image/png;base64,..."]}
```
!!! tip "Workflow Integration"
The JSON body format works seamlessly with the [Files API](api_openai_files.md): upload images once, reuse them across multiple edit requests by file ID without re-uploading.
### How Image Editing Works
#### { style="height: 1.2em; vertical-align: text-bottom;" } Image-to-Image (Stability AI Models)
Stability AI models support image-to-image transformation without masks. The source image is transformed according to the prompt:
```bash
# Transform a photo into an oil painting style
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@photo.png \
-F prompt="Transform into an oil painting style" \
-F model="stability.sd3-5-large-v1:0"
```
!!! warning "Mask Not Supported"
Stability AI image-to-image models do not support mask-based editing. Providing a `mask` parameter will result in an error.
#### { style="height: 1.2em; vertical-align: text-bottom;" } Upscale (Stability AI)
Upscale models increase image resolution while preserving quality:
```bash
# Fast upscaling (4x) - no prompt parameter needed
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@low_res.png \
-F model="stability.stable-fast-upscale-v1:0"
```
!!! note "Upscale Characteristics"
- **Fast Upscale**: Conservative 4x upscaling that preserves original details
- No prompt parameter needed or used
- Best for enlarging photos and preserving original content
#### { style="height: 1.2em; vertical-align: text-bottom;" } Style Transfer (Stability AI)
Apply visual characteristics from one image to another. The `mask` parameter is used to pass the style reference image:
```bash
# Apply style from reference image to target image
# image: content image, mask: style reference image
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@content.png \
-F mask=@style_reference.png \
-F prompt="Apply artistic style while preserving content" \
-F model="stability.stable-style-transfer-v1:0"
```
!!! note "Style Transfer Parameter Mapping"
- **`image`** (required): Target image to apply style to
- **`mask`** (required): Maps to `style_image` - the reference style image
- **`prompt`**: Guides the style application process
#### Inpainting with Masks (Amazon Models and Stability AI)
An image submitted without a `mask` is **not** auto-masked from its own transparency:
it is sent as a conditioning image for text-to-image generation instead of an
inpainting edit. To edit specific regions, always provide an explicit `mask`.
**With Explicit Mask:**
Provide an explicit mask image where transparent areas indicate regions to edit:
```bash
# Edit with explicit mask
# image: source image, mask: PNG where transparent areas mark edit regions
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@source.png \
-F mask=@edit_mask.png \
-F prompt="A beautiful flower" \
-F model="amazon.nova-canvas-v1:0"
```
**Mask format**: PNG with alpha channel where transparent pixels indicate regions to
edit, opaque pixels are preserved (standard OpenAI edits-API mask). A mask with an
alpha channel is automatically converted to the black/white RGB format each backend
requires (Nova Canvas, Titan, and the Stability AI inpaint/erase-object models); a
mask that is already black/white RGB (no alpha channel) is passed through unchanged.
### Provider-Specific Parameters
#### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Nova Canvas
**Basic Usage (Standard OpenAI Parameters):**
```bash
# Inpainting with mask
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@source.png \
-F mask=@mask.png \
-F prompt="A sunset over mountains" \
-F model="amazon.nova-canvas-v1:0"
```
**Parameter Mapping:**
| OpenAI Parameter | Maps to | Notes |
|---------------------|----------------------------------------|-------------------------------------------------------------|
| `prompt` | Depends on `taskType` | See taskType-specific mapping below |
| `image` / `image[]` | Depends on `taskType` | See taskType-specific mapping below (single image required) |
| `mask` | Depends on `taskType` | See taskType-specific mapping below |
| `size` | `imageGenerationConfig.width/height` | Output dimensions (320-4096) |
| `quality` | `imageGenerationConfig.quality` | "high" → "premium" |
| `n` | `imageGenerationConfig.numberOfImages` | 1-5 images |
**TaskType-Specific Parameter Mapping:**
| taskType | `prompt` maps to | `image` maps to | `mask` maps to |
|-----------------------------------|---------------------------------------------------------------|------------------------------------|-------------------------------------|
| `TEXT_IMAGE` (default, no mask) | `textToImageParams.text` | `textToImageParams.conditionImage` | Not used |
| `INPAINTING` (default with mask) | `inPaintingParams.text` | `inPaintingParams.image` | `inPaintingParams.maskImage` |
| `OUTPAINTING` | `outPaintingParams.text` | `outPaintingParams.image` | `outPaintingParams.maskImage` |
| `BACKGROUND_REMOVAL` | Not used | `backgroundRemovalParams.image` | Rejected if provided |
| `VIRTUAL_TRY_ON` (PROMPT) | `promptBasedMask.maskPrompt` | `virtualTryOnParams.sourceImage` | `virtualTryOnParams.referenceImage` |
| `VIRTUAL_TRY_ON` (GARMENT) | `garmentBasedMask.garmentClass` | `virtualTryOnParams.sourceImage` | `virtualTryOnParams.referenceImage` |
| `VIRTUAL_TRY_ON` (IMAGE) | `imageBasedMask.maskImage` (Base64 encoded image or data URI) | `virtualTryOnParams.sourceImage` | `virtualTryOnParams.referenceImage` |
**Advanced Task Types (with form fields):**
Default `taskType` is `"INPAINTING"` when a mask is provided, `"TEXT_IMAGE"` otherwise.
Available task types:
- `"TEXT_IMAGE"` - Prompt-driven transformation using the source image as condition
- `"INPAINTING"` - Fill masked regions
- `"OUTPAINTING"` - Extend image beyond borders
- `"BACKGROUND_REMOVAL"` - Remove background
- `"VIRTUAL_TRY_ON"` - Virtual fashion try-on
```bash
# Outpainting
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@photo.png \
-F prompt="Extend with a garden" \
-F model="amazon.nova-canvas-v1:0" \
-F taskType="OUTPAINTING"
# Background Removal (no prompt needed)
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@photo.png \
-F model="amazon.nova-canvas-v1:0" \
-F taskType="BACKGROUND_REMOVAL"
# Virtual Try-On - Prompt-Based (default)
# image: person photo, mask: garment image, prompt: area description
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@person.png \
-F mask=@garment.png \
-F prompt="upper body area" \
-F model="amazon.nova-canvas-v1:0" \
-F taskType="VIRTUAL_TRY_ON"
# Virtual Try-On - Garment-Based
# image: person photo, mask: garment image, prompt: garment class
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@person.png \
-F mask=@garment.png \
-F prompt="UPPER_BODY" \
-F model="amazon.nova-canvas-v1:0" \
-F taskType="VIRTUAL_TRY_ON" \
-F "virtualTryOnParams[maskType]=GARMENT"
# Virtual Try-On - Image-Based Mask
# image: person photo, mask: garment image, prompt: base64 mask image
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@person.png \
-F mask=@garment.png \
-F prompt="BASE64_MASK_IMAGE" \
-F model="amazon.nova-canvas-v1:0" \
-F taskType="VIRTUAL_TRY_ON" \
-F "virtualTryOnParams[maskType]=IMAGE"
```
!!! info "Full Parameter Reference"
For all available parameters and task types, see [Amazon Nova Canvas documentation](https://docs.aws.amazon.com/nova/latest/userguide/image-generation.html)
#### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Titan Image Generator
**Basic Usage (Standard OpenAI Parameters):**
```bash
# Inpainting with mask
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@source.png \
-F mask=@mask.png \
-F prompt="A beautiful garden with flowers" \
-F model="amazon.titan-image-generator-v2:0"
```
**Parameter Mapping:**
| OpenAI Parameter | Maps to | Notes |
|---------------------|----------------------------------------|-------------------------------------------------------------|
| `prompt` | Depends on `taskType` | See taskType-specific mapping below |
| `image` / `image[]` | Depends on `taskType` | See taskType-specific mapping below (single image required) |
| `mask` | Depends on `taskType` | See taskType-specific mapping below |
| `size` | `imageGenerationConfig.width/height` | Fixed sizes (512-2048) |
| `quality` | `imageGenerationConfig.quality` | "high" → "premium" |
| `n` | `imageGenerationConfig.numberOfImages` | 1-5 images |
**TaskType-Specific Parameter Mapping:**
| taskType | `prompt` maps to | `image` maps to | `mask` maps to |
|------------------------|--------------------------|---------------------------------|-------------------------------|
| `INPAINTING` (default) | `inPaintingParams.text` | `inPaintingParams.image` | `inPaintingParams.maskImage` |
| `OUTPAINTING` | `outPaintingParams.text` | `outPaintingParams.image` | `outPaintingParams.maskImage` |
| `BACKGROUND_REMOVAL` | Not used | `backgroundRemovalParams.image` | Rejected if provided |
**Advanced Task Types (with form fields):**
Default `taskType` is `"INPAINTING"`.
Available task types:
- `"INPAINTING"` - Fill masked regions
- `"OUTPAINTING"` - Extend image beyond borders
- `"BACKGROUND_REMOVAL"` (v2 only) - Remove background
```bash
# Outpainting
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@photo.png \
-F prompt="Extend with a forest" \
-F model="amazon.titan-image-generator-v2:0" \
-F taskType="OUTPAINTING"
# Background Removal (v2 only, no prompt needed)
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@photo.png \
-F model="amazon.titan-image-generator-v2:0" \
-F taskType="BACKGROUND_REMOVAL"
```
!!! info "Full Parameter Reference"
For all available parameters and task types, see [Amazon Titan Image Generator documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-titan-image.html)
#### { style="height: 1.2em; vertical-align: text-bottom;" } Stability AI Models
**Basic Usage (Standard OpenAI Parameters):**
Most Stability AI models work with standard OpenAI parameters:
```bash
# Image-to-image transformation
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@photo.png \
-F prompt="A dramatic cinematic scene" \
-F model="stability.sd3-5-large-v1:0"
```
**Parameter Mapping:**
All Stability AI models use standard OpenAI parameters directly:
| OpenAI Parameter | Stability Parameter | Notes |
|---------------------|---------------------|----------------------------------------------------|
| `image` / `image[]` | `image` | Base64-encoded input image (single image required) |
| `prompt` | `prompt` | Text description (may be unused for some models) |
| `mask` | `mask` | Base64-encoded mask (model-specific) |
| `n` | Multiple requests | Generates N images via multiple API calls |
| `size` | Model-specific | Some models support width/height |
**Model-Specific Parameters:**
| Model(s) | Required Form Fields | OpenAI `mask` Maps To | Notes |
|---------------------------------------|--------------------------|-----------------------|------------------------------------------|
| `stable-image-search-recolor-v1:0` | `select_prompt` (string) | Not used | Identifies object to recolor |
| `stable-image-search-replace-v1:0` | `search_prompt` (string) | Not used | Identifies object to find and replace |
| `stable-style-transfer-v1:0` | None (uses `mask` param) | `style_image` | Mask parameter repurposed as style image |
| `stable-image-erase-object-v1:0` | None | `mask` (required) | Prompt not used |
| `stable-image-remove-background-v1:0` | None | Not used | Prompt not used |
| `stable-fast-upscale-v1:0` | None | Not used | Prompt not used |
!!! info "Full Parameter Reference"
For all Stability AI parameters, see [Stability AI documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-stability-diffusion.html)
## Available Request Headers
This endpoint supports the same standard Bedrock headers as the other images endpoints: guardrail headers (`X-Amzn-Bedrock-GuardrailIdentifier`, `X-Amzn-Bedrock-GuardrailVersion`, `X-Amzn-Bedrock-Trace`) and performance headers (`X-Amzn-Bedrock-Service-Tier`, `X-Amzn-Bedrock-PerformanceConfig-Latency`). All headers are optional and can be combined as needed.
See the [Images Generation API headers reference](api_openai_images_generations.md#available-request-headers) for the header tables, valid values, and configuration links.
**Example with headers:**
```bash
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "X-Amzn-Bedrock-Service-Tier: priority" \
-F image=@source.png \
-F prompt="A red apple on a wooden table" \
-F model="amazon.nova-canvas-v1:0"
```
## Try It Now
### { style="height: 1.2em; vertical-align: text-bottom;" } Image-to-Image with Stability AI
```bash
# Transform image with default strength (0.35)
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@photo.png \
-F prompt="Transform into a watercolor painting" \
-F model="stability.sd3-5-large-v1:0"
```
### { style="height: 1.2em; vertical-align: text-bottom;" } Upscale with Stability AI
```bash
# Fast upscaling (4x resolution increase) - no prompt needed
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@low_res.png \
-F model="stability.stable-fast-upscale-v1:0"
```
### { style="height: 1.2em; vertical-align: text-bottom;" } Style Transfer with Stability AI
```bash
# Apply style from reference image (mask parameter is style image)
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@content.png \
-F mask=@style_reference.png \
-F prompt="Apply artistic style" \
-F model="stability.stable-style-transfer-v1:0"
```
### { style="height: 1.2em; vertical-align: text-bottom;" } Search & Replace with Stability AI
```bash
# Replace objects by search prompt (requires search_prompt form field)
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F prompt="a red car" \
-F model="stability.stable-image-search-replace-v1:0" \
-F search_prompt="blue car"
```
### { style="height: 1.2em; vertical-align: text-bottom;" } Search & Recolor with Stability AI
```bash
# Recolor objects by search prompt (requires select_prompt form field)
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F prompt="bright red color" \
-F model="stability.stable-image-search-recolor-v1:0" \
-F select_prompt="car"
```
### { style="height: 1.2em; vertical-align: text-bottom;" } Erase Object with Stability AI
```bash
# Erase object with mask - no prompt needed
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F mask=@object_mask.png \
-F model="stability.stable-image-erase-object-v1:0"
```
### { style="height: 1.2em; vertical-align: text-bottom;" } Remove Background with Stability AI
```bash
# Remove background automatically - no prompt needed
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F model="stability.stable-image-remove-background-v1:0"
```
### Inpainting with Amazon Models
```bash
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F image=@image.png \
-F prompt="A blue ocean with sailboats" \
-F model="amazon.nova-canvas-v1:0"
```
### Edit with Explicit Mask
```bash
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F image=@source.png \
-F mask=@mask.png \
-F prompt="A red sports car" \
-F model="amazon.nova-canvas-v1:0"
```
### Base64 Response Format
```bash
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F image=@image.png \
-F prompt="A sunny day with blue sky" \
-F model="amazon.nova-canvas-v1:0" \
-F response_format="b64_json"
```
### Multiple Edited Images
```bash
# Generate three edited images from the same source (n parameter)
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F image=@image.png \
-F prompt="A fantasy castle" \
-F n=3 \
-F model="amazon.nova-canvas-v1:0"
```
### The `image[]` Array Parameter
!!! warning "One Source Image Per Request"
The schema accepts the repeated `image[]` multipart parameter for OpenAI wire compatibility, but every model currently available through the gateway edits exactly one source image and rejects a request carrying more than one with an error. Send a single `image` field, or an `image[]` array with a single entry.
```bash
# OpenAI-compatible array syntax, with the single source image every model expects
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F "image[]=@gift-basket.png" \
-F prompt="Add a ribbon around the basket" \
-F model="amazon.nova-canvas-v1:0"
```
### Generate from Sketch or Structure (Control Models)
```bash
# Control Sketch - generate from sketch
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@sketch.png \
-F prompt="A realistic portrait" \
-F model="stability.stable-image-control-sketch-v1:0"
# Control Structure - preserve structure
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@structure.png \
-F prompt="A modern building" \
-F model="stability.stable-image-control-structure-v1:0"
```
### Inpainting & Outpainting with Stability AI
```bash
# Stability AI Inpainting - mask marks edit region
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@photo.png \
-F mask=@edit_mask.png \
-F prompt="A beautiful sunset" \
-F model="stability.stable-image-inpaint-v1:0"
# Outpainting - extend image beyond borders
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@photo.png \
-F prompt="Extend with a forest landscape" \
-F model="stability.stable-outpaint-v1:0"
```
### Style Guide
```bash
# Extract and apply style from reference
curl -X POST "$BASE/v1/images/edits" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@content.png \
-F prompt="Apply impressionist style" \
-F model="stability.stable-image-style-guide-v1:0"
```
---
**Ready to transform your images?** Explore available image models in the [Models API](api_openai_models.md).
---
## Images Variations
Source: https://stdapi.ai/md/api_openai_images_variations.md
# Images API - Image Variations
Create variations of existing images using Amazon Bedrock image models through an OpenAI-compatible interface.
## Why Choose the Image Variations API?
- :material-image-multiple: __Multiple Variations__
Generate diverse versions of an existing image while maintaining composition.
- :material-palette-swatch: __Artistic Exploration__
Explore different artistic interpretations and styles.
- :material-auto-fix: __Quick Iterations__
Rapidly create variations without manual editing.
- :material-aws: __Flexible Models__
Amazon Titan, Amazon Nova Canvas, and Stability AI SD3.5 — each with unique variation modes (standard variation, text-guided conditioning, color-guided generation).
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|-------------------------|--------|----------------------------------------|-----------------------------|------------------------|
| `/v1/images/variations` | `POST` | Create variations of an existing image | Amazon Bedrock Image Models | `openai_image_variation` |
## Feature Compatibility
| Feature | Status | Notes |
|--------------------------------|:----------------------------------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Variations** | | |
| Image-to-image (`/variations`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Create variations of existing images |
| **Parameters** | | |
| `image` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Source image file (required) |
| `model` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Required parameter |
| `n` (number of images) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Multiple variations per request; accepted range is 1-10 (default: 1), but the effective maximum is model-dependent (e.g. Amazon Titan and Nova Canvas cap at 5) |
| `size` (WIDTHxHEIGHT) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Output dimensions (default: 1024x1024, format validated; `auto` resolves to the default) |
| `response_format` | :material-check-circle:{ .success role="img" aria-label="Supported" } | `url` or `b64_json` (default: `url`) |
| **Output** | | |
| URL response format | :material-check-circle:{ .success role="img" aria-label="Supported" } | Temporary download URLs, valid for 60 minutes (requires AWS_S3_BUCKET) |
| Base64 JSON format | :material-check-circle:{ .success role="img" aria-label="Supported" } | Inline base64-encoded images |
| PNG format | :material-check-circle:{ .success role="img" aria-label="Supported" } | Default output format |
| JPEG format | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Via the provider-specific `output_format` extra parameter on supporting models (`/variations` has no `output_format` parameter) |
| WebP format | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Via the provider-specific `output_format` extra parameter on supporting models (`/variations` has no `output_format` parameter) |
| **Usage tracking** | | |
| Input image tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Count of input images (always 1 for variations) |
| Output image tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Sourced from AWS billing data when available; falls back to the image count (`n`) |
| **Other** | | |
| `user` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Logged but not used for abuse monitoring |
| Extra parameters via form data | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Provider-specific parameters passed through |
| JSON body request format | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Reference images via Files API ID or URL instead of file upload |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Available on Select Models** — Check your model's capabilities
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
## Model Support
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Models
| Model | Supported Task Types | Notes |
|-----------------------------------|------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|
| amazon.nova-canvas-v1:0 (legacy) | `IMAGE_VARIATION`, `TEXT_IMAGE`, `COLOR_GUIDED_GENERATION` | Supports standard variations plus text-guided conditioning and color-guided generation with 8 style presets |
| amazon.titan-image-generator-v1 (legacy) | `IMAGE_VARIATION` | Basic variation support with similarity control |
| amazon.titan-image-generator-v2:0 (legacy) | `IMAGE_VARIATION`, `TEXT_IMAGE`, `COLOR_GUIDED_GENERATION` | Enhanced with text-guided conditioning (CANNY_EDGE, SEGMENTATION) and color-guided generation |
!!! note "Legacy Amazon Image Models"
AWS has scheduled `amazon.nova-canvas-v1:0` and the Titan image models to reach end of life on September 30, 2026. Deployments with existing access can keep using them until then (legacy models are hidden unless [`AWS_BEDROCK_LEGACY=true`](operations_configuration.md#bedrock-legacy)); the Stability AI Stable Image family is the long-term successor.
### { style="height: 1.2em; vertical-align: text-bottom;" } Stability AI Models
| Model | Notes |
|-----------------------------------|-----------------------------------------------------------|
| stability.sd3-5-large-v1:0 | Image-to-image transformation |
| stability.stable-image-core-v1:1 | Image-to-image transformation, balanced quality and speed |
| stability.stable-image-ultra-v1:1 | Image-to-image transformation, premium quality and detail |
!!! info "No Built-In Aliases for OpenAI Image Model Names"
OpenAI's default image model names (`dall-e-2`, `dall-e-3`, `gpt-image-1`) have **no built-in alias**, so requests using them fail with a model-not-found error — the most common first-call issue. Pass one of the model IDs above, or map the OpenAI names to your preferred models with [`MODEL_ALIASES`](operations_configuration.md#model-aliases).
!!! warning "Configuration Required"
You must configure the `AWS_S3_BUCKET` environment variable with a bucket to use the URL response format.
## Advanced Features
### Request Formats
The `/v1/images/variations` endpoint accepts two request formats:
#### Multipart Form-Data (Binary Uploads)
The classic format — upload an image file directly via the `image` field.
```bash
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F image=@input.png \
-F model="amazon.nova-canvas-v1:0"
```
#### JSON Body (Files API or URL References) :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" }
Reference an image already stored in the Files API or accessible via URL. Send `Content-Type: application/json` with an `image` object containing either `file_id` or `image_url`:
```bash
# Variation from a Files API file ID
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-canvas-v1:0",
"image": {"file_id": "file-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"},
"n": 2,
"size": "1024x1024"
}'
# Variation from an HTTP URL
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-canvas-v1:0",
"image": {"image_url": "https://example.com/photo.png"},
"response_format": "b64_json"
}'
```
**`image` object fields:**
| Field | Type | Description |
|-------------|--------|----------------------------------------------------------|
| `file_id` | string | Files API file identifier (`file-*` or `file_*` prefix) |
| `image_url` | string | HTTP/HTTPS URL, data URI (`data:image/png;base64,...`), S3 URI (`s3://bucket/key`), or Files API reference (`file-id:file-` — see [Files API](api_openai_files.md#referencing-uploaded-files-via-the-file-id-uri-scheme)) |
Provide one of `file_id` or `image_url`; if both are given, `file_id` takes
precedence. `image` may also be a plain reference string (equivalent to
`image_url`) — the shape MCP clients derive from the tool schema:
```json
{"model": "amazon.nova-canvas-v1:0", "image": "data:image/png;base64,..."}
```
!!! tip "Workflow Integration"
The JSON body format works seamlessly with the [Files API](api_openai_files.md): upload images once, reuse them across multiple variation requests by file ID without re-uploading.
### Provider-Specific Parameters
#### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Nova Canvas
**Basic Usage (Standard OpenAI Parameters):**
```bash
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F model="amazon.nova-canvas-v1:0"
```
**Parameter Mapping:**
| OpenAI Parameter | Maps to | Notes |
|------------------|----------------------------------------|-------------------------------------|
| `image` | Depends on `taskType` | See taskType-specific mapping below |
| `size` | `imageGenerationConfig.width/height` | Flexible (320-4096) |
| `style` | `textToImageParams.style` | Not a standard variations field — pass via extra param `textToImageParams[style]` (TEXT_IMAGE task) |
| `n` | `imageGenerationConfig.numberOfImages` | 1-5 variations |
**TaskType-Specific Parameter Mapping:**
| taskType | `image` maps to |
|-----------------------------|----------------------------------------------|
| `IMAGE_VARIATION` (default) | `imageVariationParams.images` |
| `TEXT_IMAGE` | `textToImageParams.conditionImage` |
| `COLOR_GUIDED_GENERATION` | `colorGuidedGenerationParams.referenceImage` |
**Advanced Variation Modes (with form fields):**
Default `taskType` is `"IMAGE_VARIATION"`.
Available task types:
- `"IMAGE_VARIATION"` - Standard image variations
- `"TEXT_IMAGE"` - Text-guided generation with condition image
- `"COLOR_GUIDED_GENERATION"` - Color palette-based variations
```bash
# Text-to-Image with Condition
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F model="amazon.nova-canvas-v1:0" \
-F taskType="TEXT_IMAGE" \
-F "textToImageParams[text]=Photorealistic version"
# Color-Guided Variations
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F model="amazon.nova-canvas-v1:0" \
-F taskType="COLOR_GUIDED_GENERATION" \
-F "colorGuidedGenerationParams[colors][]=#FF6B35" \
-F "colorGuidedGenerationParams[colors][]=#F7931E" \
-F "colorGuidedGenerationParams[colors][]=#FDC830"
```
!!! info "Full Parameter Reference"
For all parameters, styles, and task types, see [Amazon Nova Canvas documentation](https://docs.aws.amazon.com/nova/latest/userguide/image-generation.html)
#### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Titan Image Generator
**Basic Usage (Standard OpenAI Parameters):**
```bash
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F model="amazon.titan-image-generator-v2:0"
```
**Parameter Mapping:**
| OpenAI Parameter | Maps to | Notes |
|------------------|----------------------------------------|-------------------------------------|
| `image` | Depends on `taskType` | See taskType-specific mapping below |
| `size` | `imageGenerationConfig.width/height` | Fixed sizes (512-2048) |
| `n` | `imageGenerationConfig.numberOfImages` | 1-5 variations |
**TaskType-Specific Parameter Mapping:**
| taskType | `image` maps to |
|-----------------------------|----------------------------------------------|
| `IMAGE_VARIATION` (default) | `imageVariationParams.images` |
| `TEXT_IMAGE` | `textToImageParams.conditionImage` |
| `COLOR_GUIDED_GENERATION` | `colorGuidedGenerationParams.referenceImage` |
**Advanced Variation Modes (with form fields):**
Default `taskType` is `"IMAGE_VARIATION"`.
Available task types:
- `"IMAGE_VARIATION"` - Standard image variations
- `"TEXT_IMAGE"` - Text-guided generation with condition image
- `"COLOR_GUIDED_GENERATION"` - Color palette-based variations
```bash
# Text-Guided Condition Image
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F model="amazon.titan-image-generator-v2:0" \
-F taskType="TEXT_IMAGE" \
-F "textToImageParams[text]=Photorealistic version"
# Color-Guided Variations
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F model="amazon.titan-image-generator-v2:0" \
-F taskType="COLOR_GUIDED_GENERATION" \
-F "colorGuidedGenerationParams[colors][]=#FF6B6B" \
-F "colorGuidedGenerationParams[colors][]=#4ECDC4" \
-F "colorGuidedGenerationParams[colors][]=#45B7D1"
```
!!! info "Full Parameter Reference"
For all parameters and task types, see [Amazon Titan Image Generator documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-titan-image.html)
#### { style="height: 1.2em; vertical-align: text-bottom;" } Stability AI Models
**Basic Usage (Standard OpenAI Parameters):**
```bash
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F model="stability.sd3-5-large-v1:0"
```
**Parameter Mapping:**
| OpenAI Parameter | Maps to | Notes |
|------------------|------------------|-----------------------------------------------|
| `image` | `image` (base64) | Source image for variation |
| `size` | `aspect_ratio` | Inferred from size (e.g., 1024x1024 → "1:1") |
| `n` | Multiple calls | Each variation is a separate request |
**Model-Specific Features:**
| Model | Output Formats | Notes |
|----------------------------|-----------------|-------------------------------|
| stability.sd3-5-large-v1:0 | png, jpeg, webp | Image-to-image transformation |
!!! info "Full Parameter Reference"
For all Stability AI parameters, see [Stability AI documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-stability-diffusion.html)
## Available Request Headers
This endpoint supports the same standard Bedrock headers as the other images endpoints: guardrail headers (`X-Amzn-Bedrock-GuardrailIdentifier`, `X-Amzn-Bedrock-GuardrailVersion`, `X-Amzn-Bedrock-Trace`) and performance headers (`X-Amzn-Bedrock-Service-Tier`, `X-Amzn-Bedrock-PerformanceConfig-Latency`). All headers are optional and can be combined as needed.
See the [Images Generation API headers reference](api_openai_images_generations.md#available-request-headers) for the header tables, valid values, and configuration links.
**Example with headers:**
```bash
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "X-Amzn-Bedrock-Service-Tier: priority" \
-F image=@input.png \
-F model="amazon.nova-canvas-v1:0"
```
## Try It Now
**Create a simple variation:**
```bash
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F model="amazon.nova-canvas-v1:0"
```
**Create multiple variations:**
```bash
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F model="amazon.nova-canvas-v1:0" \
-F n=3
```
**Base64 response format:**
```bash
curl -X POST "$BASE/v1/images/variations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F image=@input.png \
-F model="amazon.nova-canvas-v1:0" \
-F response_format="b64_json"
```
---
**Ready to explore new versions of your images?** Discover available image models in the [Models API](api_openai_models.md).
---
## Videos
Source: https://stdapi.ai/md/api_openai_videos.md
# Videos API
Generate videos from text prompts and reference images with Amazon Bedrock video models through the OpenAI Videos API shape.
Video generation is **asynchronous**: creating a video returns a job object immediately. Poll the job until its status is `completed`, then download the MP4 content — exactly like the OpenAI (Sora) workflow, so the official OpenAI SDKs work unchanged.
## Why Choose the Videos API?
- :material-movie-open: __Text-to-Video & Image-to-Video__
Generate studio-quality clips from a prompt, optionally guided by a reference image used as the first frame.
- :material-swap-horizontal: __Drop-in OpenAI Compatibility__
Follows the OpenAI Videos API shape. `client.videos.create(...)` works by changing the base URL.
- :material-server-off: __Durable Job IDs__
Video IDs stay valid across restarts and behind a load balancer — retrieve, download and delete work from any instance.
- :material-cloud-lock: __Private AWS Backend__
Videos are generated by Amazon Bedrock and stored in your own S3 buckets — no traffic to third-party endpoints.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|--------------------------------|----------|--------------------------------------------------|-----------------------------|--------------------------|
| `/v1/videos` | `POST` | Start an asynchronous video generation job | Amazon Bedrock Video Models | `openai_video_generation` |
| `/v1/videos` | `GET` | List video generation jobs across regions | Amazon Bedrock | `openai_video_list` |
| `/v1/videos/{video_id}` | `GET` | Retrieve the current state of a job | Amazon Bedrock | `openai_video_get` |
| `/v1/videos/{video_id}/content`| `GET` | Download the generated MP4 once completed | Amazon S3 | `openai_video_content` |
| `/v1/videos/{video_id}` | `DELETE` | Delete the stored video output | Amazon S3 | `openai_video_delete` |
**Example with the OpenAI Python SDK:**
```python
import time
from openai import OpenAI
client = OpenAI(base_url="https://your-host/v1", api_key="your-api-key")
video = client.videos.create(
model="luma.ray-v2:0", prompt="Closeup of a seashell on a sandy beach, gentle waves"
)
while video.status in ("queued", "in_progress"):
time.sleep(10)
video = client.videos.retrieve(video.id)
with open("video.mp4", "wb") as file:
file.write(client.videos.download_content(video.id).read())
client.videos.delete(video.id)
```
**Example request (curl):**
```bash
curl -X POST "https://your-host/v1/videos" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "luma.ray-v2:0",
"prompt": "Closeup of a seashell on a sandy beach, gentle waves",
"seconds": "5",
"size": "1280x720"
}'
```
**Example response:**
```json
{
"id": "video_eyJhcm4iOiJhcm46YXdzOmJlZHJvY2s6...",
"object": "video",
"model": "luma.ray-v2:0",
"status": "queued",
"progress": 0,
"created_at": 1783805314,
"seconds": "5",
"size": "1280x720",
"prompt": "Closeup of a seashell on a sandy beach, gentle waves"
}
```
## Feature Compatibility
| Feature | Status | Notes |
|-----------------------------|:----------------------------------------:|------------------------------------------------------------------------|
| **Creation** | | |
| `prompt` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Full support |
| `model` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Required — this gateway has no implicit default video model |
| `seconds` / `size` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Supported values depend on the model (see table below) |
| `input_reference` | :material-check-circle:{ .success role="img" aria-label="Supported" } | First-frame image; also accepts URLs/S3/Files API IDs in JSON requests |
| Extra model-specific params | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Extra fields are forwarded to the model (e.g. `seed`, `loop`); a string value that parses as a JSON number, boolean, or null is forwarded as that type |
| **Lifecycle** | | |
| Retrieve / poll job | :material-check-circle:{ .success role="img" aria-label="Supported" } | `progress` is `0` while running and `100` when completed |
| Download content (`video`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Streamed MP4 |
| Delete video | :material-check-circle:{ .success role="img" aria-label="Supported" } | Removes the stored output from S3 |
| `variant=thumbnail/spritesheet` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Amazon Bedrock generates only the video asset |
| List videos (`GET /v1/videos`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Merged across regions; listed while AWS retains the job record |
| Remix video | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Not available on Amazon Bedrock |
| Edits / extensions / characters | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | `/v1/videos/edits`, `/videos/extensions`, and `/videos/characters` are not available on Amazon Bedrock |
| `expires_at` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Reported only when a [retention period](operations_configuration.md#aws-s3-videos-expires-after) is configured |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Available on Select Models** — Check your model's capabilities
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
!!! note "Listing and Retrieval Behaviour"
The `prompt` is echoed only in the creation response, not when retrieving or listing. A deleted video stays visible in listings until AWS expires its job record. Listing merges jobs across every configured region; a region that is temporarily unavailable is omitted rather than failing the request.
## Model Support
Any video generation model available in your configured Amazon Bedrock regions can be used, for example:
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Models
| Model | Model ID | Durations (`seconds`) | Sizes (`size`) |
|----------------------|-------------------------|------------------------------------|----------------|
| Amazon Nova Reel 1.1 | `amazon.nova-reel-v1:1` | `6`, or multiples of 6 up to `120` | `1280x720` |
| Amazon Nova Reel 1.0 | `amazon.nova-reel-v1:0` | `6` | `1280x720` |
!!! note "Amazon Nova Reel Is Marked Legacy by AWS"
AWS flags the Nova Reel models as legacy, so they are hidden by default. Set [`AWS_BEDROCK_LEGACY=true`](operations_configuration.md#bedrock-legacy) to expose and use them.
### { style="height: 1.2em; vertical-align: text-bottom;" } Luma AI Models
| Model | Model ID | Durations (`seconds`) | Sizes (`size`) |
|------------|-----------------|-----------------------|-----------------------------------------------------------------------------------------------------|
| Luma Ray 2 | `luma.ray-v2:0` | `5` or `9` | 540p/720p in ratios 1:1, 16:9, 9:16, 4:3, 3:4, 21:9, 9:21 (e.g. `1280x720`, `720x1280`, `960x540`) |
When `seconds` or `size` is omitted, the model's shortest duration and default resolution are used. For Luma Ray, the requested size selects the model's resolution (smaller dimension: 540 or 720) and aspect ratio (reduced width:height).
**Find compatible models:** Call [`/search_models`](api_search_models.md) with `route=openai_video_generation` to discover model IDs that support video generation in your deployment.
## Reference Images (Image-to-Video)
The optional `input_reference` image is used as the video's first frame:
- **`multipart/form-data`** — upload the image file directly (this is what the OpenAI SDKs send). The SDK object form (`input_reference={"image_url": ...}` or `{"file_id": ...}`) is also accepted.
- **`application/json`** — pass a base64 string, data URI, HTTPS URL, S3 URI, or [Files API](api_openai_files.md) ID.
Amazon Nova Reel requires a PNG or JPEG matching the video resolution (1280x720) and only supports reference images for 6-second videos.
## How It Works
1. `POST /v1/videos` starts an [Amazon Bedrock asynchronous invocation](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_StartAsyncInvoke.html) in a region where the model is available and a regional S3 bucket is configured ([`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets)).
2. Amazon Bedrock renders the video and writes the MP4 to that bucket under [`AWS_S3_VIDEOS_PREFIX`](operations_configuration.md#aws-s3-videos-prefix) (`videos/` by default).
3. `GET /v1/videos/{video_id}` reads the job state directly from Amazon Bedrock; `.../content` streams the MP4 from S3; `DELETE` removes the stored objects.
!!! warning "S3 Bucket Requirement"
Amazon Bedrock requires the output bucket to be **in the same region as the invocation**. Configure a bucket for each region offering video models via `AWS_S3_REGIONAL_BUCKETS`, otherwise video generation requests fail with a configuration error.
## Retention
By default videos persist until deleted through the API. Set [`AWS_S3_VIDEOS_EXPIRES_AFTER`](operations_configuration.md#aws-s3-videos-expires-after) to enforce a retention period: the `Video` object then reports `expires_at` (completion time plus the retention period) and downloading expired content returns a 404, matching the OpenAI API's automatic video expiry.
## Billing
AWS bills video generation per **second of generated video**. Billed seconds appear in [usage logs and cost tracking](operations_logging_monitoring.md) as `output_seconds`, recorded when the job is started. Standard S3 storage costs apply to the stored videos until they are deleted.
---
**Ready to bring your ideas to motion?** Explore available video models in the [Models API](api_openai_models.md).
---
## Audio Speech
Source: https://stdapi.ai/md/api_openai_audio_speech.md
# Text to Speech API
Generate natural-sounding speech from text with Amazon Polly through an OpenAI-compatible interface.
## Why Choose the Text to Speech API?
- :material-earth: __Global Support__
30+ languages supported. Choose from Neural, Generative, and Long-Form engines.
- :material-account-voice: __60+ Voices__
Professional narration to conversational voices. Use OpenAI voice names with automatic language detection or specify any Polly voice ID directly.
- :material-auto-fix: __Automatic Language Detection__
Using OpenAI voice names? Amazon Comprehend automatically detects your content's language and selects an appropriate Polly voice—matching language, gender, and quality.
- :material-xml: __Advanced Control with SSML__
Fine-tune pronunciation, emphasis, pauses, and prosody with SSML markup for complex audio requirements.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|---------------------|--------|----------------------------------------|----------------------------------|--------------------|
| `/v1/audio/speech` | `POST` | Turn text into natural-sounding speech | Amazon Polly + Amazon Comprehend | `openai_audio_speech` |
## Feature Compatibility
| Feature | Status | Notes |
|-----------------------------|:----------------------------------------:|-----------------------------------------------------------------|
| **Voice Selection** | | |
| OpenAI voice names | :material-check-circle:{ .success role="img" aria-label="Supported" } | Mapped to Polly voices |
| Polly voice IDs | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | 60+ voices across 30+ languages |
| Dynamic voice selection | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Select best Polly voice based on the detected language |
| **Input** | | |
| Plain text | :material-check-circle:{ .success role="img" aria-label="Supported" } | Standard text input |
| SSML markup | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Fine-grained speech control |
| Long input | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Up to 100,000 characters, 24× OpenAI's limit — [20,000 with a generative voice, 100,000 with a bucket](#long-input) |
| **Output Formats** | | |
| MP3 | :material-check-circle:{ .success role="img" aria-label="Supported" } | Native Polly format |
| PCM | :material-check-circle:{ .success role="img" aria-label="Supported" } | 24 kHz per OpenAI's contract; resampled from Polly's native rate |
| Opus | :material-check-circle:{ .success role="img" aria-label="Supported" } | Native Polly format |
| AAC | :material-check-circle:{ .success role="img" aria-label="Supported" } | Encoded from PCM |
| FLAC | :material-check-circle:{ .success role="img" aria-label="Supported" } | Encoded from PCM |
| WAV | :material-check-circle:{ .success role="img" aria-label="Supported" } | Encoded from PCM |
| OGG (Vorbis) | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Native Polly format |
| **Control** | | |
| `speed` parameter | :material-check-circle:{ .success role="img" aria-label="Supported" } | 0.2x to 2.0x playback speed; rejected with SSML input (set the speed in SSML instead) |
| `instructions` parameter | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted for OpenAI API compatibility and ignored (no Amazon Polly equivalent) |
| Extra model-specific params | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Extra model-specific parameters via JSON body |
| **Streaming** | | |
| Byte streaming | :material-check-circle:{ .success role="img" aria-label="Supported" } | Default streaming mode |
| SSE streaming | :material-check-circle:{ .success role="img" aria-label="Supported" } | Event-based streaming |
| **Usage tracking** | | |
| Input text tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Characters count (billing unit) |
| Output tokens | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Not available |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
## Model Support
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Polly Models
| Model | Polly Engine | Notes |
|---------------------------|--------------|----------------------------------------------------|
| `amazon.polly-standard` | Standard | Lowest cost, widest language coverage |
| `amazon.polly-neural` | Neural | Higher-quality, natural-sounding voices |
| `amazon.polly-long-form` | Long-form | Expressive voices for narration-length content |
| `amazon.polly-generative` | Generative | Most human-like, conversational voices; speaks [long input](#long-input) with no bucket |
Each engine supports a different subset of voices and languages — see the [Polly voice list](https://docs.aws.amazon.com/polly/latest/dg/voicelist.html) for details. OpenAI voice names work with every model through automatic language detection and voice selection, or specify any Polly voice ID directly for 60+ voices across 30+ languages.
!!! tip "OpenAI Model Compatibility"
stdapi.ai includes built-in model aliases that map OpenAI model names to Amazon Polly engines:
- `tts-1` → `amazon.polly-standard`
- `tts-1-hd` → `amazon.polly-neural`
These aliases enable seamless compatibility with OpenAI-based tools and applications without any configuration changes. You can also [customize or override these aliases](operations_configuration.md#model-aliases) to suit your needs.
## Advanced Features
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Polly Features
- **SSML Support** :material-star-circle:{ .highlight }: Fine-grained control over pronunciation, emphasis, pauses, and prosody — [SSML docs](https://docs.aws.amazon.com/polly/latest/dg/ssml.html). With SSML input, the `speed` parameter is rejected: set the speaking rate with SSML `` instead.
- **Flexible Formats**: mp3, ogg, wav, flac, aac, opus, pcm
- **Streaming Options**: Raw bytes (default) or SSE events with `stream_format: "sse"` — once an SSE stream is accepted, a synthesis failure at any point is reported as a terminal `error` event inside the `200` response, and `speech.audio.done` is then omitted
- **Speed Control**: Adjust playback from 0.2x to 2.0x
- **Speech Marks**: Word, sentence, viseme, and SSML timing metadata with `SpeechMarkTypes` (returned as JSON instead of audio)
- **Character-Based Billing**: Usage tracks character counts—the native billing unit for Amazon Polly and Amazon Comprehend—rather than OpenAI-style tokens
!!! tip "Performance Tips: Optimize Speed & Cost"
- **Prefer mp3 or ogg** for the lowest latency; `pcm` is returned at OpenAI's 24 kHz contract unless you request an explicit `SampleRate` (see [Sample Rate](#provider-specific-parameters))
- **Specify a Polly voice ID** to bypass language detection—faster responses, no Amazon Comprehend charges
- **Configure a default language** via `DEFAULT_TTS_LANGUAGE` environment variable to skip language detection for all requests using OpenAI voice names
!!! info "Language Detection Behavior"
When using OpenAI voice names without specifying a default language, the system analyzes only the first 500 characters of your text to detect the language. This approach:
- **Works best** with long, single-language texts where the first 500 characters are representative
- **May be inconsistent** with very short texts (< 100 characters) where language detection has limited context
- **Can produce mixed results** with multi-language content where different parts use different languages
**For consistent behavior across requests**, consider:
- Setting `DEFAULT_TTS_LANGUAGE` for applications serving primarily one language
- Using Polly voice IDs directly when you know the target language
- Structuring multi-language applications to make separate API calls per language
!!! tip "Default Streaming Mode: API vs MCP"
- **API usage**: Default is **byte streaming** (raw audio data)
- **MCP tool usage**: Default is **SSE streaming** (`stream_format: "sse"`)
When used as an MCP tool, the response defaults to SSE events (`speech.audio.delta`, `speech.audio.done`) for better client compatibility. Override by explicitly setting `stream_format: "audio"` in your request.
### Long Input
A single request accepts up to **3,000 characters** (6,000 including SSML markup, which is not billed). Beyond that, how the audio is produced depends on the voice.
**Generative voices** (`amazon.polly-generative`) speak up to **20,000 characters** without any further configuration, and the audio starts arriving while the rest is still being spoken. Two cases keep the behaviour described below instead: an `input` written as an SSML document, and a request using `SpeechMarkTypes`.
**Every other voice**, and generative input longer than 20,000 characters, is synthesized into an S3 bucket co-located with the serving region, so longer input requires a bucket in a region that can serve the request: a [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets) entry for that region, or [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) when the region is the first [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) entry — the only region `AWS_S3_BUCKET` covers. With [`AWS_POLLY_REGION`](operations_configuration.md#aws-polly-region) pinned to any other region, a regional-bucket entry is the only option. The audio object is written under [`AWS_S3_TMP_PREFIX`](operations_configuration.md#aws-s3-tmp-prefix) and deleted once the request ends. On a request that ends before Amazon Polly has finished — a timeout, a failure, or a client that disconnected — the deletion is issued while the synthesis is still running, so an object written after it is **not** removed by the request: the recommended lifecycle rule on that prefix is what expires it, and it must be in place.
- **With a bucket configured**: up to **100,000 characters** (200,000 including SSML markup), against OpenAI's 4,096-character limit. Expect roughly one extra second per 1,000 characters, bounded by [`AI_RESPONSE_TIMEOUT`](operations_configuration.md#ai-response-timeout).
- **Without one**: a generative voice still speaks up to 20,000 characters; every other request above 3,000 characters is rejected, however long, with the length the server does accept — so callers can split their text.
!!! tip "Same response either way"
Nothing else changes: the response is the complete audio file in the requested `response_format`, and `stream_format: "sse"` still delivers `speech.audio.delta` events.
!!! warning "A configured guardrail caps the input first"
When a [guardrail](operations_configuration.md#aws-bedrock-guardrail-identifier) applies to the request, the whole `input` is checked in a single [Amazon Bedrock `ApplyGuardrail`](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ApplyGuardrail.html) call, which has a maximum input size of its own: a [Service Quotas](https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html) value per guardrail policy, counted in text units of 1,000 characters, that **differs between AWS Regions** — as low as 25 text units (25,000 characters) in some, 1,000 in others.
The reachable input length is therefore the smaller of the two: the limit above, and the quota in the guardrail's Region. Beyond the quota the request fails with `429` before any audio is synthesized, so raise the *maximum input size* quotas for the policies your guardrail applies, or keep requests under them.
!!! warning "Long input is billed on acceptance"
Amazon Polly bills the whole input as soon as it accepts a long request, before the audio is produced. A request that then reaches [`AI_RESPONSE_TIMEOUT`](operations_configuration.md#ai-response-timeout), fails, or is abandoned by the client is charged in full and still counted in the [request usage and cost](operations_cost_management.md#cost-tracking-real-time-aws-pricing) records — retrying it pays for the text twice.
### Provider-Specific Parameters
Unlock advanced Amazon Polly capabilities by passing provider-specific parameters directly in your requests. These parameters are forwarded to Polly's `SynthesizeSpeech` API and allow you to access features unique to Polly.
**How It Works:**
Add provider-specific fields at the top level of your request body alongside standard OpenAI parameters. The API automatically forwards these to Amazon Polly.
**Examples:**
**Lexicon Support:**
Apply custom pronunciation lexicons to your speech synthesis:
```json
{
"model": "amazon.polly-neural",
"voice": "Joanna",
"input": "Amazon Polly uses lexicons for custom pronunciation.",
"response_format": "mp3",
"LexiconNames": ["MyCustomLexicon"]
}
```
**Sample Rate:**
Specify custom audio sample rate (8000, 16000, 22050, or 24000 Hz; PCM output supports 8000 and 16000 only):
```json
{
"model": "amazon.polly-neural",
"voice": "Matthew",
"input": "High quality audio at 24kHz.",
"response_format": "mp3",
"SampleRate": "24000"
}
```
!!! info "PCM output defaults to OpenAI's 24 kHz contract"
Per OpenAI's TTS API, `response_format: "pcm"` is raw, headerless 24 kHz 16-bit mono little-endian audio. Without an explicit `SampleRate`, `pcm` output is synthesized by Polly at 16 kHz and resampled to 24 kHz server-side. Pass a Polly-native `SampleRate` (`8000` or `16000`) to skip resampling and receive Polly's raw rate instead — since raw PCM carries no embedded rate, only do this when your client knows to play it back at that rate.
**Language Code:**
Specify the language for bilingual voices (only useful for voices that support multiple languages):
```json
{
"model": "amazon.polly-neural",
"voice": "Aditi",
"input": "Hello, how are you?",
"response_format": "mp3",
"LanguageCode": "en-IN"
}
```
**Speech Marks:**
Request word, sentence, viseme, or SSML timing marks instead of audio (useful for lip-sync, karaoke-style highlighting, or subtitle alignment):
```json
{
"model": "amazon.polly-neural",
"voice": "Joanna",
"input": "Hello, how are you?",
"SpeechMarkTypes": ["word", "sentence"]
}
```
!!! warning "Speech marks return JSON, not audio"
When `SpeechMarkTypes` is set, Polly returns timing metadata only. The response is a stream of JSON objects (one per line) with the `application/x-json-stream` content type:
- `response_format` is ignored — no audio is returned.
- `stream_format: "sse"` is rejected with HTTP 400, since the payload is not audio events.
- The `ssml` mark type requires SSML input (`…`); requesting it with plain text returns HTTP 400.
```json
{"time":0,"type":"word","start":0,"end":5,"value":"Hello"}
{"time":576,"type":"word","start":7,"end":10,"value":"how"}
```
**Configuration Options:**
**Option 1: Per-Request**
Add provider-specific parameters directly in your request body (as shown in examples above).
**Option 2: Server-Wide Defaults**
Configure default parameters for specific models via the `DEFAULT_MODEL_PARAMS` environment variable:
```bash
export DEFAULT_MODEL_PARAMS='{
"amazon.polly-neural": {
"SampleRate": "24000"
}
}'
```
**Note:** Per-request parameters override server-wide defaults.
**Behavior:**
**Compatible parameters** are forwarded to Polly and applied; **unsupported parameters** return HTTP 400 with an error message.
**Available Parameters:**
The following parameters from the Amazon Polly [SynthesizeSpeech API](https://docs.aws.amazon.com/polly/latest/dg/API_SynthesizeSpeech.html) can be used:
- `LexiconNames` (list): Apply pronunciation lexicons
- `SampleRate` (string): Audio sample rate in Hz — `8000`, `16000`, `22050`, or `24000` (`pcm` output: `8000` or `16000`; omit it to get OpenAI's 24 kHz `pcm` contract instead)
- `LanguageCode` (string): Language code for bilingual voices only (e.g., `en-IN`, `hi-IN`)
- `SpeechMarkTypes` (list): Timing marks to return instead of audio — `sentence`, `ssml`, `viseme`, `word`
## Available Request Headers
This endpoint supports standard Bedrock headers for enhanced control over your requests. All headers are optional and can be combined as needed.
### Content Safety (Guardrails)
| Header | Purpose | Valid Values |
|--------------------------------------|------------------------------------|----------------------------|
| `X-Amzn-Bedrock-GuardrailIdentifier` | Guardrail ID for content filtering | Your guardrail identifier |
| `X-Amzn-Bedrock-GuardrailVersion` | Guardrail version | Version number (e.g., `1`) |
The guardrail evaluates the text to synthesize; the audio produced from it is not itself evaluated. `X-Amzn-Bedrock-Trace` is accepted but has no effect on this route — no guardrail trace is returned.
**Example with headers:**
```bash
curl -X POST "$BASE/v1/audio/speech" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Amzn-Bedrock-GuardrailIdentifier: your-guardrail-id" \
-H "X-Amzn-Bedrock-GuardrailVersion: 1" \
-d '{
"model": "amazon.polly-neural",
"voice": "Amy",
"input": "Welcome to the future of voice technology!"
}' \
--output speech.mp3
```
!!! note "No performance headers on this route"
`X-Amzn-Bedrock-Service-Tier` and `X-Amzn-Bedrock-PerformanceConfig-Latency` have no effect here: speech is synthesized by Amazon Polly, which is not invoked through the Amazon Bedrock runtime.
!!! info "Detailed Documentation"
For complete information about these headers, configuration options, and use cases, see:
- [Bedrock Guardrails Configuration](operations_configuration.md#bedrock-guardrails)
## Try It Now
**Stream audio as bytes (default):**
```bash
curl -OJ -X POST "$BASE/v1/audio/speech" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.polly-neural",
"voice": "Amy",
"input": "Welcome to the future of voice technology!",
"response_format": "mp3"
}'
```
**Stream audio as SSE events:**
```bash
curl -N -X POST "$BASE/v1/audio/speech" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.polly-neural",
"voice": "Amy",
"input": "This audio streams as SSE events!",
"response_format": "mp3",
"stream_format": "sse"
}'
```
---
**Ready to add voice to your application?** Explore available voices and models in the [Models API](api_openai_models.md).
---
## Audio Transcriptions
Source: https://stdapi.ai/md/api_openai_audio_transcriptions.md
# Speech to Text API
Transcribe audio to text with Amazon Transcribe or Amazon Bedrock audio-capable models through an OpenAI-compatible interface.
## Why Choose the Speech to Text API?
- :material-translate: __Multiple Transcription Options__
Choose Amazon Transcribe for 100+ languages with speaker diarization, or use Bedrock audio models for advanced capabilities.
- :material-clock-fast: __Real-Time or Batch__
Stream transcriptions in real-time via SSE or process files efficiently with either service.
- :material-subtitles: __Subtitle Generation__
Generate SRT and VTT subtitle files directly with precise timing for video content.
- :material-account-multiple: __Advanced Features__
Speaker diarization, word-level timestamps, and automatic language detection. Feature availability varies by model choice.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|-----------------------------|--------|------------------------------------------|--------------------------------------------------|---------------------------|
| `/v1/audio/transcriptions` | `POST` | Convert spoken audio to written text | Amazon Transcribe or Amazon Bedrock Audio Models | `openai_audio_transcription` |
## Feature Compatibility
| Feature | Status | Notes |
|----------------------------|:----------------------------------------:|------------------------------------------------------------------|
| **Input** | | |
| Audio file upload | :material-check-circle:{ .success role="img" aria-label="Supported" } | Multipart file upload |
| JSON body input | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Base64, data URI, HTTPS URL, S3 URI, or `file-id:` reference — for MCP / AI agents |
| **Output Formats** | | |
| `json` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Structured transcription |
| `text` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Plain text output |
| `verbose_json` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | With timestamps and details (Amazon Transcribe; not Bedrock models) |
| `diarized_json` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | With speaker identification (Amazon Transcribe; not Bedrock models); rejected with `stream=true` |
| `srt` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Subtitle format with timing (Amazon Transcribe; not Bedrock models); rejected with `stream=true` |
| `vtt` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | WebVTT subtitle format (Amazon Transcribe; not Bedrock models); rejected with `stream=true` |
| **Language** | | |
| Language specification | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | ISO-639-1 language codes |
| `languages` (expected languages) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Expected-language list (ISO-639-1) for multi-language audio; cannot be combined with `language`. Drives Amazon Transcribe multi-language identification; folded into the transcription context on Bedrock models |
| Auto language detection | :material-check-circle:{ .success role="img" aria-label="Supported" } | Automatic identification |
| Detected `languages` in response | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | The `json` response reports the detected language(s) as a `languages` array (Amazon Transcribe only) |
| **Streaming** | | |
| `stream` (SSE streaming) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Set `stream: true` to receive incremental results as server-sent events. Carries text only, so `srt`, `vtt` and `diarized_json` are rejected rather than answered without their cues or speaker labels; `verbose_json` is accepted but degrades to text-only events — timestamps and segments are dropped |
| **Advanced** | | |
| `timestamp_granularities` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Word or segment level; requires `response_format=verbose_json` (Amazon Transcribe only) |
| Speaker diarization | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Automatic speaker separation; requires `response_format=diarized_json` (Amazon Transcribe only) |
| `known_speaker_names` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored — diarization falls back to generic speaker labels |
| `known_speaker_references` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored — diarization falls back to generic speaker labels |
| `chunking_strategy` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Only `auto` is accepted; other values are rejected |
| `temperature` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Bedrock models only; rejected by Amazon Transcribe |
| `prompt` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Bedrock models only; rejected by Amazon Transcribe |
| `keywords` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Bedrock models only (folded into the transcription context); rejected by Amazon Transcribe — use a pre-created custom vocabulary via the `VocabularyName` extra parameter |
| `include` (`logprobs`) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Accepted on Bedrock models but never populated (`logprobs` is always `null`); rejected by Amazon Transcribe |
| Extra model-specific params | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Amazon Transcribe optional settings via JSON body (see below) |
| **Usage tracking** | | |
| Input audio duration | :material-check-circle:{ .success role="img" aria-label="Supported" } | Seconds (billing unit on Amazon Transcribe) |
| Output text tokens | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | On models from Bedrock |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Available on Select Models** — Check your model's capabilities
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
## Model Support
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Models
| Model | Supported Languages | Notes |
|-------------------|---------------------|------------------------------------------------------------------------------------------------------------|
| amazon.transcribe | 100+ | Full-featured transcription with speaker diarization and subtitle generation at the cost of higher latency |
!!! warning "Configuration Required"
You must configure a bucket to use this model, through `AWS_S3_BUCKET`, `AWS_TRANSCRIBE_S3_BUCKET`, or an `AWS_S3_REGIONAL_BUCKETS` entry for a region where Amazon Transcribe is a candidate. This bucket is used for temporary storage during transcription processing.
### { style="height: 1.2em; vertical-align: text-bottom;" } Mistral Models
| Model | Supported Languages | Notes |
|---------------------------------|---------------------------|----------------------------------------------------|
| mistral.voxtral-mini-3b-2507 | Multilingual (auto-detected) | Compact model for fast transcription |
| mistral.voxtral-small-24b-2507 | Multilingual (auto-detected) | Larger model for enhanced accuracy |
!!! warning "Mistral Voxtral Limitations"
Mistral Voxtral models have the following restrictions when running on Amazon Bedrock:
- **File size limit**: ~2MB maximum input file size
- **Audio channels**: Mono channel audio only (single channel)
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Nova Sonic
| Model | Supported Languages | Notes |
|---------------------------|------------------------------|---------------------------------------------|
| amazon.nova-2-sonic-v1:0 | Multilingual (auto-detected) | Low-cost real-time speech recognition |
Name this model to transcribe through Amazon Nova Sonic instead of Amazon Transcribe. It is the cheapest transcription option (about $0.006 per minute of audio at current Amazon Bedrock rates) and returns punctuated text in the language that was spoken. Transcription is model selection, never automatic: requests that do not name this model are unaffected.
!!! warning "What this model does not provide"
- **Response formats**: `json` and `text` only. `srt`, `vtt`, `verbose_json` and `diarized_json` are rejected, as is `timestamp_granularities` — this model returns no timestamps and does not report which language it detected. Use `amazon.transcribe` for subtitles, timestamps, speaker diarization or detected-language reporting.
- **Audio length**: up to 10 minutes per request. Longer recordings are rejected; use `amazon.transcribe`, which has no such limit.
### Other Amazon Bedrock Models
Any Amazon Bedrock model that accepts the `SPEECH` input modality through the Converse API can transcribe out of the box: the gateway sends the audio together with a transcription prompt and returns the model's text output.
!!! tip "Audio Input Formats on Bedrock Models"
Uploads in the formats the Bedrock Converse audio block accepts — `aac`, `flac`, `m4a`, `mka`, `mkv`, `mp3`, `mp4`, `mpeg`, `mpga`, `ogg`, `opus`, `pcm`, `wav`, `webm`, and `x-aac` — are sent through as-is. Any other audio or video upload is automatically converted to FLAC before transcription (requires FFmpeg on the server), including the audio track of a video container. An upload that is neither audio nor video is rejected with the list of accepted formats; an audio or video file whose track cannot be decoded is rejected as carrying no decodable audio.
## Advanced Features
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Transcribe Features
**Model & Features:**
- Use `amazon.transcribe` with the same interface as OpenAI's Whisper API
- **Or use OpenAI model names directly**: `whisper-1`, `gpt-transcribe`, `gpt-live-transcribe`, `gpt-4o-transcribe`, and `gpt-4o-mini-transcribe` work out of the box (they map to `amazon.transcribe`)
- Auto-detect language or specify it for faster processing, or list the expected languages with `languages` for multi-language audio
- Word-level or segment-level timestamps with `verbose_json`
- **Speaker Diarization** :material-account-multiple:{ .highlight }: Automatically identify and label different speakers with `diarized_json`
- **Native Subtitles** :material-file-video:{ .highlight }: SRT/VTT files generated directly by Amazon Transcribe with precise timing
!!! tip "OpenAI Model Compatibility"
stdapi.ai includes built-in model aliases that map the OpenAI model names to Amazon Transcribe:
- `whisper-1` → `amazon.transcribe`
- `gpt-transcribe` → `amazon.transcribe`
- `gpt-live-transcribe` → `amazon.transcribe`
- `gpt-4o-transcribe` → `amazon.transcribe`
- `gpt-4o-mini-transcribe` → `amazon.transcribe`
These aliases enable seamless compatibility with OpenAI-based tools and applications without any configuration changes. You can also [customize or override these aliases](operations_configuration.md#model-aliases) to suit your needs.
**Note:** With `amazon.transcribe`, the `prompt`, `temperature`, `keywords`, and `include` parameters are rejected with an error to ensure consistent transcription accuracy (for `keywords`, the error points at the pre-created custom vocabulary alternative via the `VocabularyName` extra parameter). The `known_speaker_names` and `known_speaker_references` parameters are accepted but ignored for every model: Amazon Transcribe's automatic speaker diarization runs without known speaker references, falling back to generic speaker labels.
!!! tip "Performance Tips: Optimize Speed & Cost"
- **Specify the language** if you know it—skips auto-detection for faster processing and lower AWS costs
### Provider-Specific Parameters
Unlock advanced Amazon Transcribe capabilities by passing provider-specific parameters directly in your request body. These parameters are forwarded to Transcribe's `StartTranscriptionJob` API.
!!! warning "JSON body required"
Unlike the `multipart/form-data` upload, extra parameters are only reachable through the `application/json` request body (`file` as base64, data URI, HTTPS URL, or `file-id:` reference) — the multipart path only accepts the documented OpenAI fields.
**PII Redaction:**
Redact personally identifiable information from the transcript (only the single-output `redacted` mode is supported):
```json
{
"model": "amazon.transcribe",
"file": "data:audio/mp3;base64,",
"ContentRedaction": {
"RedactionType": "PII",
"PiiEntityTypes": ["NAME", "SSN", "CREDIT_DEBIT_NUMBER"]
}
}
```
**Custom Vocabulary and Filtering:**
Improve recognition of domain-specific terms and mask or remove profanity/sensitive words:
```json
{
"model": "amazon.transcribe",
"file": "data:audio/mp3;base64,",
"VocabularyName": "MyCustomVocabulary",
"VocabularyFilterName": "MyProfanityFilter",
"VocabularyFilterMethod": "mask"
}
```
**Alternative Transcriptions and Channel Identification:**
Request multiple candidate transcriptions per segment, or transcribe each audio channel separately (e.g. two-party phone calls recorded in stereo):
```json
{
"model": "amazon.transcribe",
"file": "data:audio/mp3;base64,",
"ShowAlternatives": true,
"MaxAlternatives": 3,
"ChannelIdentification": true
}
```
!!! warning "Incompatible with diarized_json"
`ChannelIdentification` cannot be combined with `response_format=diarized_json`, which already forces AWS speaker-label diarization. Requesting both returns HTTP 400.
**Toxicity Detection:**
Flag toxic content (profanity, hate speech, harassment) in the transcript:
```json
{
"model": "amazon.transcribe",
"file": "data:audio/mp3;base64,",
"ToxicityDetection": [{"ToxicityCategories": ["ALL"]}]
}
```
**Multi-Language Identification:**
Detect and transcribe multiple languages spoken in the same audio, optionally restricted to a candidate list:
```json
{
"model": "amazon.transcribe",
"file": "data:audio/mp3;base64,",
"IdentifyMultipleLanguages": true,
"LanguageOptions": ["en-US", "es-US", "fr-FR"]
}
```
**Per-Language Custom Resources:**
A custom vocabulary, vocabulary filter or custom language model can only be attached per candidate language when the language is identified rather than given. Pair `LanguageIdSettings` with `LanguageOptions` so the dialect your resources were created for is the one identified:
```json
{
"model": "amazon.transcribe",
"file": "data:audio/mp3;base64,",
"LanguageOptions": ["en-US", "es-US"],
"LanguageIdSettings": {
"en-US": {"VocabularyName": "MedicalTermsEnUs"},
"es-US": {"VocabularyName": "MedicalTermsEsUs"}
}
}
```
!!! warning "Identification required"
`LanguageIdSettings` applies to identified languages only. Combined with a fixed `language` (or a single-entry `languages`), the request returns HTTP 400 rather than silently dropping the custom resources — use the flat `VocabularyName`, `VocabularyFilterName` and `ModelSettings` parameters in that case. `LanguageModelName` is not available with `IdentifyMultipleLanguages`.
!!! tip "Standard `languages` parameter"
The standard OpenAI `languages` parameter drives the same multi-language identification with plain ISO-639-1 codes (e.g. `["en", "es", "fr"]`), works on the multipart path too, and the detected language(s) come back in the `json` response's `languages` array. A single-entry list behaves like `language`. Do not combine it with `language` or with the provider-specific parameters above.
**Configuration Options:**
**Option 1: Per-Request**
Add provider-specific parameters directly in your JSON request body (as shown in examples above).
**Option 2: Server-Wide Defaults**
Configure default parameters for `amazon.transcribe` via the `DEFAULT_MODEL_PARAMS` environment variable:
```bash
export DEFAULT_MODEL_PARAMS='{
"amazon.transcribe": {
"VocabularyFilterName": "MyProfanityFilter",
"VocabularyFilterMethod": "mask"
}
}'
```
**Note:** Per-request parameters override server-wide defaults.
**Behavior:**
**Compatible parameters** are forwarded to Amazon Transcribe and applied; **unsupported parameters or values** return HTTP 400 with an error message.
**Available Parameters:**
The following parameters from Amazon Transcribe's [StartTranscriptionJob API](https://docs.aws.amazon.com/transcribe/latest/APIReference/API_StartTranscriptionJob.html) can be used:
- `ContentRedaction` (object): PII redaction — `RedactionType` (`PII`), `PiiEntityTypes` (list), `RedactionOutput` (`redacted` only; `redacted_and_unredacted` is rejected — the unredacted copy is not tracked for automatic cleanup)
- `VocabularyName` (string): Custom vocabulary to improve recognition accuracy
- `VocabularyFilterName` / `VocabularyFilterMethod` (string / `mask`, `remove`, `tag`): Profanity or sensitive-word filtering
- `ShowAlternatives` / `MaxAlternatives` (bool / integer `2`-`10`): Return multiple candidate transcriptions per segment
- `ChannelIdentification` (bool): Transcribe each audio channel separately (incompatible with `diarized_json`)
- `MaxSpeakerLabels` (integer `2`-`30`): Maximum speakers to identify with `response_format=diarized_json` (default `10`)
- `ShowSpeakerLabels` (bool): Always on with `response_format=diarized_json`; setting it directly with another format runs AWS speaker labeling without exposing speaker data in the response
- `ToxicityDetection` (list): Toxic-content flagging — `[{"ToxicityCategories": ["ALL"]}]`
- `IdentifyMultipleLanguages` / `LanguageOptions` (bool / list): Multi-language identification, optionally restricted to a candidate list (supersedes `language`; cannot be combined with the standard `languages` parameter)
- `LanguageIdSettings` (object): Per-language `VocabularyName`, `VocabularyFilterName` and `LanguageModelName`, keyed by language code (up to five) — the only way to attach them when the language is identified rather than given
- `ModelSettings` (object): `LanguageModelName` — custom language model selection
`VocabularyName`, `VocabularyFilterName`, and custom language models must already exist in your AWS account (created via the AWS Transcribe console, CLI, or SDK) before being referenced here.
## :material-lightning-bolt: Streaming { #streaming }
`stream=true` returns the transcript as server-sent events — `transcript.text.delta` events followed by a final `transcript.text.done` — instead of one response body. A [ready-to-run example](#try-it-now) is below.
Each phrase is sent as it is recognized, rather than after the whole recording, whenever the request names the language to expect: send `language`, or two or more expected `languages`. That path needs no S3 bucket, so a deployment with no storage configured serves streamed transcriptions.
A request naming neither is still streamed, but its events arrive together once the recording has been read and its language detected. Operators can set [`AWS_TRANSCRIBE_STREAM_LANGUAGES`](operations_configuration.md#aws-transcribe-stream-languages) to the languages their callers actually send, which gives those requests the faster path too. The same applies to a request using any provider-specific parameter above other than `VocabularyName`, `VocabularyFilterName` and `VocabularyFilterMethod`, which are the only ones a phrase-by-phrase transcript can carry.
Streamed events carry text only, which is why `srt`, `vtt` and `diarized_json` are rejected with `stream=true` rather than answered without their cues or speaker labels.
## Available Request Headers
This endpoint supports standard Bedrock headers for enhanced control over your requests. All headers are optional and can be combined as needed.
### Content Safety (Guardrails)
| Header | Purpose | Valid Values |
|--------------------------------------|------------------------------------|----------------------------|
| `X-Amzn-Bedrock-GuardrailIdentifier` | Guardrail ID for content filtering | Your guardrail identifier |
| `X-Amzn-Bedrock-GuardrailVersion` | Guardrail version | Version number (e.g., `1`) |
The guardrail evaluates the transcript the model produced, not the audio sent. On a streamed request the events are withheld until the transcript is complete, so the guardrail sees the whole text before any of it is delivered. `X-Amzn-Bedrock-Trace` is accepted but has no effect on this route — no guardrail trace is returned.
### Performance Optimization
| Header | Purpose | Valid Values |
|--------------------------------------------|------------------------|-------------------------------------------|
| `X-Amzn-Bedrock-Service-Tier` | Service tier selection | `default`, `flex`, `priority`, `reserved` |
| `X-Amzn-Bedrock-PerformanceConfig-Latency` | Latency optimization | `standard`, `optimized` |
Both apply only to models transcribed through the Amazon Bedrock runtime. They have no effect on `amazon.transcribe`, which is served by Amazon Transcribe, nor on Amazon Nova Sonic, which is served over its own bidirectional stream.
**Example with headers:**
```bash
curl -X POST "$BASE/v1/audio/transcriptions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "X-Amzn-Bedrock-GuardrailIdentifier: your-guardrail-id" \
-H "X-Amzn-Bedrock-GuardrailVersion: 1" \
-F file=@meeting-recording.mp3 \
-F model=amazon.transcribe \
-F response_format=json
```
!!! info "Detailed Documentation"
For complete information about these headers, configuration options, and use cases, see:
- [Bedrock Guardrails Configuration](operations_configuration.md#bedrock-guardrails)
- [Service Tier and Performance Configuration](operations_configuration.md#bedrock-service-tier-and-performance-configuration)
## Try It Now
**Transcribe audio to JSON:**
```bash
curl -X POST "$BASE/v1/audio/transcriptions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F file=@meeting-recording.mp3 \
-F model=amazon.transcribe \
-F response_format=json
```
**Transcribe via JSON body (MCP and AI agents):**
When using MCP tools or HTTP clients that cannot construct multipart requests, pass the audio as a data URI or URL:
```bash
# Data URI (inline base64)
curl -X POST "$BASE/v1/audio/transcriptions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"file": "data:audio/mp3;base64,",
"model": "amazon.transcribe",
"response_format": "json"
}'
```
```bash
# HTTPS URL (server fetches the audio)
curl -X POST "$BASE/v1/audio/transcriptions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"file": "https://example.com/audio.mp3",
"model": "amazon.transcribe"
}'
```
```bash
# Files API reference (file-id: URI scheme)
curl -X POST "$BASE/v1/audio/transcriptions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"file": "file-id:file-0190c51c7de7455d9b8c2efe27dfbf67",
"model": "amazon.transcribe"
}'
```
See [Files API → Referencing Uploaded Files](api_openai_files.md#referencing-uploaded-files-via-the-file-id-uri-scheme) for the full description of the `file-id:` URI scheme.
**Generate subtitles:**
```bash
curl -X POST "$BASE/v1/audio/transcriptions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F file=@video-audio.mp3 \
-F model=amazon.transcribe \
-F response_format=srt \
-F language=en
```
**Transcribe with speaker diarization:**
```bash
curl -X POST "$BASE/v1/audio/transcriptions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F file=@meeting-recording.mp3 \
-F model=amazon.transcribe \
-F response_format=diarized_json
```
**Stream a transcription as SSE events:**
Set `stream=true` to receive the transcript incrementally as server-sent events (`transcript.text.delta` events followed by a final `transcript.text.done` event):
```bash
curl -N -X POST "$BASE/v1/audio/transcriptions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F file=@meeting-recording.mp3 \
-F model=amazon.transcribe \
-F language=en \
-F stream=true
```
!!! tip "Naming the language starts the transcript sooner"
`language` is what lets each phrase be sent as it is recognized rather than after the whole recording — see [Streaming Transcriptions](#streaming).
!!! info "`verbose_json` streams as plain text"
`stream=true` combined with `response_format=verbose_json` is accepted rather than rejected, but the streamed events carry `transcript.text.delta` / `.done` only — segment timings, word timings and language details are not included. Request `verbose_json` without `stream` to get them.
---
**Ready to transcribe audio?** Explore available transcription models in the [Models API](api_openai_models.md).
---
## Audio Translations
Source: https://stdapi.ai/md/api_openai_audio_translations.md
# Speech to English API
Translate audio from any language to English text with Amazon Transcribe + Translate or Amazon Bedrock audio-capable models through an OpenAI-compatible interface.
## Why Choose the Speech to English API?
- :material-earth-arrow-right: __Automatic Language Detection__
Upload audio in any language. AWS automatically detects the source language and translates to English text.
- :material-account-network: __Multiple Translation Options__
Choose Amazon Transcribe + Translate for a traditional pipeline, or use Bedrock audio models with built-in translation capabilities.
- :material-file-multiple: __Multiple Output Formats__
Choose from text, JSON, verbose JSON with timestamps, or translated subtitle files (SRT/VTT).
- :material-subtitles: __Subtitle Translation__
Generate translated SRT and VTT subtitle files directly with precise timing for international video content.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|--------------------------|--------|--------------------------------------------------|--------------------------------------------------------------|-----------------------------|
| `/v1/audio/translations` | `POST` | Transcribe any language and translate to English | Amazon Transcribe + Translate or Amazon Bedrock Audio Models | `openai_audio_translation` |
## Feature Compatibility
| Feature | Status | Notes |
|-------------------------|:---------------------------------------:|-------------------------------|
| **Input** | | |
| Audio file upload | :material-check-circle:{ .success role="img" aria-label="Supported" } | Multipart file upload |
| JSON body input | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" }| Base64, data URI, HTTPS URL, S3 URI, or `file-id:` reference — for MCP / AI agents |
| **Output Formats** | | |
| `json` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Structured translation |
| `text` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Plain English text |
| `verbose_json` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | With timestamps (Amazon Transcribe; not Bedrock models) |
| `srt` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | English subtitles with timing (Amazon Transcribe; not Bedrock models) |
| `vtt` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | English WebVTT subtitles (Amazon Transcribe; not Bedrock models) |
| **Language** | | |
| Auto language detection | :material-check-circle:{ .success role="img" aria-label="Supported" } | Automatic source detection |
| **Translation** | | |
| Translation to English | :material-check-circle:{ .success role="img" aria-label="Supported" } | Amazon Translate (with `amazon.transcribe`) or native model translation (Bedrock models) |
| **Advanced** | | |
| `prompt` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Bedrock models only; rejected by Amazon Transcribe |
| `temperature` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Bedrock models only; rejected by Amazon Transcribe |
| Extra model-specific params | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" }| Amazon Transcribe + Translate optional settings via JSON body (see below) |
| **Usage tracking** | | |
| Input audio duration | :material-check-circle:{ .success role="img" aria-label="Supported" } | Seconds (billing unit on Amazon Transcribe) |
| Output text tokens | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | On models from Bedrock |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Available on Select Models** — Check your model's capabilities
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
## Model Support
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Models
| Model | Supported Languages | Notes |
|-------------------|---------------------|------------------------------------------------------------------------------------------------------------|
| amazon.transcribe | 100+ | Full-featured transcription with speaker diarization and subtitle generation at the cost of higher latency |
!!! warning "Configuration Required"
You must configure a bucket to use this model, through `AWS_S3_BUCKET`, `AWS_TRANSCRIBE_S3_BUCKET`, or an `AWS_S3_REGIONAL_BUCKETS` entry for a region where Amazon Transcribe is a candidate. This bucket is used for temporary storage during transcription processing.
### { style="height: 1.2em; vertical-align: text-bottom;" } Mistral Models
| Model | Supported Languages | Notes |
|---------------------------------|---------------------------|----------------------------------------------------|
| mistral.voxtral-mini-3b-2507 | Multilingual (auto-detected) | Compact model for fast transcription |
| mistral.voxtral-small-24b-2507 | Multilingual (auto-detected) | Larger model for enhanced accuracy |
!!! warning "Mistral Voxtral Limitations"
Mistral Voxtral models have the following restrictions when running on Amazon Bedrock:
- **File size limit**: ~2MB maximum input file size
- **Audio channels**: Mono channel audio only (single channel)
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Nova Sonic
| Model | Supported Languages | Notes |
|---------------------------|------------------------------|----------------------------------------------|
| amazon.nova-2-sonic-v1:0 | Multilingual (auto-detected) | Speech translated to English in one request |
Name this model to translate through Amazon Nova Sonic instead of Amazon Transcribe and Amazon Translate. The model listens to the audio and answers with the English text directly, in a single request.
!!! warning "What this model does not provide"
- **Response formats**: `json` and `text` only. `srt`, `vtt` and `verbose_json` are rejected — this model returns no timestamps and does not report which language it detected. Use `amazon.transcribe` for subtitles or timestamps.
- **Audio length**: up to 10 minutes per request. Longer recordings are rejected; use `amazon.transcribe`, which has no such limit.
- **Cost**: translation is billed for the English answer the model produces as well as for the audio it hears, so it costs more per minute than transcription with the same model.
### Other Amazon Bedrock Models
Any Amazon Bedrock model that accepts the `SPEECH` input modality through the Converse API can also translate audio out of the box: the gateway sends the audio together with a translation prompt and returns the model's English text output.
!!! tip "Audio Input Formats on Bedrock Models"
Uploads in the formats the Bedrock Converse audio block accepts — `aac`, `flac`, `m4a`, `mka`, `mkv`, `mp3`, `mp4`, `mpeg`, `mpga`, `ogg`, `opus`, `pcm`, `wav`, `webm`, and `x-aac` — are sent through as-is. Any other audio or video upload is automatically converted to FLAC before translation (requires FFmpeg on the server), including the audio track of a video container. An upload that is neither audio nor video is rejected with the list of accepted formats; an audio or video file whose track cannot be decoded is rejected as carrying no decodable audio.
## Advanced Features
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Transcribe Features
**Model & Features:**
- Use `amazon.transcribe` with the same interface as OpenAI's Whisper API
- **Or use OpenAI model names directly**: `whisper-1`, `gpt-transcribe`, `gpt-4o-transcribe`, and `gpt-4o-mini-transcribe` work out of the box (they map to `amazon.transcribe`)
- Automatic transcription + translation pipeline in one request
- Multiple output formats: `text`, `json`, `verbose_json`, `srt`, `vtt`
- Automatic source language detection (zero configuration)
- **Smart Subtitle Translation** :material-translate:{ .highlight }: Subtitle timing is preserved during translation
!!! tip "OpenAI Model Compatibility"
stdapi.ai includes built-in model aliases that map the OpenAI model names to Amazon Transcribe:
- `whisper-1` → `amazon.transcribe`
- `gpt-transcribe` → `amazon.transcribe`
- `gpt-4o-transcribe` → `amazon.transcribe`
- `gpt-4o-mini-transcribe` → `amazon.transcribe`
These aliases enable seamless compatibility with OpenAI-based tools and applications without any configuration changes (the realtime-oriented `gpt-live-transcribe` is not aliased: it belongs to a streaming API this route does not emulate). You can also [customize or override these aliases](operations_configuration.md#model-aliases) to suit your needs.
**Note:** With `amazon.transcribe`, the `prompt` and `temperature` parameters are rejected with an error to ensure consistent translation accuracy. Bedrock audio models accept both.
!!! warning "Source languages Amazon Translate does not cover"
Amazon Transcribe recognises more languages than Amazon Translate can translate into English. When the detected source language is not one of [Amazon Translate's supported languages](https://docs.aws.amazon.com/translate/latest/dg/what-is-languages.html), the request returns HTTP 400 listing the supported language codes instead of a partial result. Transcribe the audio with [`/v1/audio/transcriptions`](api_openai_audio_transcriptions.md) to keep it in its original language.
### Provider-Specific Parameters
`amazon.transcribe` first transcribes the audio, then translates it, and each step has its own provider-specific parameters — both reachable via the same `application/json` request body.
**Transcription step (Amazon Transcribe):** the same [extra parameters documented for `/v1/audio/transcriptions`](api_openai_audio_transcriptions.md#provider-specific-parameters) (`ContentRedaction`, `VocabularyName`, `VocabularyFilterName`/`VocabularyFilterMethod`, `ShowAlternatives`/`MaxAlternatives`, `ChannelIdentification`, `ToxicityDetection`, `IdentifyMultipleLanguages`/`LanguageOptions`, `LanguageIdSettings`, `ModelSettings`) apply here too.
**Translation step (Amazon Translate):** `Settings` and `TerminologyNames` control the English output register and glossary:
```json
{
"model": "amazon.transcribe",
"file": "data:audio/mp3;base64,",
"Settings": {
"Formality": "FORMAL",
"Profanity": "MASK"
},
"TerminologyNames": ["MyProductGlossary"]
}
```
- `Settings.Formality` (`FORMAL` or `INFORMAL`): Register of the translated text, for languages that support formality
- `Settings.Profanity` (`MASK`): Mask profane words and phrases in the translation
- `Settings.Brevity` (`ON`): Per AWS Translate, reduces the length of the translation output for most translations; unsupported language pairs silently ignore it
- `TerminologyNames` (list): Apply one or more custom terminologies (domain-specific glossaries) to the translation
Both settings apply consistently to the primary translated text and, for `response_format=verbose_json`, to every per-segment translation.
!!! warning "`verbose_json` translates the transcript twice"
For `response_format=verbose_json`, the full transcript is sent to AWS Translate once, and then every segment is sent again individually so segment-level translations are available. AWS Translate bills by character, so `verbose_json` costs roughly double the translation characters of `text` or `json` for the same audio.
!!! warning "Terminologies must already exist"
`TerminologyNames` references AWS Translate custom terminology resources created ahead of time via the AWS Translate console, CLI, or SDK (`ImportTerminology`) — stdapi.ai does not create or manage them. An unknown name is rejected by AWS Translate with a client error.
Invalid `Settings` values (e.g. an unsupported `Formality`) are rejected with HTTP 400 before any partial translation occurs.
## Available Request Headers
This endpoint supports standard Bedrock headers for enhanced control over your requests. All headers are optional and can be combined as needed.
### Content Safety (Guardrails)
| Header | Purpose | Valid Values |
|--------------------------------------|------------------------------------|----------------------------|
| `X-Amzn-Bedrock-GuardrailIdentifier` | Guardrail ID for content filtering | Your guardrail identifier |
| `X-Amzn-Bedrock-GuardrailVersion` | Guardrail version | Version number (e.g., `1`) |
The guardrail evaluates the English translation the model produced, not the audio sent. `X-Amzn-Bedrock-Trace` is accepted but has no effect on this route — no guardrail trace is returned.
### Performance Optimization
| Header | Purpose | Valid Values |
|--------------------------------------------|------------------------|-------------------------------------------|
| `X-Amzn-Bedrock-Service-Tier` | Service tier selection | `default`, `flex`, `priority`, `reserved` |
| `X-Amzn-Bedrock-PerformanceConfig-Latency` | Latency optimization | `standard`, `optimized` |
Both apply only to models translated through the Amazon Bedrock runtime. They have no effect on `amazon.transcribe`, which is served by Amazon Transcribe and AWS Translate, nor on Amazon Nova Sonic, which is served over its own bidirectional stream.
**Example with headers:**
```bash
curl -X POST "$BASE/v1/audio/translations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "X-Amzn-Bedrock-GuardrailIdentifier: your-guardrail-id" \
-H "X-Amzn-Bedrock-GuardrailVersion: 1" \
-F file=@spanish-interview.mp3 \
-F model=amazon.transcribe \
-F response_format=json
```
!!! info "Detailed Documentation"
For complete information about these headers, configuration options, and use cases, see:
- [Bedrock Guardrails Configuration](operations_configuration.md#bedrock-guardrails)
- [Service Tier and Performance Configuration](operations_configuration.md#bedrock-service-tier-and-performance-configuration)
## Try It Now
**Translate foreign audio to English text:**
```bash
curl -X POST "$BASE/v1/audio/translations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F file=@spanish-interview.mp3 \
-F model=amazon.transcribe \
-F response_format=json
```
**Translate via JSON body (MCP and AI agents):**
When using MCP tools or HTTP clients that cannot construct multipart requests, pass the audio as a data URI or URL:
```bash
# Data URI (inline base64)
curl -X POST "$BASE/v1/audio/translations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"file": "data:audio/mp3;base64,",
"model": "amazon.transcribe"
}'
```
```bash
# HTTPS URL (server fetches the audio)
curl -X POST "$BASE/v1/audio/translations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"file": "https://example.com/audio.mp3",
"model": "amazon.transcribe"
}'
```
```bash
# Files API reference (file-id: URI scheme)
curl -X POST "$BASE/v1/audio/translations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"file": "file-id:file-0190c51c7de7455d9b8c2efe27dfbf67",
"model": "amazon.transcribe"
}'
```
See [Files API → Referencing Uploaded Files](api_openai_files.md#referencing-uploaded-files-via-the-file-id-uri-scheme) for the full description of the `file-id:` URI scheme.
**Translate foreign audio to English subtitles:**
```bash
curl -OJ -X POST "$BASE/v1/audio/translations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F file=@spanish-interview.mp3 \
-F model=amazon.transcribe \
-F response_format=srt
```
---
**Ready to translate multilingual audio?** Explore available models in the [Models API](api_openai_models.md).
---
## Realtime
Source: https://stdapi.ai/md/api_openai_realtime.md
# Realtime API
Hold a live, bidirectional speech-to-speech conversation over a single WebSocket, through the OpenAI Realtime API shape. Audio flows in both directions on the same connection: send the caller's speech as it is captured, and receive the model's spoken answer as it is generated — no request/response round trip per turn.
## Why Choose the Realtime API?
- :material-swap-horizontal: __Drop-in OpenAI Compatibility__
The same client event and server event vocabulary as the OpenAI Realtime API. `client.realtime.connect(model=...)` works by changing the base URL.
- :material-incognito: __Browser-Safe Ephemeral Secrets__
Mint a short-lived credential server-side and hand it to an untrusted browser or mobile client — your API key never leaves your backend.
- :material-server-off: __Stateless at Any Scale__
An ephemeral secret is a signed token, not a server-side record: any instance behind a load balancer verifies one minted by any other, with no shared session store.
- :material-phone-in-talk: __Telephony-Ready Audio__
24 kHz PCM by default, or G.711 (`audio/pcmu`, `audio/pcma`) at 8 kHz for direct interoperability with telephony and SIP media.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|-------------------------------|--------|-------------------------------------------------------------------|------------------|-------------------------------|
| `/v1/realtime/client_secrets` | `POST` | Mint a short-lived, signed client secret carrying a session configuration | Amazon Bedrock | `openai_realtime_client_secret` |
| `/v1/realtime?model=` | `WS` | Open a live, bidirectional speech-to-speech session | Amazon Bedrock | Not applicable — a persistent connection |
!!! info "WebSocket only — no WebRTC and no SIP"
`POST /v1/realtime/calls`, which upstream uses to negotiate a WebRTC peer connection or to accept a SIP call, is **not available** and answers `404`. Every session runs over the WebSocket above — a browser's included.
[Transports](#transports) covers the whole picture: how a browser connects, and how to put WebRTC or a phone line in front of this deployment today.
## Feature Compatibility
| Feature | Status | Notes |
|------------------------------------------------|:-----------------------------------------:|---------------------------------------------------------------------|
| **Client Events** | | |
| `session.update` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Voice, instructions and audio formats are fixed once the conversation opens — see [below](#voice-instructions-and-audio-formats-are-fixed-once-the-conversation-opens) |
| `input_audio_buffer.append` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Base64-encoded audio in the session's configured input format; at most 4 MiB per event, so send it in chunks as it is captured |
| `input_audio_buffer.commit` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Required to end a turn when `turn_detection` is `null`; at most 5.7 MB of audio may wait for one |
| `input_audio_buffer.clear` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Discards buffered, not-yet-committed audio |
| `conversation.item.create` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Text items only — an audio item is refused with a clear `error`; send speech through `input_audio_buffer.append` |
| `conversation.item.truncate` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Answered with `conversation.item.truncated` — see [below](#truncating-an-answer-the-caller-spoke-over) |
| `conversation.item.retrieve` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Answered with `conversation.item.retrieved`, carrying the item's role, status and transcript; audio is not retained, so the item carries no `audio` field |
| `conversation.item.delete` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Answered with `conversation.item.deleted`; the item stops being addressable, and the model keeps its own memory of the conversation |
| `response.create` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Ends any open turn and starts the model answering; its per-response `response` payload is ignored, and the session's own configuration serves every answer |
| `response.cancel` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Ends the answer in progress with `status: "cancelled"`; what the model keeps speaking is dropped rather than reported |
| `output_audio_buffer.clear` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Acknowledged with `output_audio_buffer.cleared` |
| **Server Events** | | |
| `session.created` / `session.updated` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Sent on connect and after every accepted `session.update` |
| `input_audio_buffer.speech_started` / `.speech_stopped` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Server-side voice activity detection only |
| `input_audio_buffer.committed` / `.cleared` | :material-check-circle:{ .success role="img" aria-label="Supported" } | |
| `conversation.item.added` / `.done` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Sent for every item: a written one, the caller's committed audio, and each answer — the answer's `.done` precedes its `response.done` |
| `conversation.item.created` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Sent beside `conversation.item.added` for a written item, for clients predating the added/done pair |
| `conversation.item.truncated` / `.retrieved` / `.deleted` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Answers to the matching client event |
| `conversation.item.input_audio_transcription.delta` / `.completed` | :material-cog:{ .model-dep role="img" aria-label="Conditional" } | Only when `audio.input.transcription` is set on the session |
| `conversation.item.input_audio_transcription.failed` | :material-cog:{ .model-dep role="img" aria-label="Conditional" } | Sent instead of a transcript when a caller turn could not be read, so it is not mistaken for a caller who said nothing; only when `audio.input.transcription` is set |
| `conversation.item.input_audio_transcription.segment` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Not emitted — a transcript carries no per-speaker segments or timings |
| `response.created` / `response.done` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Both carry the whole response object — see [below](#what-a-response-object-reports); `response.done` adds the answer's token usage |
| `response.output_item.added` / `.done` | :material-check-circle:{ .success role="img" aria-label="Supported" } | |
| `response.content_part.added` / `.done` | :material-check-circle:{ .success role="img" aria-label="Supported" } | |
| `response.output_audio.delta` / `.done` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Spoken answers only |
| `response.output_audio_transcript.delta` / `.done` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Spoken answers only |
| `response.output_text.delta` / `.done` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Text-only answers (`output_modalities: ["text"]`) |
| `output_audio_buffer.cleared` | :material-check-circle:{ .success role="img" aria-label="Supported" } | |
| `error` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Non-fatal for a rejected event; terminal (closes the socket) for a fatal one |
| `rate_limits.updated` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Not emitted |
| `input_audio_buffer.timeout_triggered` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Not emitted — it reports an `idle_timeout_ms` that is not available |
| `output_audio_buffer.started` / `.stopped` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Not emitted — they belong to the WebRTC and SIP [transports](#transports), which this API does not serve |
| **Audio Formats** | | |
| `audio/pcm` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Default — 24 kHz, 16-bit, mono, little-endian |
| `audio/pcmu`, `audio/pcma` | :material-check-circle:{ .success role="img" aria-label="Supported" } | G.711 at 8 kHz, for telephony interoperability |
| Independent input/output formats | :material-check-circle:{ .success role="img" aria-label="Supported" } | Configured separately under `audio.input.format` / `audio.output.format` |
| **Turn Detection** | | |
| Server-side voice activity detection | :material-check-circle:{ .success role="img" aria-label="Supported" } | Default; ends each turn automatically |
| Manual turns (`turn_detection: null`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | End each turn yourself with `input_audio_buffer.commit` |
| `threshold`, `prefix_padding_ms`, `silence_duration_ms`, `idle_timeout_ms`, `eagerness` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted and ignored — detection sensitivity is not tunable |
| `create_response`, `interrupt_response` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted and ignored — a detected turn always starts a response, and interruption is the model's own decision |
| Barge-in (caller speaks over the answer) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Handled by the model itself |
| **Voices** | | |
| OpenAI voice names | :material-check-circle:{ .success role="img" aria-label="Supported" } | `alloy`, `ash`, `ballad`, `cedar`, `coral`, `echo`, `marin`, `sage`, `shimmer`, `verse` — each served by the model's own nearest voice, so the timbre is not the upstream one |
| Any other voice name | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Passed through to the model as given, so a model voice can be named directly |
| **Not Available** | | |
| `POST /v1/realtime/calls` (WebRTC, SIP) | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Answers `404`; sessions run over the WebSocket only — see [Transports](#transports) for the browser and telephony route |
| `tools`, `tool_choice`, `parallel_tool_calls` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted and ignored — the session calls no tools |
| `prompt` (prompt templates) | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted and ignored |
| `reasoning`, `tracing`, `truncation` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted and ignored |
| `include` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted and ignored — no extra output fields are available |
| `audio.input.noise_reduction` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted and ignored — incoming audio is not filtered |
| `audio.input.transcription.model` / `.language` / `.prompt` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted and ignored — the transcript comes from the session's own model, which detects the language and takes no vocabulary hint. Setting the `transcription` object at all is what turns the events on |
| `audio.output.speed` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted and ignored — the spoken answer is not time-scaled |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-cog:{ .model-dep role="img" aria-label="Conditional" } **Conditional** — Depends on session configuration
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
### Voice, instructions and audio formats are fixed once the conversation opens
The model's voice, its system `instructions`, and both audio formats are set when the conversation with the model opens, and cannot change for the rest of that session. The conversation opens on the **first thing sent into it** — the first `input_audio_buffer.append` under the default server voice activity detection, or the first `input_audio_buffer.commit`, `response.create` or `conversation.item.create` under manual turns — which is well before the model has answered anything.
Send `session.update` with these settings **before sending anything else**, or open a new session to change them. Afterwards, a `session.update` touching only other fields (`turn_detection`, `max_output_tokens`, transcription settings, and so on) is still accepted; one that would change voice, instructions or an audio format is refused with an `error` event.
### Answering a written turn
`conversation.item.create` carrying an `input_text` part adds the text to the conversation without starting an answer. Follow it with `response.create`, and the model answers it exactly as it answers a spoken one — the same `response.output_audio.delta` chunks and the same transcript — so a written nudge into a voice session ("the caller has been on hold", "wrap up now") needs no second channel.
### What a response object reports
`response.created` and `response.done` carry the same response object, and every field the upstream API sends is present on both — a voice framework validates each frame against its own models, and a missing field is the event never arriving rather than a cosmetic difference.
| Field | What it carries |
|---|---|
| `status_details` | `null` while the answer is in progress and once it has completed. An answer the caller spoke over reports `{"type": "incomplete", "reason": "turn_detected"}`, and one ended by `response.cancel` reports `{"type": "cancelled", "reason": "client_cancelled"}` |
| `conversation_id` | The conversation the answer was added to — one per session, so every answer of a session names the same one |
| `output_modalities` | `["audio"]`, or `["text"]` when the session asked for text-only answers |
| `max_output_tokens` | The session's `max_output_tokens`, or `"inf"` when it sets none |
| `audio` | The session's effective output `format` and `voice`; `voice` is `null` when the session named none and the model answered in its own |
| `metadata` | Always `null` — an answer carries no metadata, since `response.create` takes no per-response configuration |
| `output`, `usage` | The answer's conversation item, and the tokens it used (on `response.done`) |
### Truncating an answer the caller spoke over
The model generates speech faster than it is played, so a caller who interrupts has heard less of the answer than was sent. Send `conversation.item.truncate` with the item's `id`, `content_index: 0` and the `audio_end_ms` your player actually reached; the session cuts its record of that item to what was heard and answers `conversation.item.truncated`.
- The item's **transcript is removed whole**, not trimmed: nothing aligns a transcript to a position in the audio, and leaving text the caller never heard in the record is the failure this event exists to prevent.
- `audio_end_ms` past the end of the item's audio, an item that is not an assistant message, and an item this session never sent are each refused with an `error`.
- What the model itself remembers of the answer is the model's own; truncation aligns the record this session reports through `conversation.item.retrieved`.
### Guardrail coverage
When the deployment configures an [Amazon Bedrock guardrail](operations_configuration.md#bedrock-guardrails), a realtime session is checked **per turn**: what the caller said (as the model transcribes it) as `INPUT`, and each completed answer as `OUTPUT`. A blocked turn ends the session with a terminal `error` event and close code `3000`.
Unlike a request/response route, the check cannot come before the content reaches the client: the model's speech is streamed while it is being generated and its transcript is only complete once the answer is over, so a blocked answer may already have been partly heard when the session ends. Written items sent with `conversation.item.create` are checked as `INPUT` **before** they reach the model, as on every other route.
## Model Support
Every deployment's catalog differs, and a model's own name is never guaranteed stable across accounts. Find which models serve this route:
```bash
curl "$BASE/search_models?route=openai_realtime" \
-H "Authorization: Bearer $OPENAI_API_KEY"
```
Pass the returned model ID as `model` on the WebSocket URL, or in the `session.model` field of an ephemeral secret's configuration. See the [Search Models API](api_search_models.md) for the full filter syntax.
## Authentication
Open the WebSocket with one of three credentials, carried in whichever channel the client can use:
| Client | Credential carrier |
|--------------------------------------------|------------------------------------------------------------------------|
| Server-side SDKs | `Authorization: Bearer ` header |
| Other gateway clients | `x-api-key: ` header |
| Browser (cannot set custom WebSocket headers) | `Sec-WebSocket-Protocol` list entry `openai-insecure-api-key.` |
Either the deployment's own API key or an [ephemeral client secret](#ephemeral-client-secrets) (`ek_...`) works as the credential. The `model` query parameter (`/v1/realtime?model=`) selects the model serving the session; it may be omitted when the credential is an ephemeral secret whose session configuration already names one. See [Authentication & Security](operations_authentication_security.md) for how the API key itself is configured.
!!! warning "A refused credential is not an HTTP status"
The WebSocket upgrade always completes first, so a rejected or expired credential is **not** answered with `401`/`403`. The connection opens, the first and only event is a terminal `error` with `code: "invalid_api_key"`, and the socket is then closed with close code `3000` and reason `invalid_request_error.invalid_api_key` — the same shape the upstream API uses. Instrument the `error` event and the close code, not the handshake status.
## Ephemeral Client Secrets
`POST /v1/realtime/client_secrets` mints a short-lived credential — a value starting with `ek_` — that carries a session configuration. Hand it to a browser or mobile client so it can open a session directly, without ever holding the deployment's own API key.
```bash
curl -X POST "$BASE/v1/realtime/client_secrets" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"expires_after": {"anchor": "created_at", "seconds": 60},
"session": {
"type": "realtime",
"model": "amazon.nova-2-sonic-v1:0",
"instructions": "You are a helpful support agent."
}
}'
```
```json
{
"value": "ek_...",
"expires_at": 1731000060,
"session": {
"type": "realtime",
"model": "amazon.nova-2-sonic-v1:0",
"instructions": "You are a helpful support agent.",
"audio": { "...": "..." }
}
}
```
- `expires_after.seconds` accepts **10 to 7,200** seconds, defaulting to **600** (10 minutes) when omitted. This bounds how long the secret can be used to *open* a session — a session already opened with it keeps running for its own [session limit](#session-lifecycle-and-limits).
- `session` accepts the same configuration a client would otherwise send in a `session.update` event; it is applied to every session opened with the secret. It also accepts `{"type": "transcription"}`, which opens sessions that only transcribe the caller and never answer out loud — the only way to ask for one, since the socket itself takes no session type.
- By default the carried configuration is a **default, not a constraint**: the client may name another model on the `?model=` query string and change the configuration with its own `session.update`, as it can upstream. Set [`REALTIME_ALLOW_SESSION_OVERRIDE=false`](operations_configuration.md#realtime-allow-session-override) to make the model, the `instructions` and `max_output_tokens` the secret was minted with final — a mismatching `?model=` is then refused at connect, and a `session.update` changing one of them answers an `error`.
!!! warning "What a secret grants until it expires"
A secret cannot be revoked: rotating [`REALTIME_CLIENT_SECRET_KEY`](operations_configuration.md#realtime-client-secret-key) invalidates every outstanding one at once, and nothing else does. Until then it may open **any number of concurrent sessions**, each billed to the deployment — so keep `expires_after.seconds` as short as the flow allows.
Its payload is signed, not encrypted: whoever holds the secret can read the session configuration it carries. Nothing confidential belongs in `instructions`.
!!! info "Stateless, and signed"
Nothing is stored server-side: the secret is the session configuration plus a signature, so **any instance behind a load balancer verifies a secret minted by any other** — no shared session store, no sticky routing required.
The signing key is derived from the deployment's configured API key by default. When the deployment runs with **no API key configured at all**, the signing key falls back to a random value generated **per process**: minted secrets then only verify on the instance that minted them, and stop working once a request reaches a different one. Set [`realtime_client_secret_key`](operations_configuration.md#realtime-client-secret-key) explicitly to fix a key shared by every instance regardless of the API key configuration.
## Transports
Upstream offers a realtime session over three transports — WebSocket, WebRTC and SIP. **This API serves the WebSocket, and only the WebSocket.** `POST /v1/realtime/calls`, the endpoint upstream uses to trade an SDP offer for a WebRTC peer connection or to accept an inbound SIP call, has no route here and answers `404`; it is planned for a later release.
### A browser connects to that same WebSocket
There is no separate browser transport to be missing. The page opens `wss:///v1/realtime?model=` itself, carrying an [ephemeral client secret](#ephemeral-client-secrets) in the way [Authentication](#authentication) describes, so the deployment's own API key never leaves your backend. It is two steps — mint the secret server-side, connect with it client-side — and the [browser example](#browser-ephemeral-client-secret) below is both of them.
What the page owns in exchange is the media. Capturing the microphone, resampling it to the session's input format and playing back the `response.output_audio.delta` chunks are its own work, because a WebSocket carries the audio bytes handed to it and nothing else: no jitter buffer, no packet-loss concealment, no echo cancellation. On a good network that is unremarkable; on a lossy one it is audible, and it is the reason to reach for a media stack rather than write one.
### Put WebRTC or a phone line in front of the gateway
The route to a browser peer connection or a phone call does not run through `POST /v1/realtime/calls`. The two frameworks most voice agents are already built on — **LiveKit Agents** and **Pipecat** — terminate WebRTC themselves (and SIP, through their telephony transports), and reach the model over exactly the WebSocket this API serves. The caller speaks WebRTC or SIP to the framework; the framework speaks this API. Pointing one at this deployment asks no more of it than the rest of this gateway does: the base URL, the deployment's API key, and the model name only where the name differs.
**LiveKit Agents** takes an HTTP base URL and derives the WebSocket from it, exactly as the official SDK does:
```python
from livekit.agents import AgentSession
from livekit.plugins import openai
session = AgentSession(
llm=openai.realtime.RealtimeModel(
model="amazon.nova-2-sonic-v1:0",
base_url="https://your-deployment.example.com/v1",
api_key="YOUR_API_KEY",
)
)
```
`base_url` also reads from the `OPENAI_BASE_URL` environment variable, and `api_key` from `OPENAI_API_KEY`. A base URL ending in `/v1` has `/realtime` appended for you; a deployment served under a non-default [`OPENAI_ROUTES_PREFIX`](operations_configuration.md#openai-routes-prefix) has to name the full path itself.
**Pipecat** takes the WebSocket URL whole, `/v1/realtime` included:
```python
import os
from pipecat.services.openai.realtime.llm import OpenAIRealtimeLLMService
llm = OpenAIRealtimeLLMService(
base_url="wss://your-deployment.example.com/v1/realtime",
api_key=os.environ["OPENAI_API_KEY"],
settings=OpenAIRealtimeLLMService.Settings(model="amazon.nova-2-sonic-v1:0"),
)
```
On a telephony leg, set the session's audio formats to `audio/pcmu` or `audio/pcma` — G.711 at 8 kHz is what a phone call already carries, so nothing resamples it twice.
!!! note "Check the constructor against the version you install"
The parameters above are as shipped in `livekit-plugins-openai` 1.6.10 and `pipecat-ai` 1.7.0. Both projects have renamed this integration before — Pipecat's service moved out of `pipecat.services.openai_realtime_beta`, and LiveKit does not list `base_url` on its own parameters page — so read [LiveKit's OpenAI realtime plugin](https://docs.livekit.io/agents/models/realtime/plugins/openai/) and [Pipecat's OpenAI realtime service](https://docs.pipecat.ai/api-reference/server/services/s2s/openai) for the release you pin.
### Why WebRTC is a different kind of endpoint
An HTTP request that returns an SDP answer is the small, visible part of WebRTC. The rest is a media transport in its own right: a UDP path negotiated separately from the HTTPS connection, carrying encrypted RTP, with its own address discovery, NAT traversal and congestion control. Serving it means **terminating that media path**, not routing one more request — and the two are not the same thing to build, nor the same thing to put an ingress in front of. SIP has the same shape under different names: signalling on one connection, audio on another.
That is also why a framework is the shorter path rather than a stopgap. LiveKit and Pipecat already own a media stack, already run wherever your users are, and already speak this API on the other side. If you do intend to run a media terminator of your own, [WebRTC and SIP need their own ingress](operations_deploy_advanced.md#webrtc-and-sip-need-their-own-ingress) covers what that costs a deployment.
## Session Lifecycle and Limits
- **Duration cap** — a session lasts at most **8 minutes**. When it is reached, the server closes the connection with WebSocket close code `1000` and reason `session_expired`; reconnect to continue the conversation.
- **Conversation ended by the model side** — when the model ends the conversation itself, the connection closes with close code `1000` and reason `session_ended`. Normal, and reconnecting starts a new session.
- **Server shutdown** — a session still open when the deployment shuts down is closed with close code `1001` and reason `server_shutdown`.
- **Fatal errors** — a fatal error sends a terminal `error` event, then closes the connection with close code `3000`, whose reason is `.` (e.g. `invalid_request_error.model_not_found`).
- **Event size** — a single client event may carry at most **4 MiB**, base64 included; a larger one answers an `error` and is dropped. Append audio in the small chunks it is captured in rather than whole files.
- **Uncommitted audio** — under manual turns (`turn_detection: null`), at most **5.7 MB** of decoded audio may be buffered before an `input_audio_buffer.commit` (about 2 minutes of 24 kHz PCM, longer for G.711); past that the append answers an `error`. Commit each turn, or clear the buffer with `input_audio_buffer.clear`.
- **Addressable items** — the session keeps its **200** most recent conversation items addressable, dropping the oldest past that. `conversation.item.truncate`, `.retrieve` and `.delete` answer an `error` for an item that has fallen out; the model's own memory of the conversation is unaffected.
## Billing { #cost }
A realtime session bills audio and text tokens continuously, in both directions, for as long as the connection is open — not per request. Usage is reported per answer, in each `response.done` event, and recorded in the gateway's usage log the same way, so a session that drops mid-conversation still accounts for everything spoken before it. Speech tokens are priced well above text tokens by AWS, and the two are recorded and priced separately here. See [Cost Management](operations_cost_management.md) for how usage becomes cost.
## Try It Now
### Python (server-side, with the official SDK)
Requires the `openai` package with realtime support (`pip install "openai[realtime]"`). Point the client's `base_url` at this deployment; `client.realtime.connect(...)` derives the WebSocket URL from it automatically.
```python
import base64
from openai import AsyncOpenAI
client = AsyncOpenAI(
api_key="YOUR_API_KEY", base_url="https://your-deployment.example.com/v1"
)
async def main() -> None:
async with client.realtime.connect(model="amazon.nova-2-sonic-v1:0") as connection:
await connection.session.update(
session={
"type": "realtime",
"instructions": "You are a concise, friendly voice assistant.",
}
)
# Stream 24 kHz, 16-bit, mono, little-endian PCM in the small chunks it
# is captured in -- ~100 ms each here, well under the per-event limit.
with open("question.pcm", "rb") as audio_file:
while chunk := audio_file.read(4800):
await connection.input_audio_buffer.append(
audio=base64.b64encode(chunk).decode()
)
await connection.input_audio_buffer.commit()
await connection.response.create()
async for event in connection:
if event.type == "response.output_audio.delta":
# event.delta is base64-encoded audio in the session's output format.
...
elif event.type == "response.output_audio_transcript.delta":
print(event.delta, end="", flush=True)
elif event.type == "response.done":
break
```
### Browser (ephemeral client secret)
Mint the secret from your backend — never expose the deployment's own API key to the browser — then connect directly from client-side JavaScript using the `openai-insecure-api-key.` subprotocol, since a browser cannot set custom WebSocket headers:
```javascript
// Fetched from your own backend, which called POST /v1/realtime/client_secrets
const { value: ephemeralSecret } = await fetch("/api/realtime-secret").then((r) => r.json());
const ws = new WebSocket(
"wss://your-deployment.example.com/v1/realtime?model=amazon.nova-2-sonic-v1:0",
["realtime", `openai-insecure-api-key.${ephemeralSecret}`],
);
ws.addEventListener("open", async () => {
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
// Your own capture code: resample the microphone to the session's input
// format (24 kHz, 16-bit, mono PCM by default) and hand over small chunks.
captureAudioChunks(stream, (pcmChunk) => {
ws.send(
JSON.stringify({
type: "input_audio_buffer.append",
audio: btoa(String.fromCharCode(...new Uint8Array(pcmChunk))),
}),
);
});
// Nothing else is needed: server voice activity detection ends each turn and
// starts the answer. Under `turn_detection: null`, send
// `input_audio_buffer.commit` yourself instead.
});
ws.addEventListener("message", (event) => {
const serverEvent = JSON.parse(event.data);
if (serverEvent.type === "response.output_audio.delta") {
// serverEvent.delta is base64-encoded audio in the session's output format.
}
});
```
---
**Ready to add voice to your application?** Find compatible models with the [Search Models API](api_search_models.md), or explore the full [audio suite](api_openai_audio_speech.md) for turn-based speech and transcription.
---
## Embeddings
Source: https://stdapi.ai/md/api_openai_embeddings.md
# Embeddings API
Generate vector embeddings for semantic search and RAG applications with Amazon Bedrock embedding models through an OpenAI-compatible interface.
## Why Choose the Embeddings API?
- :material-magnify: __Semantic Search__
Find content based on meaning and context, not just exact words. For knowledge bases and document retrieval.
- :material-lightning-bolt: __High Performance__
Amazon Bedrock embedding models deliver fast vectors optimized for production workloads. Batch processing for large-scale operations.
- :material-puzzle: __Flexible Dimensions__
Choose vector dimensions that match your needs. Balance accuracy and storage/compute costs with model-specific dimension control.
- :material-image-multiple: __Multimodal Embeddings__
Process images, videos, audio, and PDF documents alongside text. Unified embeddings for cross-modal search using base64 data URI input.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|------------------|--------|-------------------------------------------------------------|------------------------------|--------------------|
| `/v1/embeddings` | `POST` | Transform text and multimodal content into semantic vectors | Amazon Bedrock Embedding Models | `openai_embedding` |
## Feature Compatibility
| Feature | Status | Notes |
|------------------------------|:----------------------------------------:|-----------------------------------------------------------------|
| **Input Types** | | |
| Text input (single string) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Full support for text embeddings |
| Multimodal input | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Image, audio, video, document (image + text) |
| Multiple input (batch array) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Process multiple inputs efficiently |
| Token array input | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Array of token integers not supported |
| **Output Formats** | | |
| Float vectors | :material-check-circle:{ .success role="img" aria-label="Supported" } | Standard floating-point arrays |
| Base64 encoding | :material-check-circle:{ .success role="img" aria-label="Supported" } | Base64-encoded float32 arrays |
| **Model Parameters** | | |
| `dimensions` override | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Some models support dimension reduction |
| `encoding_format` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Choose `float` or `base64` |
| Extra model-specific params | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Extra model-specific parameters not supported by the OpenAI API |
| **Usage tracking** | | |
| Input text tokens | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Reported as billed by Amazon Bedrock; zero on models and inputs for which Bedrock returns no token count |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Model-Dependent** — Behavior depends on the model or backend; check the Notes column
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
## Model Support
Any Amazon Bedrock model that produces embeddings answers on this route — the Amazon Titan Embed and Amazon Nova embedding families, Cohere Embed, and TwelveLabs Marengo. A model that produces something else is refused with `400`, naming it.
To list the models this deployment serves on this route, call [`search_models`](api_search_models.md) with `route=openai_embedding`. Which of them accept images, video or audio rather than text alone is covered under [Multimodal Embeddings](#multimodal-embeddings).
!!! note "No Bedrock Mantle models here"
[Bedrock Mantle](features.md#bedrock-mantle-models) serves chat models only. Embeddings always come from the classic `bedrock-runtime` catalogue, whether or not Mantle is enabled.
### Model Name Aliases
Cohere models carry the dotted version their own API publishes, so the name you already send resolves without change:
- `embed-v4.0` → `cohere.embed-v4:0`
- `embed-english-v3.0` → `cohere.embed-english-v3`
## Advanced Features
### Embedding a Corpus in Bulk
A large corpus does not have to be embedded on the request path. Upload the
requests as a file and run them through the [Batch API](api_openai_batches.md)
against `/v1/embeddings`: they run without a connection held open, at the
published batch rate, and the vectors are read back from the result file. One
batched request embeds one `input`, and its vectors come back as numbers.
### Provider-Specific Parameters
Access advanced embedding capabilities by passing provider-specific parameters directly in your requests. These parameters are forwarded to Amazon Bedrock and allow you to access features unique to each embedding model provider.
**Documentation:** [Bedrock Embedding Model Parameters](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html)
**How It Works:**
Add provider-specific fields at the top level of your request body alongside standard OpenAI parameters. The API automatically forwards these to the appropriate model provider via Amazon Bedrock.
**Examples:**
**Cohere Embed v4 - Input Type:**
```json
{
"model": "cohere.embed-v4:0",
"input": "Semantic search transforms how we find information",
"input_type": "search_query"
}
```
**Amazon Titan Embed v2 - Normalization:**
```json
{
"model": "amazon.titan-embed-text-v2:0",
"input": "Product description for similarity matching",
"normalize": true
}
```
**Configuration Options:**
**Option 1: Per-Request**
Add provider-specific parameters directly in your request body (as shown in examples above).
**Option 2: Server-Wide Defaults**
Configure default parameters for specific models via the `DEFAULT_MODEL_PARAMS` environment variable:
```bash
export DEFAULT_MODEL_PARAMS='{
"cohere.embed-v4:0": {
"input_type": "search_document",
"truncate": "END"
}
}'
```
**Note:** Per-request parameters override server-wide defaults.
**Behavior:**
- :material-check-circle:{ .success role="img" aria-label="Supported" } **Compatible parameters**: Forwarded to the model and applied
- :material-alert-circle:{ .warning } **Unsupported parameters**: Return HTTP 400 with an error message
## Multimodal Embeddings
Go beyond text! Supported models can process images, videos, and audio through base64 data URI input. This enables powerful cross-modal search and similarity features.
### Input Format
Multimodal content is passed as base64-encoded data URIs:
```text
data:;base64,
```
### Example: Image Embedding
```bash
# First, encode your image to base64
IMAGE_B64=$(base64 -w 0 image.jpg)
# Send the embedding request
curl -X POST "$BASE/v1/embeddings" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"model\": \"amazon.nova-2-multimodal-embeddings-v1:0\",
\"input\": \"data:image/jpeg;base64,$IMAGE_B64\"
}"
```
### Example: Video Embedding
**Option 1: Base64-encoded video (for small files)**
```bash
# First, encode your video to base64
VIDEO_B64=$(base64 -w 0 video.mp4)
# Send the embedding request
curl -X POST "$BASE/v1/embeddings" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"model\": \"amazon.nova-2-multimodal-embeddings-v1:0\",
\"input\": \"data:video/mp4;base64,$VIDEO_B64\"
}"
```
!!! info "Automatic S3 Upload"
When you provide Base64-encoded data that exceeds the model's size limit (or Bedrock's 25 MB quota), the server automatically stages it in S3 so the request still succeeds.
To allow this behavior, configure regional S3 buckets via `AWS_S3_REGIONAL_BUCKETS` in the same region as your Bedrock model. See [configuration guide](operations_configuration.md#aws-s3-regional-buckets).
!!! warning "Usage Not Available for Large Inputs"
Amazon Nova models report no token usage for large inputs (those exceeding the model's inline size limit): the `usage` field then reports zero tokens.
!!! warning "Large Base64 Files and Memory Configuration"
While passing large files as Base64 is supported, ensure your server has sufficient memory configured. Large Base64-encoded files (especially videos) can consume significant memory during processing. Consider using S3 URLs directly for very large files, or adjust your server's memory limits accordingly.
**Option 2: S3 URL (for large files)**
```bash
# Send the embedding request with S3 URL
curl -X POST "$BASE/v1/embeddings" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-2-multimodal-embeddings-v1:0",
"input": "s3://my-bucket/path/to/video.mp4"
}'
```
!!! warning "S3 URL Requirements"
When using S3 URLs directly:
- S3 bucket **must be in the same AWS region** as the Bedrock model
- The stdapi.ai server **must have read access** to the S3 object
- For TwelveLabs Marengo models: S3 bucket **must be in the same AWS account** as the stdapi.ai server
**Option 3: Files API reference (`file-id:`)**
Reference a file previously uploaded via the [Files API](api_openai_files.md#referencing-uploaded-files-via-the-file-id-uri-scheme) using the project-local `file-id:` URI scheme:
```bash
curl -X POST "$BASE/v1/embeddings" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-2-multimodal-embeddings-v1:0",
"input": "file-id:file-0190c51c7de7455d9b8c2efe27dfbf67"
}'
```
### Example: PDF Document Embedding
For PDFs, convert each page to an image and send via inputs along with page metadata (e.g., file_name, entities) in adjacent text parts. **For RAG applications, smaller chunks often improve retrieval accuracy and reduce costs.**
**{ style="height: 1.2em; vertical-align: text-bottom;" } Cohere Embed v4** (supports multiple text+image pairs in one request):
```bash
# Convert PDF pages to images (using ImageMagick or similar tool)
convert -density 150 document.pdf page-%d.jpg
# Encode each page image to base64
PAGE_1=$(base64 -w 0 page-0.jpg)
PAGE_2=$(base64 -w 0 page-1.jpg)
# Generate document embedding with metadata
curl -X POST "$BASE/v1/embeddings" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"model\": \"cohere.embed-v4:0\",
\"input\": [
\"file_name: report.pdf, page: 1\",
\"data:image/jpeg;base64,$PAGE_1\",
\"file_name: report.pdf, page: 2\",
\"data:image/jpeg;base64,$PAGE_2\"
]
}"
```
**{ style="height: 1.2em; vertical-align: text-bottom;" } TwelveLabs Marengo v3** (requires exactly one text + one image per request):
!!! warning "Usage Not Available"
TwelveLabs Marengo models do not return token usage information. The `usage` field in the response will always report zero tokens.
!!! info "Text+Image Pairing for Marengo v3"
When using `twelvelabs.marengo-embed-3-0-v1:0`, if you provide exactly **2 inputs** where one is text and one is image, they are automatically combined into a single `text_image` embedding. This creates a unified multimodal representation of the text-image pair.
```bash
# Encode image to base64
IMAGE_B64=$(base64 -w 0 page-0.jpg)
# Generate text+image embedding (automatically uses text_image mode)
curl -X POST "$BASE/v1/embeddings" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"model\": \"twelvelabs.marengo-embed-3-0-v1:0\",
\"input\": [
\"A diagram showing the quarterly sales report\",
\"data:image/jpeg;base64,$IMAGE_B64\"
]
}"
```
### Mixed-Content Batching
Combine text and multimodal inputs in a single request:
```bash
curl -X POST "$BASE/v1/embeddings" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"model\": \"cohere.embed-v4:0\",
\"input\": [
\"A beautiful sunset over mountains\",
\"data:image/jpeg;base64,/9j/4AAQSkZJRg...\",
\"Nature photography collection\"
]
}"
```
### Use Cases
- **Visual Search**: Find images similar to a query image or text description
- **Video Analysis**: Search and retrieve video content based on visual similarity or text descriptions
- **Audio Similarity**: Find similar audio clips or match audio to text descriptions
- **Document Retrieval**: Find relevant PDFs based on visual and textual content
- **Cross-Modal Recommendations**: Recommend images, videos, or audio based on text queries and vice versa
- **Content Moderation**: Analyze and classify multimodal content at scale
---
**Build smarter search and recommendations!** Explore available embedding models in the [Models API](api_openai_models.md).
## Available Request Headers
This endpoint supports standard Bedrock headers for enhanced control over your requests. All headers are optional and can be combined as needed.
### Content Safety (Guardrails)
| Header | Purpose | Valid Values |
|--------------------------------------|------------------------------------|---------------------------------------|
| `X-Amzn-Bedrock-GuardrailIdentifier` | Guardrail ID for content filtering | Your guardrail identifier |
| `X-Amzn-Bedrock-GuardrailVersion` | Guardrail version | Version number (e.g., `1`) |
| `X-Amzn-Bedrock-Trace` | Guardrail trace level | `disabled`, `enabled`, `enabled_full` |
### Performance Optimization
| Header | Purpose | Valid Values |
|--------------------------------------------|------------------------|-------------------------------|
| `X-Amzn-Bedrock-Service-Tier` | Service tier selection | `default`, `flex`, `priority`, `reserved` |
| `X-Amzn-Bedrock-PerformanceConfig-Latency` | Latency optimization | `standard`, `optimized` |
**Example with headers:**
```bash
curl -X POST "$BASE/v1/embeddings" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Amzn-Bedrock-Service-Tier: flex" \
-H "X-Amzn-Bedrock-PerformanceConfig-Latency: standard" \
-d '{
"model": "amazon.nova-2-multimodal-embeddings-v1:0",
"input": ["Batch text 1", "Batch text 2", "Batch text 3"]
}'
```
!!! info "Detailed Documentation"
For complete information about these headers, configuration options, and use cases, see:
- [Bedrock Guardrails Configuration](operations_configuration.md#bedrock-guardrails)
- [Service Tier and Performance Configuration](operations_configuration.md#bedrock-service-tier-and-performance-configuration)
## Try It Now
**Single text embedding:**
```bash
curl -X POST "$BASE/v1/embeddings" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-2-multimodal-embeddings-v1:0",
"input": "Semantic search transforms how we find information"
}'
```
**Batch processing with base64 encoding:**
```bash
curl -X POST "$BASE/v1/embeddings" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-2-multimodal-embeddings-v1:0",
"input": ["Product description", "User query", "Related content"],
"encoding_format": "base64"
}'
```
---
## Moderations
Source: https://stdapi.ai/md/api_openai_moderations.md
# Moderations API
Classify content for harm with [Amazon Bedrock Guardrails](https://aws.amazon.com/bedrock/guardrails/), inline [guardrail checks](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-use-invoke-guardrail-checks.html) (no guardrail resource needed), or [Amazon Comprehend toxicity detection](https://docs.aws.amazon.com/comprehend/latest/dg/trust-safety.html) through an OpenAI-compatible Moderations interface.
## Why Choose the Moderations API?
- :material-shield-check: __Configurable Content Safety__
Bring your own Bedrock guardrail: categories, thresholds, denied topics, word filters, and sensitive-information policies are fully configurable in AWS.
- :material-flash: __Works Out of the Box__
Inline guardrail checks and Amazon Comprehend toxicity detection require no setup at all, so `/v1/moderations` works immediately on any deployment.
- :material-swap-horizontal: __Drop-in OpenAI Compatibility__
OpenAI moderation model names are accepted as aliases. Existing integrations work by changing the base URL.
- :material-cloud-lock: __Private AWS Backend__
Classifications run entirely in your own AWS account — no traffic to third-party endpoints.
| Feature | Status | Notes |
|----------------------------------|:---------------------------------------:|---------------------------------------------------------------------------|
| **Input** | | |
| `input` string / array of strings | :material-check-circle:{ .success role="img" aria-label="Supported" } | Each element yields one independent result |
| `input` text parts | :material-check-circle:{ .success role="img" aria-label="Supported" } | `{"type": "text", "text": ...}` |
| `input` image parts | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Guardrail resource models only (not guardrail checks); PNG and JPEG |
| Empty string input | :material-check-circle:{ .success role="img" aria-label="Supported" } | Returns an unflagged result without calling AWS (OpenAI parity); whitespace-only strings are **not** covered by this shortcut and are classified normally (billed Comprehend call) |
| `model` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Guardrail, Comprehend, or an OpenAI moderation model alias (see below) |
| **Output** | | |
| `flagged` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Also raised by guardrail policies without a mapped category, and on Comprehend by the overall toxicity score or unmapped labels such as profanity |
| `categories` / `category_scores` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Mapped categories only; OpenAI categories without a counterpart stay `false` / `0.0` |
| `category_applied_input_types` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Reflects each classified element's modality |
| **Usage tracking** | | |
| Guardrail text units / images | :material-check-circle:{ .success role="img" aria-label="Supported" } | Billing units (one text unit per 1,000 characters per input; one unit per image) |
| Comprehend units | :material-check-circle:{ .success role="img" aria-label="Supported" } | Billing unit (100 characters, minimum 3 per call) |
| **Other** | | |
| Model discovery | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Moderation models and their aliases appear in the [model listings](api_search_models.md) |
| Long text inputs | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Comprehend inputs of any length are split into API-sized segments transparently |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with OpenAI API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Model-Dependent** — Behavior depends on the model or backend; check the Notes column
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond OpenAI API
## Model Support
The moderation models appear in the [`/v1/models`](api_openai_models.md) and [`/search_models`](api_search_models.md) listings (`route=openai_moderation`) with their OpenAI aliases.
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Bedrock Guardrails
| Model | OpenAI aliases | Notes |
|------------------------------------|------------------------------------------------------|------------------------------------------------------------------------|
| `amazon.bedrock-runtime-guardrail` | `omni-moderation-latest`, `omni-moderation-2024-09-26` | The server's default guardrail. Text and image inputs. Listed only when a guardrail is configured |
| `amazon.bedrock-runtime-guardrail-checks` | `omni-moderation-*` when no guardrail is configured | Inline guardrail content filter checks ([`InvokeGuardrailChecks`](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-use-invoke-guardrail-checks.html)) — no guardrail resource needed. Text inputs only. Listed only when a configured Bedrock region offers the operation |
| ``, `:`, or guardrail ARN | — | Any explicit guardrail (requires guardrail override to be allowed) |
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Comprehend
| Model | OpenAI aliases | Notes |
|------------------------------|-----------------------------------------------------|------------------------------------------------|
| `amazon.comprehend-toxicity` | `text-moderation-latest`, `text-moderation-stable` | Toxicity detection. English-optimized; non-English input is classified on a best-effort basis, no images |
### Comparison
| Capability | Amazon Bedrock Guardrails | Amazon Bedrock Guardrail Checks | Amazon Comprehend |
|---------------------------|-----------------------------------------------------------------------------|--------------------------------------------------------|--------------------------------------------------------|
| Setup | Create and configure a guardrail in Amazon Bedrock | None — works out of the box in [supported regions](#selecting-the-model) | None — works out of the box |
| Text inputs | :material-check-circle:{ .success role="img" aria-label="Supported" } Any language supported by the guardrail | :material-check-circle:{ .success role="img" aria-label="Supported" } Languages supported by guardrail content filters | :material-check-circle:{ .success role="img" aria-label="Supported" } English-optimized; non-English input classified on a best-effort basis |
| Image inputs | :material-check-circle:{ .success role="img" aria-label="Supported" } PNG and JPEG | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } Not supported | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } Not supported |
| Mapped categories | `hate`, `harassment`, `sexual`, `violence`, `illicit` | `hate`, `harassment`, `sexual`, `violence`, `illicit` | `hate`, `harassment`, `sexual`, `violence`, `violence/graphic` |
| Category scores | Quantized confidence levels (`0.0` / `0.25` / `0.5` / `0.75`) | Severity scores in `0.2` increments (`0.0` – `1.0`) | Continuous scores (`0.0` – `1.0`) |
| Custom policies | Denied topics, word filters, PII, prompt attacks, contextual grounding | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } Content filters only | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } Fixed toxicity labels |
| Tunable thresholds | Per-filter strengths configured on the guardrail | Fixed flagging threshold (score ≥ 0.5) | Fixed flagging threshold (score ≥ 0.5) |
| `moderation` request parameter | :material-check-circle:{ .success role="img" aria-label="Supported" } Applied to generations natively | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } Moderations API only | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } Moderations API only |
| Input length | ApplyGuardrail text unit limits | InvokeGuardrailChecks text unit limits | Unlimited (split into 1 KB segments transparently) |
## Selecting the Model
The `model` parameter selects the moderation model:
| `model` value | Model used |
|----------------------------------------------|--------------------------------------------------------------------------------------------------|
| Omitted | The server's default guardrail; when none is configured, guardrail checks in a supported region, then Comprehend toxicity detection as a last resort |
| `amazon.bedrock-runtime-guardrail` | The server's default guardrail (an error when none is configured) |
| `amazon.bedrock-runtime-guardrail-checks` | Inline guardrail content filter checks, even when a guardrail is configured (an error when no configured region offers the operation) |
| `omni-moderation-*` | Same as an omitted model |
| `amazon.comprehend-toxicity` / `text-moderation-*` | Comprehend toxicity detection, even when a guardrail is configured |
| `` or `:` | That guardrail (requires guardrail override to be allowed) |
| Guardrail ARN | That guardrail, applied in the region embedded in the ARN |
The server guardrail comes from [`AWS_BEDROCK_GUARDRAIL_IDENTIFIER` / `AWS_BEDROCK_GUARDRAIL_VERSION`](operations_configuration.md#bedrock-guardrails), or from the `X-Amzn-Bedrock-GuardrailIdentifier` / `X-Amzn-Bedrock-GuardrailVersion` request headers when [`AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE`](operations_configuration.md#bedrock-guardrails) is enabled. Explicit guardrails in `model` also require that setting.
Guardrails are regional: a plain guardrail ID is applied in the primary Bedrock region, while an ARN selects its own region. Comprehend calls use [`AWS_COMPREHEND_REGION`](operations_configuration.md#aws-comprehend-region) (with multi-region failover otherwise).
Guardrail checks (`InvokeGuardrailChecks`) are available in a limited set of AWS regions only (currently `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-2`, `eu-north-1`, `ap-northeast-1`, and `ap-southeast-2`): calls run in the configured Bedrock regions that offer the operation, in priority order with multi-region failover, and the backend is unavailable when none of them does.
!!! note "Required IAM Permission"
Guardrail checks moderation requires the `bedrock:InvokeGuardrailChecks` IAM action. See [IAM Permissions](operations_configuration.md#iam-permissions). Deployments without it keep working: when guardrail checks are only reached as the default `omni-moderation-*` resolution, an `AccessDenied` response degrades the request to Comprehend toxicity detection with a logged warning.
!!! tip "Moderating generations directly"
The guardrail selection and category mapping also power the `moderation` request parameter of the [Chat Completions](api_openai_chat_completions.md) and [Responses](api_openai_responses.md) APIs: the guardrail is applied to the generation itself, and the classification of the input and output is reported in the response's `moderation` field — for Chat Completions on non-streaming requests only, and for Responses also on the terminal event when streaming. The `moderation` parameter requires a guardrail — Comprehend is not available there — and is rejected (`400`) on Amazon Bedrock Mantle-served models.
## Category Mapping
**Amazon Bedrock Guardrails** — content policy filters map to the OpenAI moderation categories:
| Amazon Bedrock filter | OpenAI category |
|--------------------|-----------------|
| `HATE` | `hate` |
| `INSULTS` | `harassment` |
| `SEXUAL` | `sexual` |
| `VIOLENCE` | `violence` |
| `MISCONDUCT` | `illicit` |
Filter confidence levels become scores: `NONE` → `0.0`, `LOW` → `0.25`, `MEDIUM` → `0.5`, `HIGH` → `0.75`.
Every other guardrail policy — denied topics, word filters, sensitive information (PII), prompt attacks, contextual grounding — still contributes to the top-level `flagged` field whenever the guardrail intervenes, even though no individual category is set.
**Amazon Bedrock Guardrail Checks** — the inline `contentFilter` check uses the same five-category mapping as guardrail content policy filters, but reports each category's severity score (`0.0` to `1.0` in `0.2` increments) directly in `category_scores`. A category — and the input — is flagged when its score reaches `0.5`.
**Amazon Comprehend** — toxicity labels map to the OpenAI moderation categories, with their detection scores (`0.0`–`1.0`) reported directly:
| Comprehend label | OpenAI category |
|-----------------------|--------------------|
| `HATE_SPEECH` | `hate` |
| `HARASSMENT_OR_ABUSE` | `harassment` |
| `INSULT` | `harassment` |
| `SEXUAL` | `sexual` |
| `VIOLENCE_OR_THREAT` | `violence` |
| `GRAPHIC` | `violence/graphic` |
| `PROFANITY` | *(`flagged` only)* |
An input is flagged when its overall toxicity or any label score reaches `0.5`. Long texts are split into 1 KB segments and the highest score per category is kept. Comprehend's `DetectToxicContent` API is always called with the English language code, regardless of the input's actual language.
With either model, OpenAI sub-categories without a counterpart (e.g. `self-harm`, `sexual/minors`) are always `false`.
## Inputs
Each input element is classified independently and yields one entry in `results`:
- **`input` as a string** — one text classification.
- **`input` as an array of strings** — one classification per string.
- **`input` as an array of parts** — `{"type": "text", "text": ...}` and `{"type": "image_url", "image_url": {"url": ...}}` parts. Images must be PNG or JPEG, and require a guardrail resource model (guardrail checks and Comprehend are text-only).
Each result's `category_applied_input_types` reflects the classified element's modality: `["text"]` for every category on text inputs; on image inputs, `["image"]` for the categories that support images and `[]` for the text-only ones.
An `input` array holds at most 2048 elements. Each element is classified — and billed — independently by a separate AWS call, so large arrays incur a proportional number of AWS calls.
**MCP / AI agent usage:** `image_url.url` accepts an HTTPS URL, data URI (`data:;base64,`), base64 string, or S3 URI — no binary upload needed.
```bash
curl -X POST "$BASE/v1/moderations" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": [
{"type": "text", "text": "Describe this image"},
{"type": "image_url", "image_url": {"url": "https://example.com/photo.png"}}
]
}'
```
## Billing
- **Guardrails** — AWS bills per text unit and per image processed by the ApplyGuardrail API, once for every policy the guardrail applies, each at that policy's own rate; see [Amazon Bedrock pricing](https://aws.amazon.com/bedrock/pricing/). No Bedrock model invocation is involved. The units AWS reports per policy appear in [usage logs and cost tracking](operations_logging_monitoring.md) as `text_units` and `input_images`, under one model per applied policy:
| Model | Policy |
|---|---|
| `amazon.bedrock-runtime-guardrail-content` | Content filters (`text_units`, and `input_images` for image content) |
| `amazon.bedrock-runtime-guardrail-topic` | Denied topics |
| `amazon.bedrock-runtime-guardrail-word` | Word filters |
| `amazon.bedrock-runtime-guardrail-sensitive-information` | Sensitive information (paid entity types) |
| `amazon.bedrock-runtime-guardrail-sensitive-information-free` | Sensitive information (free entity types) |
| `amazon.bedrock-runtime-guardrail-contextual-grounding` | Contextual grounding |
| `amazon.bedrock-runtime-guardrail-automated-reasoning` | Automated reasoning |
A guardrail applying several policies therefore produces several usage entries for one request, and its cost is their sum.
- **Guardrail checks** — AWS bills per text unit and per requested check; the gateway requests the `contentFilter` check only. Billed text units appear in [usage logs and cost tracking](operations_logging_monitoring.md) as `text_units` under the `amazon.bedrock-runtime-guardrail-checks` model.
- **Comprehend** — AWS bills toxicity detection per 100-character unit with a 3-unit minimum per call; see [Amazon Comprehend pricing](https://aws.amazon.com/comprehend/pricing/). Billed units appear in [usage logs and cost tracking](operations_logging_monitoring.md) as `comprehend_units` under the `amazon.comprehend-toxicity` model.
---
## Models
Source: https://stdapi.ai/md/api_openai_models.md
# Models API
Discover and list available Amazon Bedrock models across all configured regions through an OpenAI-compatible interface.
## Why Choose the Models API?
- :material-view-grid: __Complete Catalog__
Browse all available models across Amazon Bedrock regions. Chat, embeddings, images, and specialized AI services. See the [Models](models.md) page for a browsable table.
- :material-sync: __Always Up-to-Date__
Dynamic model discovery automatically shows new models as they become available in Amazon Bedrock.
- :material-map-marker-multiple: __Multi-Region Aggregation__
Combines models from all configured AWS regions in one deduplicated list; use the [Search Models API](api_search_models.md) to see per-region availability.
- :material-aws: __Comprehensive Coverage__
Includes Bedrock foundation models plus AWS AI services (Polly, Transcribe) in one unified API.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|----------|--------|--------------|------------|----------|
| `/v1/models` | `GET` | List all available models | Amazon Bedrock + AWS AI Services | `openai_model_list` |
| `/v1/models/{model_id}` | `GET` | Get details for a specific model | Amazon Bedrock + AWS AI Services | `openai_model_get` |
## OpenAI-Compatible with Amazon Bedrock Power
**Features:**
- **Multi-region aggregation**: Combines models from all configured Amazon Bedrock regions
- **Comprehensive catalog**: Includes Bedrock foundation models plus specialized models (Transcribe, Polly, etc.)
### What's Different from OpenAI?
- **Provider ownership**: `owned_by` field shows the model provider (e.g., `Amazon`, `Anthropic`, `Mistral AI`)
- **Model-specific capabilities**: Modalities and context windows vary by model—consult AWS documentation for specifics
!!! info "Created Date (`created`)"
The `created` field is a Unix timestamp (integer) representing the time at which the model was released. This value is sourced from the Amazon Bedrock model lifecycle metadata (`startOfLifeTime`). If the release date is not available from Amazon Bedrock, it defaults to `0` (Unix epoch, January 1, 1970).
## Try It Now
**List all available models:**
```bash
curl -X GET "$BASE/v1/models" \
-H "Authorization: Bearer $OPENAI_API_KEY"
```
**Get details for a specific model:**
```bash
curl -X GET "$BASE/v1/models/amazon.nova-micro-v1:0" \
-H "Authorization: Bearer $OPENAI_API_KEY"
```
**Response:**
```json
{
"id": "amazon.nova-micro-v1:0",
"object": "model",
"created": 1733212800,
"owned_by": "Amazon"
}
```
---
Browse foundation models for chat, embeddings, images, audio, and more.
---
## Messages
Source: https://stdapi.ai/md/api_anthropic_messages.md
# Messages API (Anthropic Compatible)
Generate conversational AI responses with Amazon Bedrock foundation models—including Claude, Nova, Llama, and more—through an Anthropic-compatible Messages API interface.
!!! warning "Route Prefix & Base URL"
By default, all Anthropic-compatible routes are prefixed with `/anthropic`. This means the Messages API is available at `/anthropic/v1/messages` instead of `/v1/messages`. You can customize this prefix using the `ANTHROPIC_ROUTES_PREFIX` configuration variable documented in [Operations Configuration](operations_configuration.md#anthropic-routes-prefix).
The `curl` examples below use a `$BASE` variable that **must include this prefix** — set it to your scheme and host followed by `ANTHROPIC_ROUTES_PREFIX`:
```bash
export BASE="https://your-host/anthropic" # :// + ANTHROPIC_ROUTES_PREFIX
```
## Why Choose the Messages API?
- :material-brain: __Multiple Models__
Access models from Anthropic, Amazon, Meta, and more through one API. Choose the best model for your task without vendor lock-in.
- :material-image-multiple: __Multi-Modal__
Process text, images, videos, and documents together. Support for URLs, data URIs, and direct S3 references.
- :material-shield-check: __Built-In Safety__
Bedrock Guardrails provide content filtering and safety policies.
- :material-aws: __AWS Scale & Reliability__
Run on AWS infrastructure with service tiers for optimized latency. Multi-region model access for availability and performance.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|-----------------------------|--------|--------------------------------------------|-------------------------------------------------------------------------------|----------------------------------|
| `/v1/messages` | `POST` | Conversational AI with multi-modal support | Bedrock Converse API · [Amazon Bedrock Mantle](features.md#bedrock-mantle-models) | `anthropic_message` |
| `/v1/messages/count_tokens` | `POST` | Count tokens in a message without sending | Bedrock CountTokens API · Bedrock Mantle | `anthropic_message_count_tokens` |
## Feature Compatibility
| Feature | Status | Notes |
|---------------------------------------|:----------------------------------------:|----------------------------------------------------------------------------------------------|
| **Messages & Roles** | | |
| Text messages | :material-check-circle:{ .success role="img" aria-label="Supported" } | Full support for all text content |
| Image input (`image`) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | HTTP URLs, data URIs, base64 |
| Document input (`document`) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | PDF (base64/URL), plain text, content blocks |
| Document citations | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Citation locations in responses (PDF only on some models) |
| Search result input (`search_result`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Pass search results as context |
| System messages | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | System prompts |
| Image & Document input from S3 | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | S3 URLs |
| Files API (`file_id`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Reference uploaded files in document/image sources — see [Files API](api_anthropic_files.md) |
| **Tool Calling** | | |
| Tool use (`tools`) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Full Anthropic-compatible schema |
| Tool choice (`auto`, `any`, `tool`) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Control tool selection behavior |
| Tool choice `none` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Gateway drops the tool config entirely, so the model behaves as if no tools were provided |
| Parallel tool calls | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Multiple tools in one turn |
| Web search tool (`web_search`) | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Available on models with system tool support (e.g., Amazon Nova 2) |
| Claude server tools | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Bash, text editor, computer use (Claude 3.5+), memory (Claude 3.7+) |
| **Generation Control** | | |
| `max_tokens` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Output length limits. Optional on this gateway (divergence from the Anthropic API, which requires it): the model's default output limit applies when omitted |
| `temperature` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Mapped to Bedrock inference params |
| `top_p` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Nucleus sampling control |
| `top_k` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Top-k sampling control |
| `stop_sequences` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Custom stop strings |
| Thinking | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | |
| Prompt caching | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Cache prompts to reduce costs and latency |
| Extra model-specific params | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Extra model-specific parameters not supported by the Anthropic API |
| **Streaming & Output** | | |
| Text | :material-check-circle:{ .success role="img" aria-label="Supported" } | Text messages |
| Streaming (`stream: true`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Server-Sent Events (SSE). Bedrock only reports usage in the trailing event, so `message_start.message.usage` is always `0`/`0`; read final counts from `message_delta.usage` instead |
| Thinking content | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Extended thinking output in content blocks |
| **Usage tracking** | | |
| Input text tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Billing unit |
| Output tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Billing unit |
| Cache creation tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Prompt caching metrics (streaming and non-streaming) |
| Cache read tokens | :material-check-circle:{ .success role="img" aria-label="Supported" } | Prompt caching metrics |
| Cache creation by TTL (`cache_creation`) | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Splits the cache creation tokens into `ephemeral_5m_input_tokens` and `ephemeral_1h_input_tokens`, when a per-TTL split is reported |
| Reasoning tokens (`output_tokens_details`) | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Reported only by models that return a reasoning-token count; `output_tokens` remains the billed total |
| Service tier (`usage.service_tier`) | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | `standard`, `priority` or `batch`. Omitted when the request ran on a tier the Anthropic API has no name for (`flex`, `reserved`) |
| **Other** | | |
| Refusal details (`stop_details`) | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Policy category and explanation behind `stop_reason: "refusal"`, when the model reports them |
| Metadata | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Converse path: logged only. Mantle path: `metadata.user_id` is forwarded upstream |
| Bedrock Guardrails | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Content safety policies |
| Service tiers | :material-check-circle:{ .success role="img" aria-label="Supported" } | Mapped to Bedrock service tiers and latency options |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with Anthropic API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Available on Select Models** — Check your model's capabilities
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond Anthropic API
## Model Support
All models supported by the Bedrock Converse and ConverseStream APIs are supported, plus every model served by [Bedrock Mantle](features.md#bedrock-mantle-models) when enabled — including OpenAI GPT-5.x, xAI Grok, and Google Gemma 4. Requests to Mantle models are passed through natively or converted automatically depending on the model's upstream API support — see [Bedrock Mantle](#bedrock-mantle) below.
### Bedrock Mantle
Mantle-only Claude models are passed through to the upstream Anthropic Messages API; other Mantle models are converted to an OpenAI shape (Responses or Chat Completions). Parameter fidelity differs per path:
| Parameter | Claude passthrough | Converted to an OpenAI shape |
|-----------|--------------------|------------------------------|
| Server tools (`web_search`, `code_execution`, `bash`, `text_editor`, `computer`, …) | Forwarded verbatim (`anthropic-beta` flags are **not** auto-injected on the Mantle path — pass them yourself) | Rejected with `400` |
| `thinking` | Forwarded | Dropped on conversion (use `output_config.effort` for portable reasoning control) |
| `thinking` response blocks | Returned as sent upstream | Not returned — a converted model's chain of thought is only available on the OpenAI-compatible APIs |
| `output_config.effort` | Forwarded | Mapped to reasoning effort |
| `output_config.format` | Fails upstream — not supported by the Mantle Messages API | `json_schema` mapped to OpenAI structured output |
| `top_k` | Forwarded | Dropped |
| `cache_control` markers | Forwarded (prompt caching preserved) | Dropped |
| `stop_sequences` | Forwarded | Dropped when served via the Responses API |
| `metadata.user_id` | Forwarded | Forwarded, SHA-256-hashed when over 64 characters |
| `service_tier` | Forwarded | Only `auto` is forwarded |
!!! note "Workspace attribution (`anthropic-workspace`)"
Mantle requests can be attributed to a Bedrock Workspace for cost tracking and observability with the `anthropic-workspace: ` header (a bare project ID such as `proj_abc123`, not an ARN). It is honored per-request only when [`AWS_BEDROCK_ALLOW_MANTLE_PROJECT_OVERRIDE`](operations_configuration.md#bedrock-allow-mantle-project-override) is `true`; otherwise the server default ([`AWS_BEDROCK_MANTLE_PROJECT`](operations_configuration.md#bedrock-mantle-project)) applies. This applies **only** to models served by the Bedrock Mantle endpoint — classic `bedrock-runtime` models ignore the header.
### { style="height: 1.2em; vertical-align: text-bottom;" } Claude Models Name Aliases
This API supports dynamic model name aliases matching the official Anthropic API. You can use Claude model names exactly as they appear in [Anthropic's documentation](https://platform.claude.com/docs/en/about-claude/models/overview), and they will be automatically resolved to the corresponding Bedrock model identifiers.
**Examples:**
- `claude-opus-5` → `anthropic.claude-opus-5`
- `claude-sonnet-5` → `anthropic.claude-sonnet-5`
- `claude-haiku-4-5-20251001` → `anthropic.claude-haiku-4-5-20251001-v1:0`
For Claude 4 and later, a date-stripped shortcut (e.g. `claude-haiku-4-5`) is also accepted and resolves to the latest dated variant.
Aliases for non-Anthropic models are also supported as normal.
## Advanced Features
### Prompt Caching
Reduce costs and improve response times by caching frequently-used prompt components across multiple requests. This feature is particularly effective for applications with consistent system prompts, tool definitions, or conversation contexts.
**Supported Models:**
- **Anthropic Claude**: Full support for system, messages, and tools caching
- **Amazon Nova**: Support for system and messages caching
!!! info "Documentation"
See [Amazon Bedrock Prompt Caching - Supported Models](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html#prompt-caching-models) for the complete list of models supporting prompt caching.
!!! warning "Cache Creation Costs"
Cache creation incurs a higher cost than regular token processing. Only use prompt caching when you expect a high cache hit ratio across multiple requests with similar prompts.
**How to Use:**
Add `cache_control` blocks to the content you want to cache:
```bash
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-fable-5",
"max_tokens": 1024,
"system": [
{
"type": "text",
"text": "You are a helpful assistant with extensive knowledge...",
"cache_control": {"type": "ephemeral"}
}
],
"messages": [
{"role": "user", "content": "What is 2 + 2?"}
]
}'
```
**Granular Cache Control:**
Enable caching for specific sections by adding `cache_control` blocks:
- **System messages**: Add to system text blocks
- **Messages**: Add to the last message content block you want cached
- **Tools**: Add to the last tool definition you want cached (Anthropic Claude only)
```json
{
"model": "anthropic.claude-fable-5",
"max_tokens": 1024,
"system": [
{
"type": "text",
"text": "System instructions...",
"cache_control": {"type": "ephemeral"}
}
],
"tools": [
{
"name": "get_weather",
"description": "Get weather data",
"input_schema": {...},
"cache_control": {"type": "ephemeral"}
}
],
"messages": [...]
}
```
**Benefits:**
- **Cost Reduction**: Cached tokens are billed at a lower rate than regular input tokens
- **Lower Latency**: Cached prompts eliminate reprocessing time
- **Automatic Management**: The API handles cache invalidation and updates
**Usage Tracking:**
Cached token usage is reported in the response:
```json
{
"usage": {
"input_tokens": 300,
"cache_creation_input_tokens": 1200,
"cache_creation": {
"ephemeral_5m_input_tokens": 1200,
"ephemeral_1h_input_tokens": 0
},
"cache_read_input_tokens": 0,
"output_tokens": 100
}
}
```
`cache_creation` splits the same total across the cache entry lifetimes, which are billed at different rates. It is present when a per-TTL split is reported.
In subsequent requests with cache hits:
```json
{
"usage": {
"input_tokens": 300,
"cache_creation_input_tokens": 0,
"cache_read_input_tokens": 1200,
"output_tokens": 100
}
}
```
### System Prompt
System prompts define the AI assistant's behavior, personality, and instructions (e.g., "You are a helpful assistant"). Most models support system prompts.
!!! warning "Unsupported Models"
Some models don't support system prompts (`mistral.mistral-7b-instruct-v0:2`, `mistral.mixtral-8x7b-instruct-v0:1`). By default, **stdapi.ai silently drops system messages** for these models, allowing cross-model compatibility. To receive errors instead, configure [`DROP_UNSUPPORTED_SYSTEM_PROMPT=false`](operations_configuration.md#drop-unsupported-system-prompt).
### :material-message-cog: Mid-Conversation System Messages
Beyond the top-level `system` parameter, a message may use the `system` role to steer the model from a specific point in the conversation onward.
```json
{
"model": "anthropic.claude-opus-5",
"max_tokens": 1024,
"system": "You are a helpful assistant.",
"messages": [
{"role": "user", "content": "Hello."},
{"role": "system", "content": "From now on, answer in one word."},
{"role": "assistant", "content": "Hi."},
{"role": "user", "content": "How are you?"}
]
}
```
**Handling:**
- **Claude 4.8+, Claude 5+ and later (Opus, Sonnet and Haiku alike), Fable and Mythos** (`claude-opus-4-8`, `claude-sonnet-4-9`, `claude-opus-5`, `claude-sonnet-5`, `claude-haiku-5`, `claude-fable-5`, `claude-mythos-*`): forwarded natively, so the directive applies from its position in the conversation.
- **Claude 4.7 and earlier (Opus, Sonnet and Haiku alike)** (`claude-opus-4-5`, `claude-sonnet-4-6`, `claude-haiku-4-7`, and every earlier generation down to Claude 2): the content is appended to the `system` prompt instead, keeping the same request working across every model.
!!! info "Placement"
A directive is forwarded natively only where the model accepts it: between a `user` turn and an `assistant` turn, as in the example above. Anywhere else — before the first turn, or right before the final `user` turn — it is appended to the `system` prompt instead, where it applies to the whole conversation. Requests therefore never fail because of placement.
### { style="height: 1.2em; vertical-align: text-bottom;" } S3 Image Support
Access images directly from your S3 buckets without generating pre-signed URLs or downloading files locally.
**Supported Formats:**
- **Images**: JPEG, PNG, GIF, WebP
**How to Use:**
Simply reference your S3 images using the `s3://` URI scheme in image source fields:
```json
{
"model": "anthropic.claude-fable-5",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image"},
{
"type": "image",
"source": {
"type": "url",
"url": "s3://my-bucket/images/photo.jpg"
}
}
]
}
]
}
```
!!! warning "IAM Permissions Required"
Your API service must have IAM permissions to read from the specified S3 buckets. S3 objects must be in the same AWS region as the executed model or accessible via your IAM role. Standard S3 data transfer and request costs apply.
**Benefits:**
- No pre-signed URLs - Direct S3 access without generating temporary URLs
- Security - Images stay in your AWS account with IAM-controlled access
- Performance - Optimized data transfer within AWS infrastructure
- Large images - No size limitations of data URIs or base64 encoding
### :material-file-link: Files API References (`file-id:`)
Image and document content blocks also accept the project-local `file-id:` URI scheme in their string-overloaded `source.url` and `source.data` fields, to reference a file previously uploaded via the [Anthropic Files API](api_anthropic_files.md):
```json
{
"type": "image",
"source": {
"type": "url",
"url": "file-id:file_0190c51c7de7455d9b8c2efe27dfbf67"
}
}
```
!!! info "When to use which path"
The Anthropic-native `{"type": "file", "file_id": "file_…"}` source (typed JSON) is unchanged and preferred for new code. The `file-id:` URI is the equivalent for the *string-overloaded* `source.url` / `source.data` fields, used alongside `s3://`, `https://`, and `data:` URIs. See [Files API → Referencing Uploaded Files](api_anthropic_files.md#referencing-uploaded-files-via-the-file-id-uri-scheme).
### Document Input
Send documents as context for the model to analyze and reference. Supports multiple source types:
- **Base64 PDF**: Inline PDF documents encoded in base64
- **URL PDF**: PDF documents fetched from HTTP(S) URLs (downloaded server-side)
- **Plain text**: Raw text content as documents
- **Content blocks**: Structured content with text and images
Enable `citations` on document blocks to get precise source references in responses:
```json
{
"type": "document",
"source": {
"type": "text",
"media_type": "text/plain",
"data": "The capital of France is Paris."
},
"title": "Geography",
"citations": {"enabled": true}
}
```
!!! note "Citation Support"
Citation support varies by model and document format. PDF documents generally have the best citation support across models.
### Server Tools
Server tools are built-in capabilities that foundation models can use directly without requiring you to implement backend integrations. Different model providers support different server tools through their native tool formats.
#### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Nova Tools
| Tool | Anthropic Tool Name | Amazon Nova 2 | Amazon Nova Premier (legacy) |
|------|---------------------|:-------------:|:-------------------:|
| Web Grounding | `web_search` | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } |
| Code Interpreter | `code_execution` | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } |
##### Web Grounding
The Anthropic `web_search` tool is supported on models that declare web search as a system tool. When you include a `web_search` tool in your request, it is automatically mapped to the model's native system tool (e.g., `nova_grounding` for Amazon Nova 2 models).
**Usage:**
```bash
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-2-lite-v1:0",
"messages": [
{"role": "user", "content": "What are the latest news today?"}
],
"tools": [
{"type": "web_search_20250305", "name": "web_search"}
]
}'
```
!!! warning "Region Compatibility"
Web grounding is only available in US Bedrock regions. To ensure all requests are routed to a US region, restrict the model using [`AWS_BEDROCK_MODEL_REGION_RESTRICT`](operations_configuration.md#bedrock-model-region-restrict):
```bash
export AWS_BEDROCK_MODEL_REGION_RESTRICT='{"amazon.nova-": ["us-east-1"]}'
```
**Limitations:**
- **No citation text in response blocks**: Unlike native Anthropic `web_search`, the `web_search_tool_result` content block carries only the `url` and `title` of each result — never `cited_text` or `encrypted_index`. The cited content itself is reflected only through the text content of the response.
- **No streaming citation data**: Citation information is not emitted in streaming events. The `server_tool_use` block is streamed as a start event with empty input — no citation delta is produced.
- **No search filtering on non-Claude models**: Amazon's `systemTool` grounding has no equivalent for `allowed_domains`, `blocked_domains`, `max_uses`, or `user_location`. Requests to a system-tool web search model (e.g. Amazon Nova 2) that set any of these fields are rejected with a `400 Bad Request` rather than silently running an unfiltered search. Anthropic Claude models forward these fields natively and are unaffected.
- **Not served on the OpenAI GPT models**: Amazon Bedrock serves their web search on the [Bedrock Mantle](features.md#bedrock-mantle-models) endpoint alone, and the Messages route reaches them through `bedrock-runtime`. A `web_search` tool sent to one is rejected with a `400 Bad Request` naming the two ways to route the model to Mantle, rather than reaching the model as a function tool it cannot answer.
!!! note "Model Compatibility"
Requesting `web_search` on a model that does not support it will return a `400 Bad Request` error. The OpenAI GPT-5.x family runs web search on the [Responses API](api_openai_responses.md#openai-gpt-web-search) instead: its results are reported as `web_search_call` items and `url_citation` annotations, which have no equivalent in an Anthropic `web_search_tool_result` block, so this endpoint rejects the tool rather than answering with a different shape.
##### Code Interpreter
Amazon Nova Code Interpreter enables models to securely execute Python code in isolated sandbox environments. Enable it by passing a `code_execution` tool, which is automatically mapped to the model's native `nova_code_interpreter` system tool.
!!! info "Learn More"
[Amazon Nova Built-in Tools - User Guide](https://docs.aws.amazon.com/nova/latest/nova2-userguide/using-tools.html#builtin-tools)
**Usage:**
```bash
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-2-lite-v1:0",
"messages": [
{"role": "user", "content": "Calculate the first 10 Fibonacci numbers."}
],
"tools": [
{"type": "code_execution_20250522", "name": "code_execution"}
]
}'
```
!!! note "Model Compatibility"
Requesting `code_execution` on a model that does not support it will return a `400 Bad Request` error.
#### { style="height: 1.2em; vertical-align: text-bottom;" } Anthropic Claude Server Tools
Anthropic Claude models support server-side tools that are executed by the model provider. Declare them in their native Anthropic JSON format.
**Supported Tools by Model:**
| Tool | Claude 3.5 Sonnet v2 | Claude 3.7 – 4.5 | Claude 4.6+ |
|------|:---------------------:|:----------------:|:-----------:|
| `bash` | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } |
| `text_editor` (`str_replace_based_edit_tool` or `str_replace_editor`) | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } |
| `computer` | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } |
| `memory` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } | :material-check-circle:{ .success role="img" aria-label="Supported" } |
On Claude 4.6 and later, a bare `computer` tool is promoted to the newer `computer_20251124` tool type — except on Claude Opus 5 and later, which support no computer-use tool version: there, `computer` is passed through as a regular custom tool instead of a server tool.
**Usage:**
```bash
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-sonnet-5",
"max_tokens": 4096,
"messages": [
{"role": "user", "content": "Run a Python script that prints hello world."}
],
"tools": [
{"type": "bash_20250124", "name": "bash"},
{"type": "text_editor_20250728", "name": "str_replace_based_edit_tool"}
]
}'
```
!!! tip "Beta Headers"
Claude server tools require specific `anthropic-beta` flags on Bedrock. On the classic Bedrock (Converse) path these flags are **automatically injected** when the corresponding server tools are included in the request — no manual header required (on the [Mantle](#bedrock-mantle) path they are not auto-injected; pass them yourself):
- `bash`, `text_editor` → `computer-use-2024-10-22` (Claude 3.5) or `computer-use-2025-01-24` (Claude 3.7+)
- `computer` → `computer-use-2024-10-22` (Claude 3.5), `computer-use-2025-01-24` (Claude 3.7 – 4.5), or `computer-use-2025-11-24` (Claude 4.6+, tool type `computer_20251124`)
- `memory` → `context-management-2025-06-27` (Claude 3.7+)
You can still pass additional `anthropic-beta` flags via the HTTP header or request body for non-tool beta features (e.g., `output-128k-2025-02-19`).
!!! note "Model Compatibility"
Requesting a server tool on a model that does not support it will return a `400 Bad Request` error. Non-Claude models do not support these tools.
##### Unsupported Anthropic Server Tools
The following Anthropic server tools are **not supported** via the classic Bedrock (Converse) path:
- `code_execution` — Code execution sandbox
- `web_search` — Web search (available on Amazon Nova models via `nova_grounding`; the OpenAI GPT-5.x family offers it on [`/v1/responses`](api_openai_responses.md#openai-gpt-web-search) only)
- `web_fetch` — Web page fetching
- `tool_search` — Tool search
- `container_upload` — Container file upload
Requests using these tools on Converse-served Claude models will return a `400 Bad Request` error. On [Mantle](#bedrock-mantle)-served Claude models (passthrough), server tools are instead forwarded verbatim to the upstream Messages API, which decides support; when a Mantle request must be converted to an OpenAI shape, server tools are rejected with `400`.
### Provider-Specific Parameters
Unlock advanced model capabilities by passing provider-specific parameters directly in your requests. These parameters are forwarded to Bedrock and allow you to access features unique to each foundation model provider.
!!! info "Documentation"
See [Bedrock Model Parameters](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html) for the complete list of available parameters per model.
**How It Works:**
Add provider-specific fields at the top level of your request body alongside standard Anthropic parameters. The API automatically forwards these to the appropriate model provider via Bedrock.
**Configuration Options:**
**Option 1: Per-Request**
Add provider-specific parameters directly in your request body.
**Option 2: Server-Wide Defaults**
Configure default parameters for specific models via the `DEFAULT_MODEL_PARAMS` environment variable:
```bash
export DEFAULT_MODEL_PARAMS='{
"anthropic.claude-sonnet-4-5-20250929-v1:0": {
"anthropic_beta": ["output-128k-2025-02-19"]
}
}'
```
!!! tip "Parameter Priority"
Per-request parameters override server-wide defaults.
**Behavior:**
- :material-check-circle:{ .success role="img" aria-label="Supported" } **Compatible parameters**: Forwarded to the model and applied
- :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported parameters**: Return HTTP 400 with an error message
- :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Reserved names**: `model_id` and `additional_request_fields` collide with the gateway's own request-building parameters and are rejected with a `400 invalid_request_error` naming the key, instead of being forwarded
#### { style="height: 1.2em; vertical-align: text-bottom;" } Anthropic Claude Features
Enable cutting-edge Claude capabilities including extended thinking and reasoning.
##### Extended Thinking
Enable extended thinking with the first-class `thinking` request parameter, just like the official Anthropic API — no beta header is required:
```bash
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-sonnet-5",
"max_tokens": 2048,
"thinking": {"type": "enabled", "budget_tokens": 1024},
"messages": [{"role":"user","content":"Solve a complex problem"}]
}'
```
`thinking` accepts `{"type": "enabled", "budget_tokens": }` (the budget must be less than `max_tokens`), `{"type": "disabled"}`, or `{"type": "adaptive"}`. Alternatively, control reasoning depth with `output_config.effort` (`low`, `medium`, `high`, `xhigh`, `max`).
On models whose reasoning depth is an effort level rather than a token budget (Amazon Nova 2, DeepSeek V3), `budget_tokens` turns reasoning on and the depth follows that model's own scale. Set `output_config.effort` to choose it.
!!! note "`display` Not Honored"
The `display` field (`summarized`/`omitted`) is accepted but has no effect: Bedrock's reasoning configuration has no equivalent, so full thinking text is always returned.
**Response with Thinking:**
When extended thinking is enabled, the response includes thinking content blocks:
```json
{
"id": "msg_abc123",
"type": "message",
"role": "assistant",
"content": [
{
"type": "thinking",
"thinking": "Let me think about this step by step..."
},
{
"type": "text",
"text": "Here's the solution..."
}
],
"usage": {...}
}
```
!!! tip "Server-Wide Configuration"
You can also configure default model parameters server-wide using the `DEFAULT_MODEL_PARAMS` environment variable (see [Provider-Specific Parameters](#provider-specific-parameters)).
!!! warning "Unsupported Beta Flags"
Unsupported flags that would change output return HTTP 400 errors.
!!! info "Documentation"
See [Using Claude on Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages-request-response.html) for more details on Claude-specific parameters.
## Available Request Headers
This endpoint supports standard Bedrock headers for enhanced control over your requests. All headers are optional and can be combined as needed.
### Content Safety (Guardrails)
| Header | Purpose | Valid Values |
|--------------------------------------|------------------------------------|---------------------------------------|
| `X-Amzn-Bedrock-GuardrailIdentifier` | Guardrail ID for content filtering | Your guardrail identifier |
| `X-Amzn-Bedrock-GuardrailVersion` | Guardrail version | Version number (e.g., `1`) |
| `X-Amzn-Bedrock-Trace` | Guardrail trace level | `disabled`, `enabled`, `enabled_full` |
### Performance Optimization
| Header | Purpose | Valid Values |
|--------------------------------------------|------------------------|-------------------------------|
| `X-Amzn-Bedrock-Service-Tier` | Service tier selection | `priority`, `default`, `flex` |
| `X-Amzn-Bedrock-PerformanceConfig-Latency` | Latency optimization | `standard`, `optimized` |
### Model-Specific Headers
| Header | Purpose | Valid Values | Models |
|------------------|--------------------------------|---------------------------------------------------------------------------------------------------|------------------|
| `anthropic-beta` | Enable Anthropic beta features | Comma-separated feature names (e.g., `computer-use-2025-01-24,context-management-2025-06-27`) | Anthropic Claude |
**Example with all headers:**
```bash
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-H "X-Amzn-Bedrock-GuardrailIdentifier: your-guardrail-id" \
-H "X-Amzn-Bedrock-GuardrailVersion: 1" \
-H "X-Amzn-Bedrock-Trace: enabled" \
-H "X-Amzn-Bedrock-Service-Tier: priority" \
-H "X-Amzn-Bedrock-PerformanceConfig-Latency: optimized" \
-d '{
"model": "anthropic.claude-fable-5",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
!!! info "Detailed Documentation"
For complete information about these headers, configuration options, and use cases, see:
- [Bedrock Guardrails Configuration](operations_configuration.md#bedrock-guardrails)
- [Service Tier and Performance Configuration](operations_configuration.md#bedrock-service-tier-and-performance-configuration)
## Model-Specific Features
### { style="height: 1.2em; vertical-align: text-bottom;" } TwelveLabs Pegasus
`twelvelabs.pegasus-1-2-v1:0` is a video-understanding model. Because Pegasus accepts exactly one video and one text prompt per call, this API adapts the conversation automatically:
- The **latest video** found anywhere in the conversation (any role, any position) is forwarded as the video input.
- The **latest contiguous run of user text** (back to the previous assistant or tool turn) is concatenated and forwarded as the text prompt.
- `temperature` and `max_tokens` are forwarded.
**Silently ignored** (no error): system prompts, tools, `top_p`, stop sequences, and prompt caching.
**Upstream format limitation:** The Anthropic Messages API does not define a `video` content block in its stable spec. To stay fully compatible with standard Anthropic clients, pass the video as an **`image`** content block with `media_type` set to the video MIME type (e.g. `video/mp4`) — the server detects the video MIME type automatically and routes it to Pegasus correctly.
**Video input formats**: `data:video/mp4;base64,…`, `https://…`, `s3://bucket/key`, or `file-id:…`. Videos above 18.75 MB are automatically uploaded to S3.
```bash
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "twelvelabs.pegasus-1-2-v1:0",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": [
{
"type": "image",
"source": {"type": "url", "url": "s3://my-bucket/video.mp4"}
},
{"type": "text", "text": "Describe what happens in this video."}
]
}
]
}'
```
## Try It Now
**Basic message:**
```bash
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Say hello world"}]
}'
```
**Streaming response:**
```bash
curl -N -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"max_tokens": 1024,
"stream": true,
"messages": [{"role": "user", "content": "Write a haiku about the sea."}]
}'
```
**Multi-modal with image:**
```json
{
"model": "amazon.nova-2-lite-v1:0",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image"},
{
"type": "image",
"source": {
"type": "url",
"url": "https://example.com/photo.jpg"
}
}
]
}
]
}
```
**With tool calling:**
```bash
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-fable-5",
"max_tokens": 1024,
"tools": [
{
"name": "get_weather",
"description": "Get weather information",
"input_schema": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "City name"}
},
"required": ["location"]
}
}
],
"messages": [
{"role": "user", "content": "What is the weather in Paris?"}
]
}'
```
**Count tokens (without sending a message):**
```bash
curl -X POST "$BASE/v1/messages/count_tokens" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-fable-5",
"messages": [{"role": "user", "content": "Hello, how are you?"}]
}'
```
**Response:**
```json
{"input_tokens": 13}
```
!!! info "Counted Request"
The count is computed on the exact request `anthropic_message` would send for the same body: `thinking`/`output_config.effort`, server tools in their model-native form, `cache_control` breakpoints, and mid-conversation system message placement are all taken into account.
---
**Ready to build with AI?** Check out the [Anthropic Models API](api_anthropic_models.md) to see all available foundation models, or the [Models Search API](api_search_models.md) for richer filtering.
---
## Message Batches
Source: https://stdapi.ai/md/api_anthropic_batches.md
# Message Batches API
Run a large set of Messages API requests asynchronously, at a lower price than the synchronous API, through the Anthropic Message Batches API shape.
The requests are sent inline, the batch runs without a connection held open, and its results are streamed back as JSONL — exactly the Anthropic workflow, so the official Anthropic SDKs work by changing the base URL.
## Why Choose the Message Batches API?
- :material-tag-arrow-down: __Lower Price per Token__
Batched requests are billed at the published batch rate, well below the on-demand rate for the same model.
- :material-swap-horizontal: __Drop-in Anthropic Compatibility__
`client.messages.batches.create(...)` and `.results(...)` work unchanged.
- :material-set-split: __Several Models, One Batch__
Each request names its own model, as upstream allows; the batch reports a single aggregate state.
- :material-cloud-lock: __Private AWS Backend__
Requests and results are stored in your own S3 buckets — no traffic to third-party endpoints.
## Available Endpoints
| Endpoint | Method | What It Does | MCP Tool |
|-------------------------------------------------|----------|-------------------------------------|-----------------------------------|
| `/anthropic/v1/messages/batches` | `POST` | Create a batch from inline requests | `anthropic_message_batch` |
| `/anthropic/v1/messages/batches` | `GET` | List batches, newest first | `anthropic_message_batch_list` |
| `/anthropic/v1/messages/batches/{id}` | `GET` | Retrieve a batch and its counters | `anthropic_message_batch_get` |
| `/anthropic/v1/messages/batches/{id}/results` | `GET` | Stream the results as JSONL | `anthropic_message_batch_results` |
| `/anthropic/v1/messages/batches/{id}/cancel` | `POST` | Cancel a batch that is still processing | `anthropic_message_batch_cancel` |
| `/anthropic/v1/messages/batches/{id}` | `DELETE` | Delete a batch that has ended | `anthropic_message_batch_delete` |
## Feature Compatibility
| Feature | Status | Notes |
|----------------------------------|:----------------------------------------:|---------------------------------------------------------------------------|
| **Creation** | | |
| `requests[].custom_id` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Up to 64 characters, unique within the batch |
| `requests[].params` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Same parameters as [Messages](api_anthropic_messages.md) |
| Several models in one batch | :material-check-circle:{ .success role="img" aria-label="Supported" } | Up to 8, each needing the 100-request minimum |
| `tools` / `tool_choice` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Refused when the batch is created — tool use is not available in a batch |
| Structured output schema | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Refused when the batch is created |
| `stream` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | A batch has nothing to stream to |
| `cache_control` | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Accepted and ignored — a batch reads and writes no prompt cache, and the request is answered without one |
| **Lifecycle** | | |
| Retrieve / poll | :material-check-circle:{ .success role="img" aria-label="Supported" } | `in_progress` → `canceling` → `ended` |
| Results (JSONL) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Streamed; available once `processing_status` is `ended` |
| Cancel | :material-check-circle:{ .success role="img" aria-label="Supported" } | Requests that already produced a Message keep it, the ones that never ran are reported `canceled`; cancelling twice, or cancelling a batch that has ended, changes nothing |
| Delete | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Only once the batch has ended — cancel it first, as upstream requires |
| List batches | :material-check-circle:{ .success role="img" aria-label="Supported" } | Newest first, with `before_id` / `after_id` cursors |
| `archived_at` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Results stay readable until the batch is deleted |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with Anthropic API
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
!!! note "`results_url` and Reverse Proxies"
`results_url` is an absolute URL on the address the request came in on, so `client.messages.batches.results(...)` works with no extra configuration and a client fetching it outside the SDK gets a URL it can dial as-is. Behind a reverse proxy it names the proxy's own origin, taken from the `Host` and `X-Forwarded-Proto` headers — enable [`ENABLE_PROXY_HEADERS`](operations_configuration.md#enable-proxy-headers) so the forwarded scheme is trusted, or a TLS-terminating proxy yields an `http://` URL.
!!! note "Content Guardrails and Batches"
A request that a [guardrail](operations_configuration.md#aws-bedrock-guardrail-identifier) would apply to is refused rather than run unguarded. Send those requests without batching.
!!! note "Prompt Caching and Batches"
Batched requests neither read nor write a prompt cache, on any model. A request carrying `cache_control` is still accepted and answered — the hint is dropped rather than the request — so a result reports no `cache_read_input_tokens` and no `cache_creation_input_tokens`. Nothing is lost by leaving it in: batched requests are already billed at the batch rate, and the cache discount was never available at that rate.
## Model Support
Any chat model available for batch inference in your configured Amazon Bedrock regions can be used — the same identifiers as [Messages](api_anthropic_messages.md). To shortlist them, call [`search_models`](api_search_models.md) with `route=anthropic_message&batch=true`; each entry also carries a `batch` field.
!!! warning "The shortlist is a hint, not a rule"
`batch` is advertised on a best-effort basis and never used to reject a request. A model it does not advertise — or says nothing about — may still run a batch, so submit the batch rather than ruling the model out; the answer you get back is the authoritative one.
A model that cannot serve batched requests is refused when the batch is created, naming the model; no sibling job is left running. A model this deployment normally serves through another Amazon Bedrock endpoint is batched under the identifier the batch endpoint knows it by, so it needs nothing from you.
## Workflow
### 1. Create the batch
```python
from anthropic import Anthropic
client = Anthropic(base_url="https://your-host/anthropic", api_key="...")
batch = client.messages.batches.create(
requests=[
{
"custom_id": f"req-{index}",
"params": {
"model": "amazon.nova-micro-v1:0",
"max_tokens": 256,
"messages": [{"role": "user", "content": text}],
},
}
for index, text in enumerate(documents)
]
)
```
**Example request (curl):**
```bash
curl -X POST "https://your-host/anthropic/v1/messages/batches" \
-H "x-api-key: $API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"requests": [
{
"custom_id": "req-1",
"params": {
"model": "amazon.nova-micro-v1:0",
"max_tokens": 256,
"messages": [{"role": "user", "content": "Summarize: ..."}]
}
}
]
}'
```
The samples above are abridged: a real batch needs at least 100 requests for each model it names.
**Example response:**
```json
{
"id": "msgbatch_06fvfg3lbdqarbad8kbo55g0sg5h3s4a",
"type": "message_batch",
"processing_status": "in_progress",
"request_counts": {"processing": 100, "succeeded": 0, "errored": 0, "canceled": 0, "expired": 0},
"created_at": "2026-08-12T20:53:33Z",
"expires_at": "2026-08-13T20:53:33Z"
}
```
### 2. Poll until processing ends
```python
batch = client.messages.batches.retrieve(batch.id)
print(batch.processing_status, batch.request_counts)
```
`succeeded` and `errored` move as the batch runs, so they can drive a progress bar. `canceled` and `expired` are known only once `processing_status` is `ended`.
### 3. Read the results
```python
for entry in client.messages.batches.results(batch.id):
if entry.result.type == "succeeded":
print(entry.custom_id, entry.result.message.content[0].text)
```
Each line pairs a `custom_id` with its outcome:
```json
{"custom_id": "req-1", "result": {"type": "succeeded", "message": {"id": "msg_req-1", "type": "message", "role": "assistant", "content": [{"type": "text", "text": "..."}], "stop_reason": "end_turn", "usage": {"input_tokens": 22, "output_tokens": 9}}}}
{"custom_id": "req-2", "result": {"type": "errored", "error": {"type": "error", "error": {"type": "invalid_request_error", "message": "..."}}}}
```
!!! warning "Results Are Not in Request Order"
Result lines may come back in any order, as upstream also warns. Match a result to its request with `custom_id`, never with the line number.
## Limits
| Limit | Value |
|------------------------------|--------------------------|
| Minimum requests per model | 100 (default quota) |
| Maximum requests per batch | 100,000 |
| Distinct models per batch | 8 |
| Processing window | 24 hours from creation |
A batch below the minimum, or over the model cap, is refused when it is created and the message names the shortfall — a batch naming several models must reach the minimum **for each of them**.
!!! note "The 100-request minimum is a quota default"
100 is the default of the Amazon Bedrock quota *Minimum number of records per batch inference job*, which is set **per model** and adjustable for some of them — see [Amazon Bedrock quotas](https://docs.aws.amazon.com/general/latest/gr/bedrock.html). The gateway checks against that default, not against your account's own value, so a raised quota is enforced by Amazon Bedrock rather than here — a model given 150 requests clears this check and is then refused by the backend — and a lowered one is not usable: fewer than 100 requests for a model is still refused here.
## Prerequisites
The Message Batches API is disabled until the deployment declares an AWS IAM service role that Amazon Bedrock assumes to read the requests and write the results:
- [`AWS_BEDROCK_BATCH_ROLE_ARN`](operations_configuration.md#aws-bedrock-batch-role-arn) — the service role.
- [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) — the bucket holding the batch data.
- [`AWS_S3_BATCHES_PREFIX`](operations_configuration.md#aws-s3-batches-prefix) — the prefix it is stored under.
The permissions the role and the server need are listed in [IAM Permissions](operations_iam_permissions.md#batch-inference).
While the role is unset, every batch endpoint answers `529`.
## Billing
Batched requests are billed at the published batch rate for the model, roughly half the on-demand rate. Usage is recorded once, when the batch ends. See [Cost Management](operations_cost_management.md#batch-inference).
## See Also
- [Messages API](api_anthropic_messages.md) — the per-request parameters
- [Batch API](api_openai_batches.md) — the OpenAI-shaped equivalent
- [Configuration](operations_configuration.md#aws-bedrock-batch-role-arn) — enabling batches
---
## Files
Source: https://stdapi.ai/md/api_anthropic_files.md
# Files API (Anthropic Compatible)
!!! warning "Route Prefix & Base URL"
By default, all Anthropic-compatible routes are prefixed with `/anthropic`. This means the Files API is available at `/anthropic/v1/files` instead of `/v1/files`. You can customize this prefix using the `ANTHROPIC_ROUTES_PREFIX` configuration variable documented in [Operations Configuration](operations_configuration.md#anthropic-routes-prefix).
The `curl` examples below use a `$BASE` variable that **must include this prefix** — set it to your scheme and host followed by `ANTHROPIC_ROUTES_PREFIX`:
```bash
export BASE="https://your-host/anthropic" # :// + ANTHROPIC_ROUTES_PREFIX
```
`ANTHROPIC_ROUTES_PREFIX` must always be a non-empty path and is validated at startup to differ from `OPENAI_ROUTES_PREFIX` (the server refuses to start otherwise). This Anthropic-compatible Files API is therefore always served on its own path, distinct from the [OpenAI-compatible Files API](api_openai_files.md).
Upload and manage files via an Anthropic-compatible interface. Files are stored in Amazon S3 and can be referenced directly in Messages requests as document or image sources.
## Why Choose the Files API?
- :material-upload: __Simple Upload__
Upload any file with a single `multipart/form-data` request. Files are immediately available for use in inference.
- :material-swap-vertical: __Bidirectional Pagination__
Traverse your file list in both directions using `after_id` and `before_id` cursors, matching the official Files API pagination.
- :material-file-document-multiple: __Messages Integration__
Reference uploaded files directly in Messages requests as document or image source blocks using `"type": "file"`.
- :material-download: __Content Download__
Download raw file bytes at any time via the `/content` endpoint.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|-------------------------------|----------|----------------------------|------------|--------------------------|
| `/v1/files` | `POST` | Upload a file | Amazon S3 | `anthropic_file` |
| `/v1/files` | `GET` | List files with pagination | Amazon S3 | `anthropic_file_list` |
| `/v1/files/{file_id}` | `GET` | Retrieve file metadata | Amazon S3 | `anthropic_files_get` |
| `/v1/files/{file_id}` | `DELETE` | Delete a file | Amazon S3 | `anthropic_files_delete` |
| `/v1/files/{file_id}/content` | `GET` | Download raw file bytes | Amazon S3 | `anthropic_file_content` |
## Feature Compatibility
| Feature | Status | Notes |
|--------------------------|:----------------------------------------:|------------------------------------------------------------------|
| **Upload** | | |
| `file` (multipart) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Required binary form field |
| `file` (JSON body) | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Base64, data URI, HTTPS URL, or S3 URI — for MCP / AI agents |
| **Listing** | | |
| Listing order | :material-check-circle:{ .success role="img" aria-label="Supported" } | Most recently created first, by `created_at` |
| `after_id` cursor | :material-check-circle:{ .success role="img" aria-label="Supported" } | Forward cursor: returns files older than the given ID |
| `before_id` cursor | :material-check-circle:{ .success role="img" aria-label="Supported" } | Backward cursor: returns files newer than the given ID |
| `limit` | :material-check-circle:{ .success role="img" aria-label="Supported" } | 1 – 1 000; default 20 |
| **File size cap** | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | No artificial limit; S3 object limit (~5 TB) |
| **Messages integration** | :material-check-circle:{ .success role="img" aria-label="Supported" } | `"source": {"type": "file", "file_id": "..."}` in document/image |
| `downloadable` field | :material-minus-circle:{ .partial role="img" aria-label="Partial" } | Always `true`; spec default is `false` for user-uploaded files |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with Anthropic API
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Implemented with minor deviations from spec
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond Anthropic API
## Quick Start
!!! info "Optional Anthropic headers"
The `anthropic-version` and `anthropic-beta: files-api-2025-04-14` headers sent by the official Anthropic SDKs are accepted and ignored by this gateway — the Files API works without them. The examples below include them only for parity with SDK-generated requests.
### Upload a File
```bash
curl -X POST "$BASE/v1/files" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14" \
-F "file=@document.pdf;type=application/pdf"
```
**Response:**
```json
{
"id": "file_0190c51c7de7455d9b8c2efe27dfbf67",
"type": "file",
"filename": "document.pdf",
"mime_type": "application/pdf",
"size_bytes": 102400,
"created_at": "2025-04-15T12:00:00Z",
"downloadable": true
}
```
### Upload via JSON Body (MCP and AI Agents)
When using MCP tools or HTTP clients that cannot construct `multipart/form-data` requests, pass the file as a base64 string, data URI, HTTPS URL, or S3 URI in a JSON body instead.
**Data URI (inline content):**
```bash
curl -X POST "$BASE/v1/files" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14" \
-H "Content-Type: application/json" \
-d '{"file": "data:text/plain;base64,SGVsbG8gV29ybGQ="}'
```
**HTTPS URL (server fetches the file):**
```bash
curl -X POST "$BASE/v1/files" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14" \
-H "Content-Type: application/json" \
-d '{"file": "https://example.com/document.pdf"}'
```
All variants return the same `FileMetadata` response as a multipart upload.
### Retrieve Metadata
```bash
curl "$BASE/v1/files/file_0190c51c7de7455d9b8c2efe27dfbf67" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14"
```
### List Files
```bash
# Default (newest first, up to 20 files)
curl "$BASE/v1/files" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14"
# Forward pagination: the page following a given ID (older files)
curl "$BASE/v1/files?after_id=file_0190c51c7de7455d9b8c2efe27dfbf67&limit=20" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14"
# Backward pagination: the page preceding a given ID (newer files)
curl "$BASE/v1/files?before_id=file_0190c51c7de7455d9b8c2efe27dfbf67" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14"
```
### Download Content
```bash
curl "$BASE/v1/files/file_0190c51c7de7455d9b8c2efe27dfbf67/content" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14" \
-o downloaded.pdf
```
### Delete a File
```bash
curl -X DELETE "$BASE/v1/files/file_0190c51c7de7455d9b8c2efe27dfbf67" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14"
```
**Response:**
```json
{
"id": "file_0190c51c7de7455d9b8c2efe27dfbf67",
"type": "file_deleted"
}
```
## Messages Integration
Reference an uploaded file inside a `POST /v1/messages` request as a document or image source:
**Document (PDF or other supported format):**
```bash
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-haiku-4-5-20251001-v1:0",
"max_tokens": 512,
"messages": [
{
"role": "user",
"content": [
{
"type": "document",
"source": {
"type": "file",
"file_id": "file_0190c51c7de7455d9b8c2efe27dfbf67"
}
},
{
"type": "text",
"text": "Summarize this document."
}
]
}
]
}'
```
**Image:**
```bash
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-haiku-4-5-20251001-v1:0",
"max_tokens": 256,
"messages": [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "file",
"file_id": "file_0190c51c7de7455d9b8c2efe27dfbf68"
}
},
{
"type": "text",
"text": "Describe this image."
}
]
}
]
}'
```
## End-to-End Example
```bash
# 1. Upload a file
FILE_ID=$(curl -s -X POST "$BASE/v1/files" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14" \
-F "file=@document.pdf;type=application/pdf" | jq -r .id)
echo "Uploaded: $FILE_ID"
# 2. Reference in a message
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14" \
-H "Content-Type: application/json" \
-d "{
\"model\": \"anthropic.claude-haiku-4-5-20251001-v1:0\",
\"max_tokens\": 512,
\"messages\": [{
\"role\": \"user\",
\"content\": [
{\"type\": \"document\", \"source\": {\"type\": \"file\", \"file_id\": \"$FILE_ID\"}},
{\"type\": \"text\", \"text\": \"What is the key finding in this document?\"}
]
}]
}"
# 3. Cleanup
curl -X DELETE "$BASE/v1/files/$FILE_ID" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14"
```
## Referencing Uploaded Files via the `file-id:` URI Scheme
The native `{"type": "file", "file_id": "..."}` source shown above is the Anthropic-compatible way to reference an uploaded file in Messages content blocks. For **string-overloaded** file fields that already accept URI schemes like `s3://`, `https://`, or `data:` — for example image and document content blocks with `source.type` `url` or `base64` — this implementation defines an additional project-local URI scheme:
```text
file-id:
```
!!! tip "Project-local URI scheme — `file-id:`"
`file-id:` is an **extension beyond the original Anthropic API**, parallel to the existing `s3://`, `https://`, and `data:` schemes already accepted on the same fields. It lets a client upload a file once and reuse it across Messages content blocks (image / document `source.url` and `source.data`), as well as the OpenAI-compatible routes — without re-uploading.
* **Where accepted:** any string-overloaded file field. For Anthropic Messages: image and document content blocks where `source.type` is `url` (with `source.url: "file-id:"`) or `base64` (with `source.data: "file-id:"`).
* **Where unchanged:** the typed `{"type": "file", "file_id": "..."}` source already accepted by the Anthropic API stays exactly as-is — do not wrap those bare IDs in `file-id:`.
* **Where rejected:** the Files API ingest endpoint (`POST /v1/files`) returns **400** for `file-id:` inputs, because resolving it there would silently clone an existing file.
* **Detection:** match is **case-sensitive** (`file-id:`, lowercase) with no whitespace stripping; the payload after the prefix must be a valid Files API ID, otherwise the request fails with `400 invalid_request_error`. A missing or expired file returns `404 not_found`.
### Worked Example — Send an Uploaded Image in Messages
```bash
# 1. Upload the file once.
FILE_ID=$(curl -s -X POST "$BASE/v1/files" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-beta: files-api-2025-04-14" \
-F "file=@chart.png;type=image/png" | jq -r .id)
# 2. Reference it via file-id: in a Messages request.
curl -X POST "$BASE/v1/messages" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"model\": \"anthropic.claude-haiku-4-5-20251001-v1:0\",
\"max_tokens\": 256,
\"messages\": [{
\"role\": \"user\",
\"content\": [
{\"type\": \"text\", \"text\": \"Describe this chart.\"},
{\"type\": \"image\", \"source\": {\"type\": \"url\", \"url\": \"file-id:${FILE_ID}\"}}
]
}]
}"
```
See the [OpenAI Files API documentation](api_openai_files.md#referencing-uploaded-files-via-the-file-id-uri-scheme) for the full list of supported routes — the same scheme works identically across both API surfaces.
## Errors
| HTTP | Cause |
|------|-------------------------------------------------------------------------------------|
| 400 | Invalid filename characters |
| 400 | `file-id:` URI passed to the upload endpoint (`POST /v1/files`) |
| 400 | Malformed ID after the `file-id:` prefix in a Messages content block |
| 404 | File not found or already deleted |
| 503 | `AWS_S3_BUCKET` is not configured |
## Configuration
Files are stored in S3 under the prefix configured by [`AWS_S3_FILES_PREFIX`](operations_configuration.md#aws-s3-files-prefix) (default: `files/`). All file IDs are shared across the OpenAI and Anthropic endpoints — a file uploaded via one API can be downloaded or deleted via the other.
---
**Use files across multiple requests without re-uploading.** See [OpenAI Files API](api_openai_files.md) for the OpenAI-compatible equivalent, including `expires_after` support.
---
## Models
Source: https://stdapi.ai/md/api_anthropic_models.md
# Models API (Anthropic Compatible)
Discover and list available Amazon Bedrock text models through an Anthropic-compatible interface.
!!! warning "Route Prefix & Base URL"
By default, all Anthropic-compatible routes are prefixed with `/anthropic`. This means the Models API is available at `/anthropic/v1/models` instead of `/v1/models`. You can customize this prefix using the `ANTHROPIC_ROUTES_PREFIX` configuration variable documented in [Operations Configuration](operations_configuration.md#anthropic-routes-prefix).
The `curl` examples below use a `$BASE` variable that **must include this prefix** — set it to your scheme and host followed by `ANTHROPIC_ROUTES_PREFIX`:
```bash
export BASE="https://your-host/anthropic" # :// + ANTHROPIC_ROUTES_PREFIX
```
**Note:** `ANTHROPIC_ROUTES_PREFIX` must always be a non-empty path and is validated at startup to differ from `OPENAI_ROUTES_PREFIX` (the server refuses to start otherwise). This Anthropic-compatible Models API is therefore always served on its own path, distinct from the [OpenAI-compatible Models API](api_openai_models.md).
## Why Choose the Models API?
- :material-view-grid: __Text Model Catalog__
Browse the available Bedrock text models across regions. See the [Models](models.md) page for a browsable table.
- :material-sync: __Always Up-to-Date__
Dynamic model discovery automatically shows new models as they become available in Bedrock.
- :material-map-marker-multiple: __Multi-Region Aggregation__
Combines models from all configured AWS regions in one list. See which models are available in each region.
- :material-aws: __Foundation Models__
Includes Claude, Nova, Llama, and other Bedrock foundation text models.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|-------------------------|--------|----------------------------------|----------------|------------------------|
| `/v1/models` | `GET` | List all available text models | Amazon Bedrock | `anthropic_model_list` |
| `/v1/models/{model_id}` | `GET` | Get details for a specific model | Amazon Bedrock | `anthropic_model_get` |
## Anthropic-Compatible with Amazon Bedrock Power
**Features:**
- **Multi-region aggregation**: Combines models from all configured Bedrock regions
- **Cursor-based pagination**: Use `limit`, `after_id`, and `before_id` query parameters
- **Text models only**: Returns only models with text input and text output modalities (Claude, Nova, Llama, etc.)
### What's Different from Anthropic?
- **Model IDs**: Uses Bedrock model identifiers (e.g., `anthropic.claude-haiku-4-5-20251001-v1:0`) instead of Anthropic model names
- **Extended catalog**: Includes all Bedrock text models (Claude, Nova, Llama, etc.), not just Anthropic models
- **Default page size**: `limit` defaults to `1000` (the Anthropic API defaults to `20`)
!!! info "Created Date (`created_at`)"
The `created_at` field is an RFC 3339 datetime string representing the time at which the model was released. This value is sourced from the Bedrock model lifecycle metadata (`startOfLifeTime`). If the release date is not available from Bedrock, it defaults to the Unix epoch (`"1970-01-01T00:00:00Z"`).
## Try It Now
**List all available models:**
```bash
curl -X GET "$BASE/v1/models" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01"
```
**List models with pagination:**
```bash
curl -X GET "$BASE/v1/models?limit=10" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01"
```
The `limit` query parameter accepts values from `1` to `1000` and defaults to `1000`.
**Get details for a specific model:**
```bash
curl -X GET "$BASE/v1/models/amazon.nova-micro-v1:0" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01"
```
---
Browse Bedrock foundation models for chat and completion tasks.
---
## Rerank
Source: https://stdapi.ai/md/api_cohere_rerank.md
# Rerank API (Cohere Compatible)
Rank documents by semantic relevance to a query with Amazon Bedrock rerank models through a Cohere-compatible interface.
!!! warning "Route Prefix & Base URL"
By default, all Cohere-compatible routes are prefixed with `/cohere`. This means the Rerank API is available at `/cohere/v2/rerank` instead of `/v2/rerank`. You can customize this prefix using the `COHERE_ROUTES_PREFIX` configuration variable documented in [Operations Configuration](operations_configuration.md#cohere-routes-prefix).
The `curl` examples below use a `$BASE` variable that **must include this prefix** — set it to your scheme and host followed by `COHERE_ROUTES_PREFIX`:
```bash
export BASE="https://your-host/cohere" # :// + COHERE_ROUTES_PREFIX
```
## Why Choose the Rerank API?
- :material-sort: __Better Search Relevance__
Re-order candidate documents by true semantic relevance to the query. A precise second stage after vector or keyword search.
- :material-book-open-page-variant: __Higher RAG Quality__
Feed your LLM only the most relevant passages. Reranking reduces context noise and improves answer accuracy.
- :material-swap-horizontal: __Drop-in Cohere Compatibility__
Follows the Cohere v2 Rerank API shape. Existing Cohere rerank integrations work by changing the base URL.
- :material-cloud-lock: __Private AWS Backend__
Served entirely by Bedrock rerank models in your own AWS account — no traffic to third-party endpoints.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|--------------|--------|-----------------------------------------------------|-----------------------|--------------------|
| `/v2/rerank` | `POST` | Rank documents by semantic relevance to a query | Bedrock rerank models | `cohere_rerank` |
| `/v1/rerank` | `POST` | Legacy v1 rerank for older SDKs and integrations | Bedrock rerank models | `cohere_rerank_v1` |
**Example request:**
```bash
curl -X POST "$BASE/v2/rerank" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "cohere.rerank-v3-5:0",
"query": "What is the capital of the United States?",
"documents": [
"Carson City is the capital city of Nevada.",
"Washington, D.C. is the capital of the United States.",
"Capital punishment has existed in the United States since colonial times."
],
"top_n": 2
}'
```
**Example response:**
```json
{
"id": "0f1b3c6e8d9a4b5c8e7f6a5b4c3d2e1f",
"results": [
{"index": 1, "relevance_score": 0.9871},
{"index": 2, "relevance_score": 0.3251}
],
"meta": {
"api_version": {"version": "2"},
"billed_units": {"search_units": 1}
}
}
```
## Feature Compatibility
| Feature | Status | Notes |
|-----------------------------|:----------------------------------------:|-------------------------------------------------------------------|
| **Input** | | |
| `query` + `documents` (strings) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Full support |
| `top_n` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Limits the number of returned results |
| `max_tokens_per_doc` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Forwarded to the model; support depends on the model |
| `priority` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored — request scheduling priority is not applicable on Bedrock |
| `return_documents` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored — v2 results reference input documents by `index` |
| Extra model-specific params | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Extra fields are forwarded as additional model request parameters |
| **Output** | | |
| `results` (index + score) | :material-check-circle:{ .success role="img" aria-label="Supported" } | Ordered by decreasing relevance |
| `meta.billed_units` | :material-check-circle:{ .success role="img" aria-label="Supported" } | One search unit per started batch of 100 documents |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with the Cohere API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Available on Select Models** — Check your model's capabilities
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond the Cohere API
## Model Support
Any rerank model available in your configured Bedrock regions can be used, for example:
### { style="height: 1.2em; vertical-align: text-bottom;" } Cohere Models
| Model | Model ID | Cohere Name | Notes |
|-------------------|------------------------|---------------|-------------------------------------------|
| Cohere Rerank 3.5 | `cohere.rerank-v3-5:0` | `rerank-v3.5` | Multilingual, state-of-the-art relevance |
!!! tip "Cohere's Own Model Names Resolve As They Stand"
Each Cohere model is published under the name [Cohere's API](https://docs.cohere.com/docs/models) uses as well as its Bedrock ID, derived from the ID rather than curated by hand, so an application already calling Cohere changes only its base URL. Both forms reach the same model. The Amazon rerank model has no such name, having no upstream Cohere API to stay compatible with.
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Models
| Model | Model ID | Notes |
|-------------------|----------------------|----------------------------------------------------------------|
| Amazon Rerank 1.0 | `amazon.rerank-v1:0` | Not available in every region (e.g. absent from `us-east-1`) |
**Find compatible models:** Call [`/search_models`](api_search_models.md) with `route=cohere_rerank` to discover model IDs that support reranking in your deployment.
## Cohere v1 Rerank API (Legacy)
The legacy `/v1/rerank` endpoint is also available for older Cohere SDKs (`cohere.Client`) and third-party integrations that predate the v2 API. It shares the same Bedrock backend and model support as `/v2/rerank`; new clients should prefer the v2 endpoint.
**Differences from the v2 endpoint:**
| Feature | Status | Notes |
|-----------------------------|:----------------------------------------:|-------------------------------------------------------------------|
| `documents` as objects | :material-check-circle:{ .success role="img" aria-label="Supported" } | Each document is a string, or a field->value object |
| `return_documents` | :material-check-circle:{ .success role="img" aria-label="Supported" } | When `true`, each result echoes back the document text |
| `rank_fields` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Ranks object documents on the selected fields only |
| `max_chunks_per_doc` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Rejected with 400 — no Bedrock equivalent |
| `meta.api_version.version` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Reported as `"1"` |
Object documents with a single `text` field use the same plain-text encoding as a string document. Multi-field objects (or object documents combined with a non-default `rank_fields`) are sent to Bedrock as structured JSON documents, natively reranked on their fields. When `rank_fields` is set, object documents are first reduced to the listed fields (missing fields are skipped) before being sent; `return_documents` always echoes back the *original*, unreduced document.
!!! note "Echoed text for multi-field documents"
The Cohere v1 response type only carries a single `text` string per echoed document. For multi-field object documents, this implementation joins the original fields as `key: value` lines (one per field) — an approximation, since Cohere's own algorithm for this case is not publicly documented.
**Example request:**
```bash
curl -X POST "$BASE/v1/rerank" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "cohere.rerank-v3-5:0",
"query": "What is the capital of the United States?",
"documents": [
{"title": "Nevada", "text": "Carson City is the capital city of Nevada."},
{"title": "United States", "text": "Washington, D.C. is the capital of the United States."}
],
"rank_fields": ["title", "text"],
"top_n": 1,
"return_documents": true
}'
```
**Example response:**
```json
{
"id": "0f1b3c6e8d9a4b5c8e7f6a5b4c3d2e1f",
"results": [
{
"document": {"text": "title: United States\ntext: Washington, D.C. is the capital of the United States."},
"index": 1,
"relevance_score": 0.9871
}
],
"meta": {
"api_version": {"version": "1"},
"billed_units": {"search_units": 1}
}
}
```
## How It Works
Requests are served by the [Amazon Bedrock Rerank API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent-runtime_Rerank.html), with automatic multi-region routing and failover across the regions where the selected model is available.
!!! note "Required IAM Permission"
The Rerank API requires the `bedrock:Rerank` IAM action in addition to `bedrock:InvokeModel`. See [IAM Permissions](operations_configuration.md#iam-permissions).
## Billing
AWS bills reranking per **search unit**: one search unit covers a single query with up to 100 documents. A request with more than 100 documents is billed one additional search unit per started batch of 100. Search units appear in [usage logs and cost tracking](operations_logging_monitoring.md) as `search_units`.
---
## Embed
Source: https://stdapi.ai/md/api_cohere_embed.md
# Embed API (Cohere Compatible)
Generate vector embeddings for semantic search and RAG applications with Amazon Bedrock embedding models through a Cohere-compatible interface.
This is an alternate route to the [OpenAI-compatible Embeddings API](api_openai_embeddings.md): both are served by the same embedding backends and models, so anything supported there is supported here.
!!! warning "Route Prefix & Base URL"
By default, all Cohere-compatible routes are prefixed with `/cohere`. This means the Embed API is available at `/cohere/v2/embed` instead of `/v2/embed`. You can customize this prefix using the `COHERE_ROUTES_PREFIX` configuration variable documented in [Operations Configuration](operations_configuration.md#cohere-routes-prefix).
The `curl` examples below use a `$BASE` variable that **must include this prefix** — set it to your scheme and host followed by `COHERE_ROUTES_PREFIX`:
```bash
export BASE="https://your-host/cohere" # :// + COHERE_ROUTES_PREFIX
```
## Why Choose the Embed API?
- :material-magnify: __Semantic Search__
Turn texts and images into dense vectors for similarity search that understands meaning, not just keywords.
- :material-book-open-page-variant: __Higher RAG Quality__
Build retrieval pipelines on high-quality embeddings, with `input_type` tuning for queries versus documents.
- :material-swap-horizontal: __Drop-in Cohere Compatibility__
Follows the Cohere v2 Embed API shape. Existing Cohere embed integrations work by changing the base URL.
- :material-cloud-lock: __Private AWS Backend__
Served entirely by Bedrock embedding models in your own AWS account — no traffic to third-party endpoints.
## Available Endpoints
| Endpoint | Method | What It Does | Powered By | MCP Tool |
|-------------|--------|--------------------------------------------------------|--------------------------|-------------------|
| `/v2/embed` | `POST` | Transform texts and images into semantic float vectors | Bedrock embedding models | `cohere_embed` |
| `/v1/embed` | `POST` | Legacy v1 embed for older SDKs and integrations | Bedrock embedding models | `cohere_embed_v1` |
**Example request:**
```bash
curl -X POST "$BASE/v2/embed" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "cohere.embed-multilingual-v3",
"input_type": "search_document",
"texts": ["Hello world", "Bonjour le monde"]
}'
```
**Example response:**
```json
{
"response_type": "embeddings_by_type",
"id": "0f1b3c6e8d9a4b5c8e7f6a5b4c3d2e1f",
"embeddings": {"float": [[0.012, -0.034, ...], [0.041, 0.007, ...]]},
"texts": ["Hello world", "Bonjour le monde"],
"meta": {
"api_version": {"version": "2"},
"billed_units": {"input_tokens": 8}
}
}
```
**Find compatible models:** Call [`/search_models`](api_search_models.md) with `route=cohere_embed` to discover model IDs that support embeddings — every Bedrock embedding model works, not just Cohere ones.
## Feature Compatibility
| Feature | Status | Notes |
|-------------------------------|:----------------------------------------:|--------------------------------------------------------------------|
| **Input** | | |
| `texts` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Full support |
| `images` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Multimodal models only; data URIs, plus URLs and S3 URIs |
| `inputs` (fused text + image) | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Rejected with 400 — use `texts` or `images` instead |
| **Model Parameters** | | |
| `input_type` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Applied to Cohere models; no equivalent on other providers |
| `output_dimension` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Some models support dimension reduction |
| `truncate`, `max_tokens` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Cohere models only |
| `embedding_types` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | `int8`/`uint8`/`binary`/`ubinary` on Cohere models, `binary` also on Titan Embed v2; `base64` always computed client-side; other combinations return 400 |
| `priority` | :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } | Accepted but ignored — request scheduling priority is not applicable on Bedrock |
| Extra model-specific params | :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } | Extra fields are forwarded as additional model request parameters |
| **Output** | | |
| `images` metadata array | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Echoed by models that report image dimensions (e.g. Cohere Embed) |
| **Usage tracking** | | |
| `billed_units.input_tokens` | :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } | Estimated on some models |
**Legend:**
* :material-check-circle:{ .success role="img" aria-label="Supported" } **Supported** — Fully compatible with the Cohere API
* :material-cog:{ .model-dep role="img" aria-label="Model-dependent" } **Available on Select Models** — Check your model's capabilities
* :material-minus-circle:{ .partial role="img" aria-label="Partial" } **Partial** — Supported with limitations
* :material-close-circle:{ .unsupported role="img" aria-label="Unsupported" } **Unsupported** — Not available in this implementation
* :material-plus-circle:{ .extra-feature role="img" aria-label="Extra feature" } **Extra Feature** — Enhanced capability beyond the Cohere API
## Model Support
These are the model families served by this route, with the constraint each one places on a request:
### { style="height: 1.2em; vertical-align: text-bottom;" } Cohere Models
| Model | Model ID | Cohere Name | Notes |
|------------------------------|--------------------------------|----------------------------|--------------------------------------------------------------------|
| Cohere Embed v4 | `cohere.embed-v4:0` | `embed-v4.0` | The only model accepting `texts` and `images` in the same request |
| Cohere Embed Multilingual v3 | `cohere.embed-multilingual-v3` | `embed-multilingual-v3.0` | `texts` or `images` in a request, not both |
| Cohere Embed English v3 | `cohere.embed-english-v3` | `embed-english-v3.0` | `texts` or `images` in a request, not both |
!!! tip "Cohere's Own Model Names Resolve As They Stand"
Each Cohere model is published under the name [Cohere's API](https://docs.cohere.com/docs/models) uses as well as its Bedrock ID, derived from the ID rather than curated by hand, so an application already calling Cohere changes only its base URL. Both forms reach the same model; a Cohere model Bedrock does not serve (e.g. `embed-english-light-v3.0`) returns `404` until you map it with [`MODEL_ALIASES`](operations_configuration.md#model-aliases).
### { style="height: 1.2em; vertical-align: text-bottom;" } Amazon Models
| Model | Model ID | Notes |
|-------------------------------------|-------------------------------------------|------------------------------------------------------------------|
| Amazon Nova 2 Multimodal Embeddings | `amazon.nova-2-multimodal-embeddings-v1:0` | `output_dimension` limited to `256`, `384`, `1024` or `3072` |
| Amazon Titan Embed Text v2 | `amazon.titan-embed-text-v2:0` | Text only |
| Amazon Titan Embed Image v1 | `amazon.titan-embed-image-v1` | The only Titan model accepting `images` |
| Amazon Titan Embed Text v1 | `amazon.titan-embed-text-v1` | Deprecated — served by `amazon.titan-embed-text-v2:0` instead |
### { style="height: 1.2em; vertical-align: text-bottom;" } TwelveLabs Models
| Model | Model ID | Notes |
|------------------------------|-----------------------------------|--------------------------------------------|
| TwelveLabs Marengo Embed 3.0 | `twelvelabs.marengo-embed-3-0-v1:0` | `output_dimension` is rejected with a 400 |
| TwelveLabs Marengo Embed 2.7 | `twelvelabs.marengo-embed-2-7-v1:0` | `output_dimension` is rejected with a 400 |
## Quantized and Base64 Embedding Types
Set `embedding_types` to request quantized vectors alongside, or instead of, the default `float` vectors. Bedrock Cohere Embed models natively support `int8`, `uint8`, `binary`, and `ubinary`; Titan Embed v2 natively supports `binary`. `base64` is always available: it is computed client-side (little-endian float32 bytes, base64-encoded) from the `float` embedding, matching the Cohere API encoding. Requesting a type not supported by the resolved model returns 400. Only the requested types are populated in the response.
```bash
curl -X POST "$BASE/v2/embed" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "cohere.embed-v4:0",
"input_type": "search_document",
"texts": ["Hello world"],
"embedding_types": ["int8", "base64"]
}'
```
```json
{
"response_type": "embeddings_by_type",
"id": "0f1b3c6e8d9a4b5c8e7f6a5b4c3d2e1f",
"embeddings": {
"int8": [[12, -34, ...]],
"base64": ["rBgKPw..."]
},
"texts": ["Hello world"],
"meta": {
"api_version": {"version": "2"},
"billed_units": {"input_tokens": 4}
}
}
```
## Cohere v1 Embed API (Legacy)
The legacy `/v1/embed` endpoint is also available for older Cohere SDKs (`cohere.Client`) and third-party integrations that predate the v2 API. It shares the same Bedrock backend and model support as `/v2/embed`; new clients should prefer the v2 endpoint.
**Differences from the v2 endpoint:**
| Feature | Status | Notes |
|----------------------------|:---------------------------------------:|---------------------------------------------------------------------------------|
| Default response shape | :material-check-circle:{ .success role="img" aria-label="Supported" } | Legacy `embeddings_floats`: a plain list of float vectors |
| `embedding_types` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Any value switches to the `embeddings_by_type` shape; same type support as the v2 endpoint |
| `input_type` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Optional — forwarded to Cohere models when provided; the backend defaults to `search_document` otherwise |
| `meta.api_version.version` | :material-check-circle:{ .success role="img" aria-label="Supported" } | Reported as `"1"` |
**Example request:**
```bash
curl -X POST "$BASE/v1/embed" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "cohere.embed-multilingual-v3",
"input_type": "search_document",
"texts": ["Hello world", "Bonjour le monde"]
}'
```
**Example response:**
```json
{
"response_type": "embeddings_floats",
"id": "0f1b3c6e8d9a4b5c8e7f6a5b4c3d2e1f",
"embeddings": [[0.012, -0.034, ...], [0.041, 0.007, ...]],
"texts": ["Hello world", "Bonjour le monde"],
"meta": {
"api_version": {"version": "1"},
"billed_units": {"input_tokens": 8}
}
}
```
## How It Works
Requests are served by the same Bedrock embedding backends as the [OpenAI-compatible Embeddings API](api_openai_embeddings.md), with automatic multi-region routing and failover across the regions where the selected model is available.
- When both `texts` and `images` are provided, embeddings are returned in request order: all texts first, then all images.
- Guardrail and performance headers available on the [OpenAI-compatible Embeddings API](api_openai_embeddings.md#available-request-headers) work on this route too.
## Billing
Requests are billed through Bedrock (per token), not in Cohere search units; `billed_units.input_tokens` reports the Bedrock-metered input tokens. Usage appears in [usage logs and cost tracking](operations_logging_monitoring.md) as `input_tokens`.
---
## Search Models
Source: https://stdapi.ai/md/api_search_models.md
# Search Models API
Discover available models by capability — filter by modality, route, region, streaming support, Batch API support, or legacy status. This endpoint is purpose-built for AI agents that need to identify the right model ID before invoking another endpoint.
!!! tip "Browsing rather than calling?"
The [Models](models.md) page publishes the same catalogue as an interactive table — with AWS prices, regional availability and public leaderboard scores — before you have a deployment.
## Quick Start
| Endpoint | Method | MCP Tool |
|----------|--------|----------|
| `/search_models` | `GET` | `search_models` |
## How It Works
All query parameters are optional. Parameters combine with **AND** logic — only models matching every supplied filter are returned. Results are sorted by model ID. With no filters, every active (non-legacy) model is returned (see the `legacy` note below for deprecated-model lookups).
**Agent workflow:** call `search_models` first to obtain the correct model ID, then pass it to the target endpoint. To compare costs before picking, pass the shortlisted IDs to the [Model Pricing API](api_model_pricing.md).
## Query Parameters
| Parameter | Type | Description |
|---------------------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `input_modalities` | `string` | Repeatable. Filter by input modality: `TEXT`, `IMAGE`, `VIDEO`, `AUDIO`, `SPEECH` |
| `output_modalities` | `string` | Repeatable. Filter by output modality: `TEXT`, `IMAGE`, `VIDEO`, `SPEECH`, `EMBEDDING`, `RERANKING`, `MODERATION` |
| `route` | `string` | Filter to models supporting a route path (e.g. `/v1/chat/completions`) **or** an MCP tool name (e.g. `openai_chat_completion`) — both formats are accepted transparently |
| `region` | `string` | Filter to models available in a specific AWS region (e.g. `us-east-1`) |
| `streaming` | `boolean` | `true` = streaming-capable models only, `false` = non-streaming only |
| `batch` | `boolean` | `true` = models advertised for the [Batch API](api_openai_batches.md) only, `false` = the rest. Best effort — see the note below. |
| `legacy` | `boolean` | `true` = deprecated models only, `false` = active models only. Deprecated models are excluded when omitted. |
!!! note "Modality values are case-insensitive"
`TEXT`, `text`, and `Text` are all accepted.
!!! warning "Batch support is advertised on a best-effort basis"
`batch` is a discovery hint, not a guarantee, and it is never used to reject a request: a batch naming a model that is not advertised is still submitted, and only the backend decides. So a model with `batch: false` — or with no `batch` field at all — may well run a batch successfully, and the authoritative answer is what you get back when you submit one. Treat it as a shortlist to start from, not as a list of the only models that work.
!!! note "Legacy models are excluded by default"
Deprecated models are left out of the results unless you pass `legacy=true`. Pass it if you specifically need to look up a deprecated model, for example to check its replacement — it returns deprecated models only, not the active ones plus the deprecated ones. Combine both calls (with and without `legacy=true`) if you need the full catalogue.
## Response Fields
Each item in the returned list is a `ModelDetails` object:
| Field | Description |
|-------|-------------|
| `id` | Amazon Bedrock model ID — pass this to other endpoints |
| `name` | Human-readable model name |
| `provider` | Model provider (e.g. `Anthropic`, `Amazon`, `Meta`) |
| `service` | AWS service serving the model: `AWS Bedrock Runtime`, `AWS Bedrock Mantle`, `AWS Comprehend`, `AWS Polly`, or `AWS Transcribe` |
| `input_modalities` | List of accepted input types |
| `output_modalities` | List of produced output types |
| `aliases` | Alternate model names accepted by the `model` parameter of the other endpoints (if any) |
| `supported_routes` | API routes this model can be used with |
| `supported_mcp_tools` | MCP tool names this model supports |
| `regions` | AWS regions where this model is available |
| `response_streaming` | Whether streaming responses are supported |
| `batch` | `true` = advertised for the [Batch API](api_openai_batches.md); `false` = not advertised; absent = unknown. Best effort — see the note above |
| `legacy` | `true` = deprecated model; `false` or absent = active |
| `start_of_life_time` | GA date, if known |
| `end_of_life_time` | Deprecation date, if known |
| `legacy_time` | Date the model was marked legacy, if known |
| `public_extended_access_time` | Extended public-access end date, if known |
| `inference_profiles` | Per-region inference profile IDs as a `region → profile ID` mapping (if any) |
## Examples
The `curl` examples below use a `$BASE` variable set to your scheme and host — native routes such as `/search_models` are not prefixed:
```bash
export BASE="https://your-host"
```
**All models accepting TEXT input:**
```bash
curl -G "$BASE/search_models" \
--data-urlencode "input_modalities=TEXT" \
-H "Authorization: Bearer $API_KEY"
```
**Image-generation capable models — route path or MCP tool name, both work:**
```bash
# Using the API route path
curl -G "$BASE/search_models" \
--data-urlencode "route=/v1/images/generations" \
-H "Authorization: Bearer $API_KEY"
# Using the MCP tool name — same result
curl -G "$BASE/search_models" \
--data-urlencode "route=openai_image_generation" \
-H "Authorization: Bearer $API_KEY"
```
**Active chat models (legacy models are excluded by default):**
```bash
curl -G "$BASE/search_models" \
--data-urlencode "route=openai_chat_completion" \
-H "Authorization: Bearer $API_KEY"
```
**Active streaming models in `us-east-1`:**
```bash
curl -G "$BASE/search_models" \
--data-urlencode "region=us-east-1" \
--data-urlencode "streaming=true" \
-H "Authorization: Bearer $API_KEY"
```
**Chat models advertised for the Batch API:**
```bash
curl -G "$BASE/search_models" \
--data-urlencode "route=openai_chat_completion" \
--data-urlencode "batch=true" \
-H "Authorization: Bearer $API_KEY"
```
**Look up a deprecated model (see the `legacy=true` note above):**
```bash
curl -G "$BASE/search_models" \
--data-urlencode "route=openai_chat_completion" \
--data-urlencode "legacy=true" \
-H "Authorization: Bearer $API_KEY"
```
## Status Codes
| Status | Cause |
|--------|-------|
| `200` | Success — valid filters that match zero models still return `200` with an empty list |
| `400` | Unrecognized filter value: unknown modality name, route path, MCP tool name, or a region where no model is available |
!!! tip "Empty list vs. 400"
A `400` is returned when a filter value is completely unrecognized (e.g. a typo in a modality name) or when the requested `region` serves no model at all. Otherwise, a combination of filters that happens to match zero models still returns `200` with an empty list.
## Using `search_models` as an MCP Tool
When MCP is enabled, `search_models` is exposed as an MCP tool under the same name. AI agents should call it **before** any other tool to identify which model ID to use.
The `route` parameter accepts either format — agents can pass the MCP tool name they intend to call next without needing to know the corresponding HTTP path:
```json
{
"tool": "search_models",
"arguments": {
"route": "openai_chat_completion"
}
}
```
This is the preferred tool for model discovery — it returns richer metadata than `openai_model_list` or `anthropic_model_list` and supports capability-based filtering so agents can select the most appropriate model for their task.
!!! tip "Always include `search_models` in your MCP tool set"
When configuring `MCP_INCLUDE_TOOLS`, always add `search_models` so agents can discover the right model ID dynamically rather than relying on hardcoded values. See [Operations Configuration → MCP](operations_configuration.md#mcp-model-context-protocol) for details.
---
## Model Pricing
Source: https://stdapi.ai/md/api_model_pricing.md
# Model Pricing API
Query the exact AWS unit prices for one or more models, straight from the same AWS Price List catalog the server uses for [request cost tracking](operations_cost_management.md#cost-tracking-real-time-aws-pricing). Purpose-built for cost-aware model selection: shortlist models with [`search_models`](api_search_models.md), then compare their price cards in one call.
## Quick Start
| Endpoint | Method | MCP Tool |
|----------|--------|----------|
| `/model_pricing` | `GET` | `model_pricing` |
## How It Works
Each returned row is one **published AWS price**: a billed dimension (same vocabulary as the request-log usage entries) plus its published variants — service tier, prompt-cache write TTL, serving profile, media spec, and long-context bucket. `unit_price` is an exact decimal string per **one** billed unit (token, image, second, character, request, search unit).
By default the card reflects **your server's configuration**: only the configured Bedrock regions, each model's default service tier, and its effective serving profile (routing) are returned, with the same fallbacks billing applies: a tier or routing without a distinct published rate falls back to the standard/plain row, and a standard row kept for an unpublished tier is repriced at what that tier actually bills — half the standard token rates for `flex` and `batch`, 1.75× for `priority`, per-request fees unchanged — while its `tier` field still reads `standard`. Pass `all_prices=true` for the full published price table, and explicit `region`/`tier`/`routing` filters to override any axis in either mode.
A few rates AWS publishes without a model — because one flat rate covers every model that can incur them — are exposed under a stable identifier of their own rather than repeated on each model. The built-in [web search tool](api_openai_responses.md#openai-gpt-web-search) is priced as `amazon.bedrock-web-search`, on the `grounding_requests` dimension, one unit per query; a retrieval from a fully managed [knowledge base store](api_openai_vector_stores.md#knowledge-base-stores) is priced as `amazon.bedrock-knowledge-base`, on the `search_units` dimension, one unit per query; the guardrail policies are priced under [one identifier per policy](api_openai_moderations.md#billing). Ask for these identifiers by name (`model=amazon.bedrock-web-search`): they are rates rather than models you can call, so they are not part of the unfiltered listing.
Prices are indexed **eagerly**: models not currently accessible to your account can still be priced. An empty `prices` list means AWS publishes no rows for that model (or none match the filters). A missing variant row means AWS publishes no distinct rate for it — billing then falls back the same way cost tracking does.
This endpoint responds without contacting AWS, so it is fast enough to call inline. It requires cost tracking to be enabled (`COST_TRACKING=true`), and returns a retry-later `503` while the price catalog is still loading after startup. To compare prices without a running server, the [Models](models.md) page publishes them side by side, per region and per service tier.
## Query Parameters
All parameters are optional and combine with **AND** logic.
| Parameter | Type | Description |
|-------------|-----------|---------------------------------------------------------------------------------------------------------|
| `model` | `string` | Repeatable. Model IDs or aliases to price; omit for the full list of available models |
| `region` | `string` | Only prices for this AWS region (e.g. `us-east-1`) |
| `tier` | `string` | Only this service tier: `standard`, `flex`, `priority`, `batch` |
| `dimension` | `string` | Repeatable. Only these billed dimensions (e.g. `input_tokens`, `output_tokens`, `output_images`) |
| `variants` | `boolean` | `false` = base price card only: standard tier without cache-TTL, routing, or long-context rows (media spec rows are kept) |
| `currency` | `string` | Only prices in this ISO currency code, case-insensitive; must be a currency present in the catalog (e.g. `USD`, `EUR` on the AWS European Sovereign Cloud (EUSC)) |
| `routing` | `string` | Only this published serving-profile price variant: `global`, `latency`. Row `routing` values enriched for display (geography prefixes, AWS regions) cannot be filtered on — use `region` for those |
| `context` | `string` | Only this context-length bucket: `long` (prompts beyond 200K tokens) |
| `all_prices` | `boolean` | `true` = the full published price table; `false` (default) = only the prices matching the server's configuration |
## Response Fields
One `ModelPricing` object per requested model, in request order (duplicates removed); with no `model` filter, one per available model, sorted by ID:
| Field | Description |
|-------|-------------|
| `id` | The model ID as requested |
| `service` | AWS service/API the prices apply to (e.g. `bedrock-runtime`). Note: this uses AWS API endpoint identifiers, a different vocabulary from the display names in the [`search_models`](api_search_models.md) `service` field — both match the code, but the values are not comparable |
| `default_tier` | Service tier this server applies to the model by default ([`DEFAULT_MODEL_SERVICE_TIERS`](operations_configuration.md#default-model-service-tiers)) |
| `default_routings` | Serving profiles this server can use for the model across its configured regions, in configured-region order: `global`, geography prefixes (`eu`, `us`, …), or AWS regions |
| `prices` | Price rows, sorted by region then remaining axes; empty when AWS publishes none |
Each row in `prices` (axes are omitted when not applicable):
| Field | Description |
|-------|-------------|
| `region` | AWS region the price applies to |
| `dimension` | Billed dimension — same names as the [usage log entries](operations_logging_monitoring.md#usage-metrics-fields) |
| `tier` | Service tier: `standard`, `flex`, `priority`, `batch` |
| `cache_ttl` | Prompt-cache write TTL bucket (`5m`, `1h`), when distinctly priced |
| `routing` | Serving profile: `global` (global cross-region inference), a geography prefix like `eu`/`us` (regional cross-region inference), an AWS region (single-region inference), or `latency` for the latency-optimized variant |
| `spec` | Media bucket, e.g. image `resolution:quality` (`1024:standard`) or a modality qualifier |
| `context` | `long` for the beyond-200K-tokens prompt rate |
| `unit_price` | Exact plain-decimal price per **one** billed unit (no exponent, no trailing zeros) |
| `currency` | ISO currency code (`USD` commercially, `EUR` on EUSC) |
## Examples
The `curl` examples below use a `$BASE` variable set to your scheme and host — native routes such as `/model_pricing` are not prefixed:
```bash
export BASE="https://your-host"
```
**Base price card of a shortlist:**
```bash
curl -G "$BASE/model_pricing" \
--data-urlencode "model=anthropic.claude-sonnet-4-5-20250929-v1:0" \
--data-urlencode "model=amazon.nova-pro-v1:0" \
--data-urlencode "variants=false" \
-H "Authorization: Bearer $API_KEY"
```
**Base price card of every available model:**
```bash
curl -G "$BASE/model_pricing" \
--data-urlencode "variants=false" \
-H "Authorization: Bearer $API_KEY"
```
**Token rates in one region:**
```bash
curl -G "$BASE/model_pricing" \
--data-urlencode "model=amazon.nova-pro-v1:0" \
--data-urlencode "region=us-east-1" \
--data-urlencode "dimension=input_tokens" \
--data-urlencode "dimension=output_tokens" \
-H "Authorization: Bearer $API_KEY"
```
**Response (trimmed):**
```json
[
{
"id": "amazon.nova-pro-v1:0",
"service": "bedrock-runtime",
"default_tier": "standard",
"default_routings": ["us"],
"prices": [
{"region": "us-east-1", "dimension": "input_tokens", "tier": "standard", "routing": "us", "unit_price": "0.0000008", "currency": "USD"},
{"region": "us-east-1", "dimension": "output_tokens", "tier": "standard", "routing": "us", "unit_price": "0.0000032", "currency": "USD"}
]
}
]
```
## Status Codes
| Status | Cause |
|--------|-------|
| `200` | Success — a model with no published price, or no row matching the filters, still returns `200` with an empty `prices` list |
| `400` | Unknown `tier`, `dimension`, `routing`, `context`, or `currency` filter value |
| `503` | Model pricing is not available on this server, or the price catalog is not loaded yet (retry later) |
## Using `model_pricing` as an MCP Tool
When MCP is enabled, `model_pricing` is exposed as an MCP tool under the same name. The intended agent workflow:
1. Call `search_models` to shortlist model IDs for the task.
2. Call `model_pricing` with the shortlist and compare.
3. Keep responses small: use `variants=false` and `dimension` filters.
```json
{
"tool": "model_pricing",
"arguments": {
"model": ["anthropic.claude-sonnet-4-5-20250929-v1:0", "amazon.nova-pro-v1:0"],
"variants": false,
"dimension": ["input_tokens", "output_tokens"]
}
}
```
!!! tip "Prices are strings on purpose"
`unit_price` values are exact decimal strings (`"0.000003"`), never floats — JSON floats cannot represent small per-token rates without exponent notation or rounding noise. Parse them with a decimal type for arithmetic.
---
## Deploy on AWS
Source: https://stdapi.ai/md/operations_getting_started.md
# :material-rocket-launch: Deploy stdapi.ai on AWS
Get a production-grade AI gateway running on AWS with two Terraform commands, speaking the OpenAI, Anthropic, and Cohere APIs. Terraform provisions the stack — ECS Fargate, HTTPS, auto-scaling, optional WAF, and optional monitoring. You bring the Marketplace subscription, AWS credentials, and — for a custom domain — the domain and its certificate.
!!! tip trial "14-Day Free Trial"
The AWS Marketplace subscription includes a **14-day free trial of the stdapi.ai license**. AWS charges for the infrastructure it deploys (ALB, Fargate, KMS, NAT) and for Bedrock usage apply from the first minute — see [Deployment Cost](#deployment-cost).
!!! info "Need help?"
For questions, issue reports, or assistance, see the [Contact](contact.md) page.
---
## :material-rocket-launch: Quick Start
### Prerequisites
1. **Subscribe on AWS Marketplace** — this is the action that starts your 14-day free trial:
[Subscribe on AWS Marketplace — starts your 14-day free trial](https://aws.amazon.com/marketplace/pp/prodview-su2dajk5zawpo){ .md-button .md-button--primary }
2. Install [Terraform](https://www.terraform.io/downloads) or [OpenTofu](https://opentofu.org/docs/intro/install/) >= 1.5.
3. Configure [AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) (`aws configure` or `aws sso login`).
AWS infrastructure cost depends entirely on your configuration — from a single scheduled Spot container with no load balancer to a full multi-AZ stack — so there's no one figure to quote; see [Deployment Cost](#deployment-cost) below. stdapi.ai's own license runs $0.10/container-hour ($0.09 via private offer); Bedrock usage is billed separately by AWS at cost.
!!! warning "Requires AWS administrator permissions"
The Terraform module provisions IAM roles and policies, KMS keys, ECS/Fargate, ALB, and networking. A restricted developer profile will fail during `terraform apply`.
**Strongly recommended:** deploy into a **sandbox / non-production AWS account first** to evaluate the stack, then replicate into your target account with scoped-down principals once you've validated it.
!!! tip "Confirm your AWS identity and region before deploying"
The AWS provider uses the region and profile from your environment — not a Terraform variable. Check both before running `terraform apply`:
```bash
aws sts get-caller-identity
aws configure get region
```
### Deploy
```bash
git clone https://github.com/stdapi-ai/samples.git
cd samples/getting_started_production/terraform
terraform init
terraform apply
```
??? info "No git? Download the ZIP"
```bash
curl -L https://github.com/stdapi-ai/samples/archive/refs/heads/main.zip -o samples.zip
unzip samples.zip
cd samples-main/getting_started_production/terraform
terraform init
terraform apply
```
That's it. Two Terraform commands, and you have:
- Production-grade ECS Fargate deployment with HTTPS
- Regional S3 buckets
- Auto-scaling and API key authentication
- Interactive API documentation at `/docs`
- IP-restricted access (your IP only)
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
openai[" OpenAI SDK"] -->|HTTPS| alb[" ALB"]
anthropic[" Anthropic SDK"] -->|HTTPS| alb
alb --> ecs[" stdapi.ai ECS Fargate"]
ecs --> bedrock[" Amazon Bedrock"]
ecs --> polly[" Amazon Polly"]
ecs --> transcribe[" Amazon Transcribe"]
ecs --> s3[" Amazon S3"]
ecs --> cloudwatch[" CloudWatch"]
```
### Get Your Credentials
```bash
terraform output -raw api_key
terraform output api_endpoint
terraform output docs_url
```
!!! tip "Ready-to-use Terraform example on GitHub"
:material-map-marker: **Single region** — [getting_started_production](https://github.com/stdapi-ai/samples/tree/main/getting_started_production)
!!! tip "Optional: expose the API as MCP tools"
The [MCP server](features.md#mcp-model-context-protocol) is off by default. Set `enable_mcp_streamable_http = true` on the Terraform module and every endpoint becomes a named MCP tool at `/mcp`, callable directly by Claude Code, LangGraph, or any MCP client.
Every exposed tool adds its schema to each MCP client's context window, so expose only the tools your agents actually use — for example `mcp_include_tools = "openai_chat_completion,openai_embedding,search_models"`. See the [MCP configuration reference](operations_configuration.md#summary-mcp).
!!! tip offer "Buying for an organization? Use an AWS Marketplace private offer"
Custom terms and duration, committed usage, and a preferential rate of **$0.09/container-hour** instead of $0.10. Procured through your existing AWS relationship — no new vendor onboarding, billed on your existing AWS invoice. Want to try first? Use the 14-day free trial of the public listing, then accept the private offer.
[:material-email-outline: Request a Private Offer](contact.md#private-offer)
---
## :material-check-circle: Make Your First API Call
**Want to explore the API without writing code?** Use the `docs_url` from `terraform output docs_url` to open the interactive Swagger documentation in your browser — you can browse all available endpoints and make live API calls directly from the page, no code required.
!!! info "If the docs page returns 503 or shows a TLS warning"
These are normal on a fresh deployment. The ECS service takes 2–3 minutes to pass health checks (→ 503), and the auto-generated `*.elb.amazonaws.com` domain has no trusted TLS certificate (→ browser warning; safe to bypass for testing). See [Troubleshooting](operations_troubleshooting.md) for a permanent HTTPS setup with a custom domain.
stdapi.ai is compatible with both OpenAI and Anthropic SDKs. If you've used either before, you already know how to use it — the base URL changes, along with the API key, and the model name only where it differs, since Anthropic's and OpenAI's own names for the models Bedrock serves resolve here as they stand. Here are the raw HTTP calls with `curl` so you can verify the endpoint from any shell:
=== "OpenAI-compatible"
```bash
API_ENDPOINT=$(terraform output -raw api_endpoint)
API_KEY=$(terraform output -raw api_key)
curl "$API_ENDPOINT/v1/chat/completions" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"messages": [{"role": "user", "content": "Hello! Tell me a joke."}]
}'
```
=== "Anthropic-compatible"
```bash
API_ENDPOINT=$(terraform output -raw api_endpoint)
API_KEY=$(terraform output -raw api_key)
curl "$API_ENDPOINT/anthropic/v1/messages" \
-H "x-api-key: $API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"max_tokens": 1000,
"messages": [{"role": "user", "content": "Hello! Tell me a joke."}]
}'
```
**Using the official SDKs?** Point the `base_url` (Python) / `baseURL` (Node.js) option at `$API_ENDPOINT/v1` (OpenAI SDK) or `$API_ENDPOINT/anthropic` (Anthropic SDK), and set the `model` field to a model from the catalog below. The rest of your existing code is unchanged. The [API Overview](api_overview.md) has SDK snippets for Python, Node.js, and more.
!!! tip "Discover the full model catalog"
Once your first call succeeds, switch the `model` field to any other Bedrock model — `anthropic.claude-fable-5`, `anthropic.claude-sonnet-5`, `qwen.qwen3-coder-next`, and more.
- **Browse before you switch:** the [Models](models.md) page lists everything with prices and scores
- **Browse active models (recommended):** `GET /search_models` — returns every discovered non-legacy model with full details (provider, modalities, supported routes, regions, streaming/legacy status). Add `legacy=true` to look up a deprecated model instead. Or open the interactive Swagger docs.
- **Find a model by capability:** the same endpoint filters by modality, route, region, streaming, Batch API support, or legacy status — e.g. `GET /search_models?input_modalities=IMAGE&route=/v1/chat/completions` returns only vision-capable chat models. This is also the recommended way for AI agents to discover the right model ID before calling another endpoint. See the [Search Models API](api_search_models.md) reference.
- **OpenAI SDK compatibility:** `GET /v1/models` is also available with the standard OpenAI listing format (lighter payload, no capability metadata) for tools that require the exact OpenAI schema.
**Using your own Terraform config instead of the sample?** The sample above enables authentication for you (`api_key_create = true`, retrieved above). Writing your own module config from scratch? stdapi.ai runs without authentication unless you set `api_key_create = true` — see [Authentication & Security](operations_authentication_security.md) for all options.
**Verify the deployment is healthy:**
```bash
curl $API_ENDPOINT/health
# → {"status": "ok"}
```
The `/health` endpoint requires no authentication and is used by the ALB health check.
---
## :material-wrench: Troubleshooting
The `503` and TLS-warning hiccups on first deployment are already covered above — see [Make Your First API Call](#make-your-first-api-call).
:material-arrow-right: **Full troubleshooting guide:** [Troubleshooting](operations_troubleshooting.md) — 401 auth errors, 404 model not found, ThrottlingException, S3 errors, VPC connectivity, Terraform IAM failures, and more.
!!! info "Prefer a hands-off setup?"
A [managed deployment service](https://aws.amazon.com/marketplace/pp/prodview-xknxzjgl7zi5s) is available if you'd rather not manage Terraform yourself. Choose between guided assistance (step-by-step support while you retain full control) or fully managed setup (handled on your behalf, inside your AWS account). Response time is 1 business day during the engagement.
---
## :material-currency-usd: Deployment Cost
AWS infrastructure cost is driven by configuration, not a fixed default: task count (one per Availability Zone unless overridden), Fargate Spot vs. on-demand, scheduled service hours, and whether an ALB is provisioned at all. A minimal deployment — one scheduled Fargate Spot task, no ALB, reached via Service Discovery — and a full multi-AZ production stack — ALB + WAF, one task per AZ, running 24/7 — sit at opposite ends of a wide range.
stdapi.ai's own license is billed separately at **$0.10/container-hour** ($0.09 via [private offer](contact.md#private-offer)); Bedrock and other AI service usage is billed by AWS at cost, with no stdapi.ai markup.
See [Cost-Optimized Deployment](operations_deploy_advanced.md#cost-optimized-deployment) for the Spot/scheduling configuration, and [Cost Management → Gateway Cost](operations_cost_management.md#gateway-cost) for the full tier-by-tier breakdown.
---
## :material-delete-outline: Cleaning Up
When you're done testing, tear down the stack to stop incurring AWS and license charges:
```bash
terraform destroy
```
Running from the ZIP download instead of `git clone`? Run the same command from the `samples-main/getting_started_production/terraform` directory.
---
## :material-arrow-right: Next Steps
- :material-book-open-variant: [**API Overview**](api_overview.md) — Endpoints, parameters, and usage examples
- :material-cog: [**Configuration**](operations_configuration.md) — All environment variables and options
- :material-server-network: [**Advanced Deployment**](operations_deploy_advanced.md) — VPC integration, multi-region, cost optimization, manual ECS
- :material-cash-multiple: [**Cost Management**](operations_cost_management.md) — Infrastructure, license, and AI usage cost breakdown
- :material-directions-fork: [**Resilience & Failover**](operations_resilience.md) — Multi-region routing, and the quota each enabled region adds
- :material-shield-lock: [**Data Sovereignty & Compliance**](operations_compliance.md) — GDPR-compliant region configuration
- :material-puzzle: [**Use Cases**](use_cases.md) — Open WebUI, n8n, coding assistants, and more
- :material-wrench: [**Troubleshooting**](operations_troubleshooting.md) — Common first-deployment errors and fixes
- :material-scale-balance: [**Licensing**](operations_licensing.md) — AGPL vs commercial options
---
## Run Locally with Docker
Source: https://stdapi.ai/md/operations_getting_started_local.md
# :material-docker: Local Development with Docker/Podman
Run stdapi.ai locally for development, testing, and evaluation using the free community container image (AGPL-3.0). Full API compatibility — the same endpoints and features as the production deployment.
!!! tip "New to Amazon Bedrock?"
To run stdapi.ai locally you need:
1. **[Docker](https://docs.docker.com/get-started/get-docker/) or [Podman](https://podman.io/docs/installation)** installed
2. An **AWS account** — [create one free](https://aws.amazon.com/free/)
3. **AWS credentials** configured locally — via `aws configure` or `aws sso login` ([AWS CLI setup guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html))
---
## :material-docker: Run It
**With AWS credentials (after `aws sso login`):**
```bash
docker run --rm -p 8000:8000 \
-v ~/.aws:/home/nonroot/.aws:ro \
-e AWS_BEDROCK_REGIONS=us-east-1,us-west-2 \
-e ENABLE_DOCS=true \
ghcr.io/stdapi-ai/stdapi.ai-community:latest
```
**With environment variables instead:**
```bash
docker run --rm -p 8000:8000 \
-e AWS_ACCESS_KEY_ID=your-access-key-id \
-e AWS_SECRET_ACCESS_KEY=your-secret-access-key \
-e AWS_SESSION_TOKEN=your-session-token \
-e AWS_BEDROCK_REGIONS=us-east-1,us-west-2 \
-e ENABLE_DOCS=true \
ghcr.io/stdapi-ai/stdapi.ai-community:latest
```
??? info "Podman on Fedora/RHEL with SELinux"
Add the `:z` SELinux label and `--userns=keep-id`:
```bash
podman run --rm -p 8000:8000 \
--userns=keep-id \
-v ~/.aws:/home/nonroot/.aws:ro,z \
-e AWS_BEDROCK_REGIONS=us-east-1,us-west-2 \
-e ENABLE_DOCS=true \
ghcr.io/stdapi-ai/stdapi.ai-community:latest
```
The `:z` flag relabels files for container access. Use `:Z` if multiple containers share the volume. `--userns=keep-id` maps your host user ID to the container user.
See also [Troubleshooting → Podman volume mount fails on Fedora/RHEL with SELinux](operations_troubleshooting.md#terraform-deployment) if you hit this after the fact.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
openai[" OpenAI SDK"] --> local[" stdapi.ai (community) Docker/Podman"]
anthropic[" Anthropic SDK"] --> local
local --> bedrock[" Amazon Bedrock"]
local --> polly[" Amazon Polly"]
local --> transcribe[" Amazon Transcribe"]
local --> s3[" Amazon S3"]
```
---
## :material-check-circle: Test It
```bash
# Check health
curl http://localhost:8000/health
# List all available models
curl http://localhost:8000/search_models
# Search models by capability (e.g. streaming-capable chat models only)
curl "http://localhost:8000/search_models?route=/v1/chat/completions&streaming=true"
# Chat completion
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
**Interactive API docs:** Open [http://localhost:8000/docs](http://localhost:8000/docs) for Swagger UI with all available endpoints.
**Point an application at it:** set the `base_url` (Python) / `baseURL` (Node.js) option to `http://localhost:8000/v1` (OpenAI SDK) or `http://localhost:8000/anthropic` (Anthropic SDK). Often that is the whole change: each model is published under the name its provider uses, so a `model` field already reading `claude-sonnet-5` or `gpt-oss-120b` resolves as it stands. Change it where the name differs — one your application hard-codes for a model this deployment does not serve, such as `gpt-4o` or `dall-e-3`, returns `404` here until you point it at a model from the catalog above or map it with [`MODEL_ALIASES`](operations_configuration.md#model-aliases), while every model your regions serve is one name away. No API key is required by default — pass any non-empty string if your client insists on one. The [API Overview](api_overview.md) has SDK snippets for Python, Node.js, and more.
!!! tip "Try other models"
`amazon.nova-micro-v1:0` is a fast, low-cost model — great for confirming the pipeline works. Once you see a response, switch the `model` field to `anthropic.claude-fable-5`, `anthropic.claude-sonnet-5`, or any other Bedrock model available in your configured regions.
Use `GET /search_models` (shown above) to discover what's available and filter by capability, or `GET /v1/models` for strict OpenAI SDK compatibility — see the [Search Models API](api_search_models.md) reference.
!!! tip "Optional: expose the API as MCP tools"
The [MCP server](features.md#mcp-model-context-protocol) is off by default. Add `-e ENABLE_MCP_STREAMABLE_HTTP=true` to the `docker run` command and every endpoint becomes a named MCP tool at `http://localhost:8000/mcp`, callable directly by Claude Code or any MCP client.
Every exposed tool adds its schema to each MCP client's context window, so expose only the tools you actually use — for example `-e MCP_INCLUDE_TOOLS=openai_chat_completion,openai_embedding,search_models`. See the [MCP configuration reference](operations_configuration.md#summary-mcp).
---
## :material-information: Technical Notes
**Building from source:** See the [Dockerfile](https://github.com/stdapi-ai/stdapi.ai/blob/main/Dockerfile) if you prefer to build the image yourself.
**Container runtime:** Both community and Marketplace images use [Granian](https://github.com/emmett-framework/granian), a high-performance Python ASGI server. Granian environment variables (e.g., `GRANIAN_PORT`, `GRANIAN_WORKERS`) are supported.
**Configuration:** See [Configuration Reference](operations_configuration.md) for all environment variables.
!!! tip "Ready for Production?"
When you're ready to deploy to AWS with HTTPS, auto-scaling, and enterprise features, the [production deployment guide](operations_getting_started.md) gets you running with two Terraform commands. The [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-su2dajk5zawpo) subscription includes a **14-day free trial of the license**.
---
## :material-arrow-right: Next Steps
stdapi.ai works with any OpenAI or Anthropic-compatible tool. Here are popular integrations to try locally:
- :material-chat: [**Open WebUI**](use_cases_openwebui.md) — Private ChatGPT-like interface with RAG, multi-modal support, and document upload
- :material-robot: [**n8n Workflows**](use_cases_n8n.md) — AI-powered automation with 400+ integrations
- :material-code-braces: [**AI Coding Assistants**](use_cases_coding_assistants.md) — Claude Code, Cline, OpenCode, Zed with Amazon Bedrock models
- :material-book-open-variant: [**API Overview**](api_overview.md) — All endpoints, parameters, and usage examples
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy the production stack on AWS with Terraform
- :material-wrench: [**Troubleshooting**](operations_troubleshooting.md) — Podman/SELinux, auth, model-not-found, and other common errors
---
## Advanced Deployment
Source: https://stdapi.ai/md/operations_deploy_advanced.md
# :material-server-network: Advanced Deployment
This page covers deployment scenarios beyond the [Quick Start](operations_getting_started.md). Use these when you need to integrate with existing infrastructure, deploy multi-region, optimize costs, or deploy without Terraform.
!!! tip "Start with Quick Start"
New to stdapi.ai? Begin with the [Getting Started](operations_getting_started.md) guide for the fastest path to a working deployment.
---
## :material-lan: Integration with Existing Infrastructure
Deploy stdapi.ai into your existing VPC and network infrastructure for maximum cost efficiency.
```hcl
module "stdapi_ai" {
source = "stdapi-ai/stdapi-ai/aws"
version = "~> 1.0"
# Use your existing network
subnet_ids = [
"subnet-xxx", # Your existing private subnet 1
"subnet-yyy", # Your existing private subnet 2
]
security_group_id = "sg-zzz" # Your existing security group
}
```
**What you get:**
- ECS Fargate service in your existing VPC
- No additional NAT gateways or load balancers created
- Full monitoring and security features
**How to connect to your ALB:**
After deployment, add a target group pointing to port 8000, with a health check on `/health` — see the full target group example (with recommended health-check thresholds) in the collapsed section below.
??? example "Full integration example with ALB, IAM policies, and advanced configuration"
**Complete integration configuration with all optional features:**
```hcl
module "stdapi_ai_integrated" {
source = "stdapi-ai/stdapi-ai/aws"
version = "~> 1.0"
name_prefix = "my-stdapi-integrated"
# Use existing network infrastructure
subnet_ids = [
"subnet-0123456789abcdef0",
"subnet-0123456789abcdef1",
"subnet-0123456789abcdef2"
]
security_group_id = "sg-0123456789abcdef0"
# Optional: Reuse existing S3 bucket
aws_s3_bucket = "my-existing-s3-bucket"
# Optional: Service Discovery for private communication
service_discovery_dns_namespace_id = "ns-xxxxx"
service_discovery_dns_name = "stdapi"
# Optional: Use existing Secrets Manager secret for API key
api_key_secretsmanager_secret = "my-api-keys"
api_key_secretsmanager_key = "stdapi_key"
# Optional: Attach custom IAM policies
ecs_task_role_policy_arns = [
aws_iam_policy.custom_s3_access.arn,
aws_iam_policy.api_key_secrets_access.arn
]
# Monitoring
container_insight = "enhanced"
alarms_enabled = true
sns_topic_arn = "arn:aws:sns:us-east-1:123456789012:alerts"
}
# Example: Custom IAM policy for additional S3 bucket access
data "aws_iam_policy_document" "custom_s3_access" {
statement {
sid = "S3BucketAccess"
effect = "Allow"
actions = [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
]
resources = ["arn:aws:s3:::my-existing-s3-bucket/*"]
}
statement {
sid = "KMSEncryptionForS3"
effect = "Allow"
actions = [
"kms:Decrypt",
"kms:GenerateDataKey"
]
resources = ["arn:aws:kms:us-east-1:123456789012:key/your-s3-bucket-kms-key-id"]
condition {
test = "StringEquals"
variable = "kms:ViaService"
values = ["s3.us-east-1.amazonaws.com"]
}
}
}
resource "aws_iam_policy" "custom_s3_access" {
name = "stdapi-custom-s3-access"
description = "Custom S3 access for stdapi.ai integration"
policy = data.aws_iam_policy_document.custom_s3_access.json
}
# Example: IAM policy for API key access from Secrets Manager
# Required when using api_key_secretsmanager_secret parameter
data "aws_iam_policy_document" "api_key_secrets_access" {
statement {
sid = "SecretsManagerAccess"
effect = "Allow"
actions = ["secretsmanager:GetSecretValue"]
resources = ["arn:aws:secretsmanager:us-east-1:123456789012:secret:my-api-keys-*"]
}
statement {
sid = "KMSDecryptionForSecretsManager"
effect = "Allow"
actions = ["kms:Decrypt"]
resources = ["arn:aws:kms:us-east-1:123456789012:key/your-kms-key-id"]
condition {
test = "StringEquals"
variable = "kms:ViaService"
values = ["secretsmanager.us-east-1.amazonaws.com"]
}
}
}
resource "aws_iam_policy" "api_key_secrets_access" {
name = "stdapi-api-key-secrets-access"
description = "Access to Secrets Manager for stdapi.ai API key"
policy = data.aws_iam_policy_document.api_key_secrets_access.json
}
# Alternative: IAM policy for API key access from SSM Parameter Store
# Use this when using api_key_ssm_parameter instead of Secrets Manager
data "aws_iam_policy_document" "api_key_ssm_access" {
statement {
sid = "SSMParameterAccess"
effect = "Allow"
actions = ["ssm:GetParameter"]
resources = ["arn:aws:ssm:us-east-1:123456789012:parameter/stdapi/api-key"]
}
statement {
sid = "KMSDecryptionForSSM"
effect = "Allow"
actions = ["kms:Decrypt"]
resources = ["arn:aws:kms:us-east-1:123456789012:key/your-kms-key-id"]
condition {
test = "StringEquals"
variable = "kms:ViaService"
values = ["ssm.us-east-1.amazonaws.com"]
}
}
}
resource "aws_iam_policy" "api_key_ssm_access" {
name = "stdapi-api-key-ssm-access"
description = "Access to SSM Parameter Store for stdapi.ai API key"
policy = data.aws_iam_policy_document.api_key_ssm_access.json
}
# Outputs for integration
output "ecs_service_info" {
description = "ECS service details for connecting your resources"
value = {
cluster_name = module.stdapi_ai_integrated.cluster_name
service_name = module.stdapi_ai_integrated.service_name
security_group_id = module.stdapi_ai_integrated.security_group_id
port = module.stdapi_ai_integrated.port
service_discovery = module.stdapi_ai_integrated.service_discovery_service_name
}
}
output "integration_resources" {
description = "Resources for connecting stdapi.ai to your infrastructure"
value = {
s3_bucket_id = module.stdapi_ai_integrated.bucket_id
kms_key_arn = module.stdapi_ai_integrated.kms_key_arn
log_groups = module.stdapi_ai_integrated.cloudwatch_log_groups_names
}
}
```
**Manual integration steps:**
1. **Configure your ALB target group** to point to the ECS service:
```hcl
resource "aws_lb_target_group" "stdapi" {
name = "my-stdapi-tg"
port = 8000
protocol = "HTTP"
vpc_id = "vpc-xxxxx"
target_type = "ip"
health_check {
path = "/health"
healthy_threshold = 2
unhealthy_threshold = 3
}
}
# Attach to your existing ALB listener
resource "aws_lb_listener_rule" "stdapi" {
listener_arn = aws_lb_listener.existing.arn
priority = 100
action {
type = "forward"
target_group_arn = aws_lb_target_group.stdapi.arn
}
condition {
path_pattern {
values = ["/v1/*"]
}
}
}
```
2. **Update security groups** to allow traffic:
```hcl
# Allow your ALB to reach stdapi.ai
resource "aws_security_group_rule" "alb_to_stdapi" {
type = "ingress"
from_port = 8000
to_port = 8000
protocol = "tcp"
security_group_id = module.stdapi_ai_integrated.security_group_id
source_security_group_id = var.your_alb_security_group_id
}
```
3. **Access via Service Discovery** (optional):
```bash
# OpenAI-compatible endpoint
curl -X POST "http://stdapi.your-namespace.local:8000/v1/chat/completions" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "anthropic.claude-sonnet-5", "messages": [{"role": "user", "content": "Hello"}]}'
# Anthropic-compatible endpoint
curl -X POST "http://stdapi.your-namespace.local:8000/anthropic/v1/messages" \
-H "x-api-key: YOUR_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{"model": "anthropic.claude-sonnet-5", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
!!! warning "Service discovery in an IPv6-enabled subnet needs a dual-stack listener"
The image listens on IPv4 only (`GRANIAN_HOST=0.0.0.0`), while ECS service discovery publishes an `AAAA` record for every task in an IPv6-enabled subnet. Clients that prefer that record — Node.js among them — then fail with `ECONNREFUSED` while Python clients fall back to the `A` record and hide the problem. Set `GRANIAN_HOST=::` in the task environment for a socket that answers both families; the module sets it for you when the VPC has IPv6 enabled. If `PROXY_TRUSTED_HOSTS` is also set, add the IPv4-mapped ranges alongside the plain ones — see [`PROXY_TRUSTED_HOSTS`](operations_configuration.md#proxy-trusted-hosts).
**Use cases:**
- Connect to existing internal ALB
- Private API for internal microservices
- Connect to service mesh (App Mesh, Consul)
- Custom networking with VPN/Direct Connect
- Multi-account setups with PrivateLink
- Access additional AWS resources (S3 buckets, Secrets Manager, DynamoDB, etc.)
**Custom IAM policies use cases:**
- Grant access to additional S3 buckets beyond the default one
- **Access API keys from Secrets Manager or SSM Parameter Store** (required when using `api_key_ssm_parameter` or `api_key_secretsmanager_secret`)
- Read/write to DynamoDB tables for application state
- Access to custom KMS keys for encryption
- Cross-account resource access via IAM roles
!!! warning "Important"
When using `api_key_secretsmanager_secret` or `api_key_ssm_parameter`, you must create and attach an IAM policy granting the ECS task access to the secret/parameter. The module does not automatically create these permissions.
---
## :material-transit-connection-variant: Outbound Network Requirements
Beyond the inbound API traffic, the server reaches out to the AWS endpoints behind the features you enable. Allow these destinations from the task's security group — a blocked one usually makes a capability go missing rather than fail loudly.
| Destination | Needed for |
|---|---|
| `bedrock-runtime..amazonaws.com`, `bedrock..amazonaws.com` | Model invocation and the model catalog, in every configured region |
| `bedrock-mantle..api.aws` | The models served through Amazon Bedrock Mantle |
| The other AWS service endpoints you enable | Amazon Polly, Amazon Transcribe, Amazon Translate, Amazon Comprehend, Amazon S3, AWS STS, AWS Price List |
| `cognito-idp..amazonaws.com` | The user pool key set, when Amazon Cognito authentication is enabled |
| `s3vectors..amazonaws.com` | The [Vector Stores API](api_openai_vector_stores.md), when `AWS_S3_VECTORS_BUCKET` is set — in that bucket's Region only |
| `sqs..amazonaws.com` | [Durable vector store indexing](operations_resilience.md#vector-store-indexing), when `AWS_SQS_VECTOR_STORE_QUEUE_URL` is set — in the queue's Region only |
Bedrock Mantle is a separate endpoint from classic Bedrock, on a different domain (`api.aws`, not `amazonaws.com`). A network policy that allows the one and not the other is the usual reason a deployment lists every classic model and no Mantle model at all.
### Private deployments
Most of those endpoints have an interface VPC endpoint service, so a deployment with no internet egress reaches them privately. Bedrock Mantle's is:
```text
com.amazonaws..bedrock-mantle
```
Enable private DNS on it, so `bedrock-mantle..api.aws` resolves to the endpoint. When the Terraform module builds the VPC it provisions the interface endpoints itself — including `com.amazonaws..sqs` when the indexing queue lives in the deployment Region. Deploying into your own VPC (`subnet_ids`) makes every endpoint yours to create.
!!! warning "Amazon S3 Vectors is not among the endpoints the module creates"
A deployment with no internet egress and no route to `s3vectors..amazonaws.com` serves every other route normally and fails every vector store call. Confirm the service offers an interface endpoint in your Region before planning a fully private deployment on it, or keep egress to that one endpoint open.
### Proxied deployments
Where egress is only possible through an HTTP proxy, set `HTTPS_PROXY`, `HTTP_PROXY` and `NO_PROXY` in the task environment. Every connection the server makes to an AWS endpoint honours them, so one set of variables covers the whole deployment.
Two connections deliberately do not, and neither needs an entry in `NO_PROXY`:
- **The container task metadata endpoint**, which the ECS agent serves on the task's own host and no proxy can reach.
- **Fetches of a URL a client supplied** — an image or audio URL in a request. Those are validated against the deployment's SSRF policy and connected to the exact address that was validated; routing them through a proxy would hand that decision to the proxy instead.
!!! warning "A `~/.netrc` file in the image becomes credentials on the wire"
Honouring the proxy variables also enables `.netrc` lookups, which is how the standard tooling behaves. If a `.netrc` exists in the container's home directory, its entries are matched against outbound hosts and sent as HTTP authentication.
The images stdapi.ai publishes contain no `.netrc`. Only a custom image, or a volume mounted over the home directory, can introduce one — do not add one, and check for it if you build your own image.
---
## :material-shield-check: Production Deployment (Fully Featured)
Enterprise-ready deployment with HTTPS endpoints, WAF protection, auto-scaling, regional S3 buckets, and comprehensive monitoring.
??? example "Full production example with multi-region Bedrock support"
```hcl
# Main deployment
module "stdapi_ai" {
source = "stdapi-ai/stdapi-ai/aws"
version = "~> 1.0"
# Custom public domain with TLS
alb_domain_name = "api.example.com"
alb_enabled = true
alb_public = true
# Amazon Bedrock region configuration
# Select regions to get available models in the order of preference
aws_bedrock_regions = [
"eu-west-3",
"eu-west-1",
"eu-central-1",
"eu-north-1"
]
# (Optional) In case of regional compliance requirements like GDPR,
# disable "global" cross-region inference to ensure everything is done in valid regions.
# Cross-region inference allows Amazon Bedrock to route requests to different regions for better availability.
# In this example, cross-region inferences will be in EU regions only and comply with GDPR
aws_bedrock_cross_region_inference_global = false
# AI services region extra configuration
# Left unset, a service treats every aws_bedrock_regions entry as a candidate and
# fails over between them. Pin one when the primary region does not offer the
# service, to skip a probe that fails on every call — at the cost of no failover.
# In this example, Amazon Comprehend is not offered on eu-west-3, so we use eu-west-1
aws_comprehend_region = "eu-west-1"
# Authentication (Recommended)
# Enable authentication by generating an API key that can be retrieved using the "api_key" module attribute.
api_key_create = true
# Web Application Firewall (Recommended on public APIs when ALB is enabled)
alb_waf_enabled = true
alb_waf_rate_limit = 2000 # Requests per 5 minutes per IP
alb_waf_block_anonymous_ips = true
# Monitoring & Alerts (Recommended to get alarms notifications)
alarms_enabled = true
sns_topic_arn = "arn:aws:sns:eu-west-3:123456789012:alerts"
}
# Get the API key (Generated with api_key_create = true)
output "api_key" {
value = module.stdapi_ai.api_key
sensitive = true
}
# Main/default region provider
provider "aws" {
region = "eu-west-3"
}
```
??? info "Migrating from manual bucket configuration"
If you have an existing deployment using the deprecated `module "bedrock_bucket_*"` pattern,
see the [migration guide](https://github.com/stdapi-ai/terraform-aws-stdapi-ai-s3-regional-bucket#migration)
for step-by-step `terraform state mv` instructions.
**What you get:**
- High-availability multi-AZ deployment (uses all available AZs in region)
- HTTPS with automatic SSL certificate
- WAF protection with AWS managed rules
- 5 CloudWatch alarms (memory, health, CPU anomaly, capacity, error logs)
- Auto-scaling based on load (min defaults to the number of AZs)
- S3 storage with lifecycle policies
- Container Insights enabled by default (set `container_insight = "enhanced"` for additional OS-level and application performance metrics)
- Regional S3 buckets for Amazon Bedrock multimodal operations (created automatically)
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
openai[" OpenAI SDK"] --> alb[" ALB + WAF"]
anthropic[" Anthropic SDK"] --> alb
alb --> ecs[" stdapi.ai ECS Fargate"]
ecs --> bedrock_primary[" Amazon Bedrock Region 1"]
bedrock_primary -.-|multiple regions| bedrock_region_n[" Amazon Bedrock Region N"]
ecs --> s3_primary[" Regional S3 Bucket Region 1"]
s3_primary -.-|multiple regions| s3_region_n[" Regional S3 Bucket Region N"]
ecs --> polly[" Amazon Polly"]
ecs --> transcribe[" Amazon Transcribe"]
ecs --> cloudwatch[" CloudWatch + Alarms"]
```
??? example "Simplified production example (single region, no multi-region complexity)"
```hcl
module "stdapi_ai" {
source = "stdapi-ai/stdapi-ai/aws"
version = "~> 1.0"
# HTTPS with your domain
alb_domain_name = "api.example.com"
alb_enabled = true
alb_public = true
# Security
api_key_create = true
alb_waf_enabled = true
alb_waf_rate_limit = 2000
alb_waf_block_anonymous_ips = true
# Monitoring
alarms_enabled = true
sns_topic_arn = "arn:aws:sns:us-east-1:123456789012:alerts"
}
output "api_key" {
value = module.stdapi_ai.api_key
sensitive = true
}
output "api_endpoint" {
value = module.stdapi_ai.application_url
}
```
**Deployment time:** ~5-10 minutes
!!! tip "Ready-to-use Terraform examples on GitHub"
- :material-map-marker: **Single region** — [getting_started_production](https://github.com/stdapi-ai/samples/tree/main/getting_started_production)
- :material-earth: **Multi-region GDPR (EU)** — [getting_started_production_gdpr](https://github.com/stdapi-ai/samples/tree/main/getting_started_production_gdpr)
- :fontawesome-solid-flag-usa: **Multi-region US** — [getting_started_production_us](https://github.com/stdapi-ai/samples/tree/main/getting_started_production_us)
---
## :material-currency-usd: Cost-Optimized Deployment
!!! info "How costs scale"
By default, the Terraform module deploys **one ECS Fargate container per Availability Zone (AZ)**. Both AWS infrastructure costs (ECS/Fargate) and the stdapi.ai product fee (billed per container-hour, after the 14-day trial) are **proportional to the number of running containers** — so the number of AZs directly drives your bill.
To reduce costs: limit the number of subnets/AZs passed to the module, use Fargate Spot pricing, or schedule the service to stop outside business hours.
For development, side projects, and non-critical workloads.
??? example "Low cost deployment configuration"
```hcl
module "stdapi_ai_cost_optimized" {
source = "stdapi-ai/stdapi-ai/aws"
version = "~> 1.0"
# Aggressive Auto-scaling with Fargate spot
autoscaling_min_capacity = 1
autoscaling_max_capacity = 3
autoscaling_cpu_target_percent = 85
autoscaling_scale_in_cooldown = 60 # Scale down quickly
autoscaling_scale_out_cooldown = 120
autoscaling_spot_percent = 100 # Use 100% Spot pricing (~70% discount)
# Schedule: Stop at 7 PM, start at 8 AM on weekdays (UTC)
autoscaling_schedule_stop = "cron(0 19 ? * MON-FRI *)"
autoscaling_schedule_start = "cron(0 8 ? * MON-FRI *)"
# Use Existing Subnets and security group (no VPC creation)
subnet_ids = [
"subnet-0123456789abcdef0", # Your existing private subnet 1
"subnet-0123456789abcdef1", # Your existing private subnet 2
"subnet-0123456789abcdef2" # Your existing private subnet 3
]
security_group_id = "sg-0123456789abcdef0"
# Minimal Monitoring & Logging
container_insight = "disabled" # Disable Container Insights
vpc_flow_log_enabled = false # Disable VPC Flow Logs
cloudwatch_logs_retention_in_days = 7 # Reduce log retention to 7 days
}
```
**What you get:**
- Fargate Spot for significant cost reduction
- Minimal resources (0.25 vCPU, 512 MiB and ARM64 are the Terraform module defaults — not set explicitly in this example)
- Reuse existing VPC infrastructure
- Automated scheduling (runs 8 AM-7 PM weekdays only in UTC)
- Minimal logging (7-day retention, no Container Insights, no VPC Flow Logs)
**Trade-offs:** Spot interruptions possible, minimal observability, scheduled availability only
---
## :material-web-sync: WebSocket-Capable Deployment (Realtime API)
The [Realtime API](api_openai_realtime.md) holds a session open on a single WebSocket for up to 8 minutes. That changes what a normal HTTP-shaped deployment needs to get right — the points below are what the standard setup above does not already cover.
### The load balancer must allow the upgrade end to end
An Application Load Balancer forwards a WebSocket upgrade (`Connection: Upgrade`, `Upgrade: websocket`) to its target by default — nothing extra to enable there. Anything placed **in front of** the ALB (a CDN, a reverse proxy, an API Gateway) must pass those headers through unmodified, or the upgrade never reaches the gateway and every connection attempt fails before a single event is exchanged.
### The idle timeout bounds a session
The ALB's idle timeout closes a connection with no traffic for that long — and a Realtime session, once opened, can sit with no traffic between spoken turns. Keep `alb_idle_timeout` **at or above** the Realtime session limit (8 minutes / 480 seconds); the Terraform module's default of 3600 seconds already clears it. Lowering it below the session limit — or fronting the deployment with your own load balancer or reverse proxy at a shorter idle timeout — cuts sessions off mid-conversation with no `error` event, indistinguishable from a network failure. See [ALB Resilience](operations_resilience.md#alb-resilience) for the same setting's effect on ordinary streaming responses.
### Autoscale on CPU or memory, not request count
`ALBRequestCountPerTarget` counts a WebSocket connection as **one request for its entire duration** — an hour of active voice traffic on a handful of long-lived connections looks identical, to that metric, to an idle target. A fleet serving real Realtime traffic can read as underloaded and scale in while it is actually busy. Scale ECS on CPU or memory utilization instead wherever the service handles Realtime sessions; those track the work a session actually does.
### A deploy truncates open sessions
Replacing a task — a new deployment, a scale-in, a Spot interruption — ends whatever Realtime sessions that task is holding: there is no live handoff to another task. Two settings decide how abrupt that is:
- **Deregistration delay** on the target group gives a task being drained time to finish in-flight work before it is sent a `SIGTERM`; a value shorter than a typical session length simply means sessions running past it are cut regardless.
- **Stop timeout** on the ECS task caps how long the container gets to exit gracefully after `SIGTERM` before ECS sends `SIGKILL`.
Neither setting makes a session survive its task's replacement — the gateway holds session state in the process, not externally — so a deploy during active Realtime traffic will end some conversations. Schedule deploys for low-usage windows where that matters, and have clients reconnect on an unexpected close the same way they already do for the [8-minute session limit](api_openai_realtime.md#session-lifecycle-and-limits).
### AWS WAF's `NoUserAgent_HEADER` rule reads like an auth failure
The AWS-managed Common Rule Set (bundled by `alb_waf_enabled = true`) includes `NoUserAgent_HEADER`, which blocks any request with no `User-Agent` header. A raw WebSocket client — a bespoke SIP/telephony bridge, a minimal test script — that sends no `User-Agent` is blocked by the WAF with a `403` **before the request reaches the gateway at all**. Because the gateway itself also answers a rejected credential with `403`, the two are easy to conflate; check the WAF sampled requests (or disable `NoUserAgent_HEADER` for the Realtime path) before assuming the credential is wrong. Every mainstream WebSocket client library sets a `User-Agent` by default, so this only surfaces with a hand-rolled one.
### WebRTC and SIP need their own ingress
The Realtime API serves [one transport, the WebSocket](api_openai_realtime.md#transports) — an ordinary HTTPS connection, carried by the same listener as every other route. WebRTC and SIP are not: each negotiates a separate UDP media path, and an Application Load Balancer's listeners accept [only HTTP and HTTPS](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-listeners.html). No amount of configuration makes the module's ALB carry RTP or SIP, and that costs this deployment nothing, because nothing it serves needs UDP.
**Put the media terminator beside the gateway, not behind its load balancer.** A voice-agent framework such as LiveKit Agents or Pipecat, or a telephony bridge, faces the caller on its own address and reaches the gateway over the ordinary HTTPS ingress — so the gateway's networking, WAF and autoscaling stay exactly as documented above. Run the terminator in the same VPC, behind an internal load balancer, and that leg stays on the private path; the [WebSocket points above](#the-idle-timeout-bounds-a-session) then apply to the terminator as a client, not to the caller.
**If you publish a media terminator of your own**, it needs an ingress the module does not build, and two AWS services are the usual answers:
- A [Network Load Balancer with a `UDP` or `TCP_UDP` listener](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-listeners.html), or a task or instance with a public address and the media port range opened in its security group. Both **carry packets only** — neither decrypts DTLS, decodes RTP or performs ICE, so the media stack remains the terminator's own work. Note that a UDP target group is health-checked over [TCP or HTTP](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html), so a broken media path can pass its probe.
- [Amazon Chime SDK Voice Connector](https://docs.aws.amazon.com/chime-sdk/latest/ag/voice-connectors.html) for a phone leg. It terminates SIP trunking as a managed service, which is worth more here than on the WebRTC leg: the caller is a telephone, so there is no client-side API compatibility to preserve.
The detail that catches deployments out is **ICE addressing**: behind a load balancer the terminator's task sees only its private VPC address, and nothing tells it the address:port a client actually reached it on, so the candidate it advertises must be configured out of band. A public-addressed task is the cleanest shape for that and the least stable, since the address is reassigned every time the task is replaced.
---
## :material-hand-pointing-right: Manual ECS Deployment
Deploy the stdapi.ai container image directly to AWS ECS without Terraform.
### Prerequisites
1. **Subscribe to stdapi.ai** on [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-su2dajk5zawpo) (14-day free trial included)
2. Set up an ECS cluster (Fargate or EC2)
3. Configure networking (VPC, subnets, security groups)
4. Set up IAM roles with appropriate permissions
### Container Image
After subscribing, the container image is available from AWS Marketplace ECR:
```text
709825985650.dkr.ecr.us-east-1.amazonaws.com/j-goutin/stdapi.ai:
```
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
openai[" OpenAI SDK"] --> alb[" Your ALB/NLB"]
anthropic[" Anthropic SDK"] --> alb
alb --> ecs[" stdapi.ai ECS (Fargate/EC2)"]
ecr[" AWS Marketplace ECR"] --> ecs
ecs --> bedrock[" Amazon Bedrock"]
ecs --> polly[" Amazon Polly"]
ecs --> transcribe[" Amazon Transcribe"]
ecs --> s3[" Amazon S3"]
```
### ECS Task Definition Example
The example below uses ARM64 architecture, which requires the `-arm64` image tag. Replace `ARM64` with `X86_64` and `-arm64` with `-amd64` for AMD64. Use a version tag without an architecture suffix (e.g. `:1.15.0`) to let ECS select the architecture automatically via the multi-arch manifest.
```json
{
"family": "stdapi-ai-task-definition",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"],
"cpu": "256",
"memory": "512",
"executionRoleArn": "arn:aws:iam::{account-id}:role/{execution-role-name}",
"taskRoleArn": "arn:aws:iam::{account-id}:role/{task-role-name}",
"runtimePlatform": {
"cpuArchitecture": "ARM64",
"operatingSystemFamily": "LINUX"
},
"containerDefinitions": [
{
"name": "main",
"image": "709825985650.dkr.ecr.us-east-1.amazonaws.com/j-goutin/stdapi.ai:1.15.0-arm64",
"essential": true,
"readonlyRootFilesystem": true,
"user": "65532:65532",
"portMappings": [
{
"containerPort": 8000,
"protocol": "tcp",
"name": "http"
}
],
"environment": [
{
"name": "AWS_S3_BUCKET",
"value": "{your-s3-bucket-name}"
},
{
"name": "AWS_BEDROCK_REGIONS",
"value": "us-east-1,us-west-2"
}
],
"mountPoints": [
{
"sourceVolume": "temp",
"containerPath": "/tmp"
}
],
"healthCheck": {
"command": ["CMD", "python3", "-S", "-m", "stdapi.healthcheck"],
"interval": 30,
"timeout": 5,
"retries": 3,
"startPeriod": 30
},
"linuxParameters": {
"capabilities": {
"drop": ["ALL"]
}
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/stdapi-ai",
"awslogs-region": "{region}",
"awslogs-stream-prefix": "stdapi-ai"
}
}
}
],
"volumes": [
{
"name": "temp"
}
]
}
```
!!! tip "Declare the image's own health probe"
ECS ignores the image's `HEALTHCHECK`, so the task definition must re-declare it — the `healthCheck` above is that same command. It requests `/health` on the container's own port with a `Host` header derived from [`TRUSTED_HOSTS`](operations_configuration.md#trusted-hosts), so it keeps working when Host validation is enabled. A hand-written `curl` or `urllib` probe sends an untrusted `Host` and is rejected with `400`.
`"user": "65532:65532"` is the image's own non-root user, declared explicitly because Security Hub control ECS.20 reads the task definition rather than the image.
**Note:** This is a minimal example. For production, configure:
- Environment variables (see [Configuration](operations_configuration.md))
- IAM task roles for AWS service access
- Load balancer integration
- Auto-scaling policies
- CloudWatch monitoring
**Recommendation:** Use the [Terraform module](operations_getting_started.md#quick-start) for a complete, production-ready deployment with all best practices included.
---
## :material-export: Terraform Module Outputs
After deployment, access critical information:
```hcl
output "api_endpoint" {
value = module.stdapi_ai.alb_dns_name
}
```
**Networking & Load Balancing:**
- `alb_dns_name` — ALB endpoint (if enabled)
- `alb_arn` — ALB ARN for AWS integrations
- `alb_security_group_id` — ALB security group
- `application_url` — Full URL (https://domain or http://alb)
**ECS Service:**
- `cluster_name` — Cluster name for AWS CLI/SDK
- `service_name` — Service name for management
- `security_group_id` — Security group for ingress rules
- `service_discovery_service_name` — Private DNS name (if enabled)
- `port` — Container port exposed by the application
**Storage & Encryption:**
- `bucket_id` — S3 bucket for application data
- `bucket_arn` — S3 bucket ARN
- `kms_key_id` — KMS key for encryption
- `kms_key_arn` — KMS ARN for IAM policies
**Security:**
- `alb_waf_web_acl_id` — WAF ACL ID (if enabled)
- `alb_waf_web_acl_arn` — WAF ACL ARN (if enabled)
For the complete list of outputs, see [stdapi-ai/terraform-aws-stdapi-ai/outputs.tf](https://github.com/stdapi-ai/terraform-aws-stdapi-ai/blob/main/outputs.tf).
---
## :material-wrench: Troubleshooting
### VPC Endpoint Error: "couldn't find resource" for Amazon Comprehend
**Error message:**
```text
Error: reading EC2 VPC Endpoint Services: couldn't find resource
with module.stdapi_ai.module.vpc.data.aws_vpc_endpoint_service.netdev_vpce_interface["comprehend"],
on module-stdapi-ai/module-vpc/network_devices.tf line 175, in data "aws_vpc_endpoint_service" "netdev_vpce_interface":
175: data "aws_vpc_endpoint_service" "netdev_vpce_interface" {
```
**Cause:** Amazon Comprehend is not available as a VPC endpoint service in your current region.
**When this happens:** only on a fully private deployment — one where the module builds the VPC (no `subnet_ids`), `vpc_endpoints_allowed` is left at its default, `aws_bedrock_marketplace_auto_subscribe = false`, and every AWS service resolves to the deployment region. Any other configuration needs internet egress, so no interface endpoint is created and this error cannot occur.
**Solution:** Set the `aws_comprehend_region` variable to a region where Comprehend is available:
```hcl
module "stdapi_ai" {
source = "stdapi-ai/stdapi-ai/aws"
version = "~> 1.0"
aws_comprehend_region = "us-east-1"
}
```
Common regions with Comprehend support: `us-east-1`, `us-west-2`, `eu-west-1`, `eu-central-1`
!!! warning "This fix ends the fully private deployment"
Pointing any AWS service at another region makes the deployment cross-region, which requires internet egress. **Every** interface VPC endpoint is then dropped and a NAT gateway is provisioned instead — a change in both cost and network exposure, not just for Comprehend.
To keep the private posture, deploy into a region that offers a Comprehend VPC endpoint rather than pinning the service elsewhere.
---
## :material-arrow-right: Next Steps
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Standard deployment with Terraform
- :material-directions-fork: [**Resilience & Failover**](operations_resilience.md) — Multi-region routing and infrastructure resilience
- :material-cog: [**Configuration Reference**](operations_configuration.md) — Complete list of environment variables
- :material-scale-balance: [**Compliance**](operations_compliance.md) — Security and compliance requirements
---
## Configuration
Source: https://stdapi.ai/md/operations_configuration.md
# :material-cog: Configuration Guide
stdapi.ai is configured entirely through environment variables, which are read once at startup and cannot be changed without restarting the service. This guide explains each setting category with practical examples to help you configure the service correctly.
**What you can configure:**
- **AWS regions** - Access models across multiple regions for availability and model selection
- **Data sovereignty** - Control which AWS regions are used for compliance (GDPR, HIPAA, etc.)
- **Storage** - S3 buckets for file operations, regional buckets for multi-region deployments
- **Authentication** - API keys via SSM or Secrets Manager for secure access control
- **Observability** - Logging levels, OpenTelemetry, request/response debugging
- **Security** - CORS, proxy headers, trusted hosts for production deployments
- **Performance** - Caching, model overrides, S3 acceleration
- **TLS / SSL** - End-to-end encryption using Granian environment variables
!!! tip "Zero Configuration Startup"
stdapi.ai works out of the box with zero configuration. The service automatically detects your current AWS region and discovers available Bedrock models.
!!! info "Prerequisites"
Before configuring stdapi.ai, ensure you have:
- **AWS Account** with access to Amazon Bedrock
- **AWS Credentials** configured via environment variables, AWS CLI, or IAM role (for EC2/ECS/Lambda deployments)
- **IAM Permissions** to access required AWS services (see the [IAM Permissions](operations_iam_permissions.md) guide)
- **S3 Bucket** (optional, but recommended for production use with file operations)
!!! info "Container Runtime"
Both the AWS Marketplace and community Docker images run using [Granian](https://github.com/emmett-framework/granian), a high-performance Python ASGI server. In addition to the stdapi.ai-specific configuration variables documented below, you can also use Granian environment variables to configure the server runtime (e.g., `GRANIAN_PORT`, `GRANIAN_WORKERS`, `GRANIAN_THREADS`, etc.).
The images listen on IPv4 only (`GRANIAN_HOST=0.0.0.0`). Set `GRANIAN_HOST=::` to bind a dual-stack socket answering both IPv4 and IPv6 clients. This is needed wherever a client may resolve the server to an IPv6 address — in particular with ECS service discovery, which publishes an `AAAA` record for every task in an IPv6-enabled subnet, and some clients (Node.js among them) try that address first and fail with `ECONNREFUSED` against an IPv4-only listener. The [official Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) sets it for you when the VPC has IPv6 enabled.
## :material-rocket-launch: Quick Start
For production deployments, configure these essential settings:
### Minimal Production Setup
Single-region deployment with file storage only.
```bash
# S3 bucket for file storage (must be in same region as your server)
export AWS_S3_BUCKET=my-stdapi-bucket
# AWS_BEDROCK_REGIONS is optional - will auto-detect your current AWS region if not specified
```
### Production with Authentication
Adds secure API key authentication via AWS Systems Manager.
```bash
# S3 bucket for file storage (must be in same region as your server)
export AWS_S3_BUCKET=my-stdapi-bucket
# Secure API authentication (recommended: SSM Parameter Store)
export API_KEY_SSM_PARAMETER=/stdapi/prod/api-key
# AWS_BEDROCK_REGIONS is optional - will auto-detect your current AWS region if not specified
```
### Full Production Setup (All Features Enabled)
Multi-region deployment with all AWS AI services, observability, and security features.
```bash
# Core AWS configuration - host server in first region
export AWS_BEDROCK_REGIONS=us-east-1,us-west-2,eu-west-1
# S3 bucket for file storage (must be in us-east-1, your first/primary region)
export AWS_S3_BUCKET=my-stdapi-us-east-1-bucket
# Optional: Transcribe S3 bucket (defaults to AWS_S3_BUCKET if not specified)
# Only set this if you need a separate bucket or if transcribe is in a different region
# export AWS_TRANSCRIBE_S3_BUCKET=my-stdapi-transcribe-us-east-1
# Optional: Regional buckets for async/batch inference in other regions
export AWS_S3_REGIONAL_BUCKETS='{"us-west-2": "my-stdapi-us-west-2-bucket", "eu-west-1": "my-stdapi-eu-west-1-bucket"}'
# AWS AI services regions (optional - when unset, every AWS_BEDROCK_REGIONS entry is a
# candidate with automatic failover; set one to pin the service to a single region)
export AWS_POLLY_REGION=us-east-1 # Text-to-speech
export AWS_TRANSCRIBE_REGION=us-east-1 # Speech-to-text (audio transcription)
export AWS_COMPREHEND_REGION=us-east-1 # Language detection & moderation
export AWS_TRANSLATE_REGION=us-east-1 # Text translation
# Authentication
export API_KEY_SSM_PARAMETER=/stdapi/prod/api-key
# Logging
export LOG_LEVEL=warning
export LOG_CLIENT_IP=true
# Optional: OpenTelemetry observability (AWS X-Ray integration)
# export OTEL_ENABLED=true
# export OTEL_SERVICE_NAME=stdapi-production
# export OTEL_SAMPLE_RATE=0.1
# Production security settings (when behind AWS ALB/CloudFront)
export ENABLE_PROXY_HEADERS=true
# Note: TRUSTED_HOSTS not recommended with AWS ALB - use ALB host-based routing instead
# Only use TRUSTED_HOSTS if you cannot configure host validation at the load balancer level
# Optional: CORS for browser-based web applications
# export CORS_ALLOW_ORIGINS='["https://app.example.com"]'
```
### Development Setup
Local development configuration with API documentation and debug logging enabled.
```bash
# Minimal configuration for local development
export AWS_S3_BUCKET=my-stdapi-dev-bucket
# Enable API documentation
export ENABLE_DOCS=true
export ENABLE_REDOC=true
# Full request/response logging for debugging
export LOG_LEVEL=info
export LOG_REQUEST_PARAMS=true
# AWS_BEDROCK_REGIONS is optional - will auto-detect your current AWS region if not specified
```
!!! warning "S3 Bucket Required for Certain Features"
Without an S3 bucket configured, some features will be disabled (such as image output as URL, audio transcription). See the relevant API documentation for feature requirements.
!!! info "All Other Settings Are Optional"
The configurations above are sufficient for most production deployments. All other settings can be configured as needed for your specific use case.
## :material-format-list-bulleted: Environment Variable Summary
This section provides a quick reference of all available configuration options. Detailed explanations for each variable can be found in the sections below.
### :material-star: Essential (Production) { #summary-essential }
| Variable | Default | Description |
|-----------------------------------------------|----------------|--------------------------------------------------------------------------------------|
| [`AWS_S3_BUCKET`](#aws-s3-bucket) | None | Primary S3 bucket for file storage; must be in first region of `AWS_BEDROCK_REGIONS` |
| [`AWS_BEDROCK_REGIONS`](#aws-bedrock-regions) | Current region | Comma-separated regions for Bedrock; first region is where server should be hosted |
### :material-aws: AWS Client { #summary-aws-client }
| Variable | Default | Description |
|---------------------------------------------------------|---------|-------------------------------------------------------------------------------------------------------------|
| [`AWS_ADAPTIVE_RETRY`](#aws-adaptive-retry) | `false` | Enable adaptive retry mode that throttles back under congestion rather than using fixed exponential backoff |
| [`AWS_MAX_POOL_CONNECTIONS`](#aws-max-pool-connections) | `50` | Maximum concurrent HTTP connections per AWS service client |
| [`AWS_CONNECT_TIMEOUT`](#aws-connect-timeout) | `5` | Timeout in seconds for establishing a connection to an AWS service endpoint, and for a real-time audio session to become ready in a region |
### :material-database: AWS Storage { #summary-aws-storage }
| Variable | Default | Description |
|---------------------------------------------------------|-----------------|------------------------------------------------------------------------------------------------------|
| [`AWS_S3_ACCELERATE`](#aws-s3-accelerate) | `false` | Enable S3 Transfer Acceleration for faster global downloads via CloudFront edge locations |
| [`AWS_S3_REGIONAL_BUCKETS`](#aws-s3-regional-buckets) | `{}` | Region-specific S3 buckets for Bedrock async/batch inference operations |
| [`AWS_S3_ACCEPTED_BUCKETS`](#aws-s3-accepted-buckets) | `{}` | External S3 buckets with read access, mapped to their region for S3 URI conversion and routing |
| [`AWS_S3_TMP_PREFIX`](#aws-s3-tmp-prefix) | `tmp/` | S3 prefix for temporary files used for jobs; configure lifecycle policies on this prefix |
| [`AWS_S3_FILES_PREFIX`](#aws-s3-files-prefix) | `files/` | S3 prefix for Files API objects; configure S3 lifecycle policies on this prefix |
| [`AWS_S3_VIDEOS_PREFIX`](#aws-s3-videos-prefix) | `videos/` | S3 prefix for generated videos (Videos API); persists until deleted through the API |
| [`AWS_S3_VIDEOS_EXPIRES_AFTER`](#aws-s3-videos-expires-after) | None | Retention period in seconds for generated videos; sets `Video.expires_at` and blocks expired downloads |
| [`AWS_S3_BATCHES_PREFIX`](#aws-s3-batches-prefix) | `batches/` | S3 prefix for Batch API data (requests, results, batch records); configure lifecycle policies on it |
| [`AWS_S3_VECTORS_BUCKET`](#aws-s3-vectors-bucket) | None | Amazon S3 vector bucket backing the Vector Stores API; unset disables it |
| [`AWS_S3_VECTORS_REGION`](#aws-s3-vectors-region) | First Bedrock region | Region holding `AWS_S3_VECTORS_BUCKET`; the vector bucket has no failover |
| [`AWS_S3_VECTOR_STORES_PREFIX`](#aws-s3-vector-stores-prefix) | `vector_stores/` | S3 prefix for the Vector Stores API records (stores, attached files, batches) |
| [`VECTOR_STORE_EMBEDDING_MODEL`](#vector-store-embedding-model) | `amazon.titan-embed-text-v2:0` | Model embedding the indexed files and the search queries |
| [`VECTOR_STORE_CHUNK_SIZE_TOKENS`](#vector-store-chunk-size-tokens) | `800` | Default chunk size for files indexed without an explicit `chunking_strategy` |
| [`VECTOR_STORE_CHUNK_OVERLAP_TOKENS`](#vector-store-chunk-overlap-tokens) | `400` | Default chunk overlap; must not exceed half the chunk size |
| [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](#aws-sqs-vector-store-queue-url) | None | Amazon SQS queue making vector store indexing survive the server running it; unset keeps it in-process |
| [`AWS_BEDROCK_KNOWLEDGE_BASE_IDS`](#aws-bedrock-knowledge-base-ids) | `[]` | Allowlist of Amazon Bedrock knowledge bases addressed as `vs_kb_...` vector stores; empty disables it |
| [`AWS_TRANSCRIBE_S3_BUCKET`](#aws-transcribe-s3-bucket) | `AWS_S3_BUCKET` | S3 bucket for temporary audio transcription files; must be in same region as `AWS_TRANSCRIBE_REGION` |
| [`AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN`](#aws-transcribe-output-encryption-key-arn) | None | AWS KMS key encrypting the transcription output objects; unset keeps the bucket's own encryption |
| [`AWS_TRANSCRIBE_STREAM_LANGUAGES`](#aws-transcribe-stream-languages) | `[]` | Languages a streamed transcription picks between when the request names none |
### :material-robot: AWS AI Services { #summary-aws-ai-services }
| Variable | Default | Description |
|---------------------------------------------------|-----------------------------|-------------------------------------------------------------|
| [`AWS_POLLY_REGION`](#aws-polly-region) | All `AWS_BEDROCK_REGIONS` | Region for Amazon Polly; unset = per-engine regional discovery with automatic failover |
| [`AWS_COMPREHEND_REGION`](#aws-comprehend-region) | All `AWS_BEDROCK_REGIONS` | Region for Amazon Comprehend (language detection, toxicity moderation); unset = automatic failover across all Bedrock regions |
| [`AWS_TRANSCRIBE_REGION`](#aws-transcribe-region) | All `AWS_BEDROCK_REGIONS` | Region for Amazon Transcribe; unset = failover across Bedrock regions with a co-located bucket |
| [`AWS_TRANSLATE_REGION`](#aws-translate-region) | All `AWS_BEDROCK_REGIONS` | Region for Amazon Translate; unset = automatic failover across all Bedrock regions |
### :material-directions-fork: Resilience & Failover { #summary-resilience-failover }
| Variable | Default | Description |
|---------------------------------------------------------------------------------------------------------|-----------|--------------------------------------------------------------------------------------------------------------------------|
| [`AWS_BEDROCK_REGION_ROUTING`](#bedrock-region-routing) | `ordered` | Region routing strategy: `disabled`, `ordered`, `lowest_latency`, or `round_robin` ([details](operations_resilience.md)) |
| [`AWS_BEDROCK_REGION_ROUTING_QUOTA_BACKOFF_SECONDS`](#bedrock-region-routing-quota-backoff) | `60` | Base interval in seconds for exponential quota backoff per region |
| [`AWS_BEDROCK_REGION_ROUTING_MAX_QUOTA_BACKOFF_SECONDS`](#bedrock-region-routing-max-quota-backoff) | `3600` | Hard ceiling in seconds on the exponential quota backoff per region (default: 1 hour) |
| [`AWS_BEDROCK_REGION_ROUTING_QUOTA_STALE_FACTOR`](#bedrock-region-routing-quota-stale-factor) | `2` | Multiplier on max quota backoff to determine when the consecutive-error counter resets |
| [`AWS_BEDROCK_REGION_ROUTING_UNAVAILABLE_BACKOFF_SECONDS`](#bedrock-region-routing-unavailable-backoff) | `30` | Seconds to avoid a region after unavailability errors |
| [`AWS_BEDROCK_MAX_RETRIES`](#bedrock-max-retries) | `9` | Cap on the retries per Bedrock invocation; with region routing, each candidate region is tried at most once |
| [`AWS_FAILOVER_MAX_RETRIES`](#failover-max-retries) | `2` | SDK retries per candidate region for the multi-region failover services (Polly, Transcribe, Translate, Comprehend) |
### :material-layers-triple: Bedrock Mantle { #summary-bedrock-mantle }
| Variable | Default | Description |
|-----------------------------------------------------------------------------|-----------------------|------------------------------------------------------------------------------------------------------|
| [`AWS_BEDROCK_MANTLE_ENABLED`](#bedrock-mantle-enabled) | `true` | Expose models served by the Amazon Bedrock Mantle endpoint alongside classic Bedrock Converse models |
| [`AWS_BEDROCK_MANTLE_REGIONS`](#bedrock-mantle-regions) | Mantle-capable subset of `AWS_BEDROCK_REGIONS` | AWS regions used for Bedrock Mantle, in failover priority order |
| [`AWS_BEDROCK_MANTLE_ENDPOINT_URL`](#bedrock-mantle-endpoint-url) | None | Override the Bedrock Mantle endpoint URL template (`{region}` placeholder) |
| [`AWS_BEDROCK_MANTLE_PREFERRED_MODELS`](#bedrock-mantle-preferred-models) | `[]` | Model IDs served via Mantle even when also available on the classic bedrock-runtime endpoint |
| [`AWS_BEDROCK_MANTLE_SERVICE_HEADER`](#bedrock-mantle-service-header) | `false` | Honor the `x-stdapi-service: bedrock-mantle` request header to route dual-homed models through Mantle per request |
| [`AWS_BEDROCK_MANTLE_PROJECT`](#bedrock-mantle-project) | None | Default Bedrock Project/Workspace ID applied to Mantle requests for cost tracking and observability |
| [`AWS_BEDROCK_ALLOW_MANTLE_PROJECT_OVERRIDE`](#bedrock-allow-mantle-project-override) | `false` | Allow requests to override the configured Mantle project via the `OpenAI-Project` / `anthropic-workspace` header |
| [`AWS_BEDROCK_EXTERNAL_WEB_ACCESS`](#bedrock-external-web-access) | `false` | Let the built-in web search tool reach the public web instead of the Amazon Bedrock web index |
| [`AWS_BEDROCK_ALLOW_EXTERNAL_WEB_ACCESS_OVERRIDE`](#bedrock-allow-external-web-access-override) | `false` | Allow requests to override external web access with the `external_web_access` extra model parameter |
### :material-shield-check: Bedrock Advanced { #summary-bedrock-advanced }
| Variable | Default | Description |
|---------------------------------------------------------------------------------------------------|---------|-----------------------------------------------------------------------------------------------------|
| [`AWS_BEDROCK_CROSS_REGION_INFERENCE`](#cross-region-inference) | `true` | Allow automatic model routing to other configured regions |
| [`AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL`](#cross-region-global) | `true` | Allow global cross-region inference routing to any region worldwide (disable for GDPR compliance) |
| [`AWS_BEDROCK_MODEL_REGION_RESTRICT`](#bedrock-model-region-restrict) | `{}` | Restrict a model to specific region(s) only (e.g. for region-specific features like Nova grounding) |
| [`AWS_BEDROCK_LEGACY`](#bedrock-legacy) | `false` | Allow usage of deprecated/legacy Bedrock models |
| [`AWS_BEDROCK_DEPRECATED_MODEL_FALLBACK`](#bedrock-deprecated-model-fallback) | `true` | Transparently reroute requests using a deprecated model ID to its recommended replacement |
| [`AWS_BEDROCK_DEPRECATED_MODELS`](#bedrock-deprecated-models) | `{}` | Additional deprecated model mappings merged with the built-in registry at startup |
| [`AWS_BEDROCK_MARKETPLACE_AUTO_SUBSCRIBE`](#bedrock-marketplace-auto-subscribe) | `true` | Allow automatic subscription to new models in AWS Marketplace |
| [`AWS_BEDROCK_ALLOW_CROSS_REGION_INFERENCE_PROFILE_ARN`](#bedrock-allow-cross-region-profile-arn) | `false` | Allow users to pass cross-region inference profile ARNs directly as model IDs |
| [`AWS_BEDROCK_ALLOW_APPLICATION_INFERENCE_PROFILE_ARN`](#bedrock-allow-application-profile-arn) | `false` | Allow users to pass application inference profile ARNs directly as model IDs |
| [`AWS_BEDROCK_ALLOW_PROMPT_ROUTER_ARN`](#bedrock-allow-prompt-router-arn) | `false` | Allow users to pass prompt router ARNs directly as model IDs |
| [`AWS_BEDROCK_ALLOW_PROMPT_ARN`](#bedrock-allow-prompt-arn) | `false` | Allow users to reference Prompt Management prompt ARNs in the Responses API `prompt` parameter |
| [`AWS_BEDROCK_MODEL_ARN_MAPPING`](#bedrock-model-arn-mapping) | `{}` | Map model IDs to custom inference profile or prompt router ARNs (server-controlled routing) |
| [`AWS_BEDROCK_GUARDRAIL_IDENTIFIER`](#aws-bedrock-guardrail-identifier) | None | Bedrock Guardrails ID for content filtering and safety controls |
| [`AWS_BEDROCK_GUARDRAIL_VERSION`](#aws-bedrock-guardrail-version) | None | Bedrock Guardrails version number (required with identifier) |
| [`AWS_BEDROCK_GUARDRAIL_TRACE`](#aws-bedrock-guardrail-trace) | None | Guardrails trace level: `disabled`, `enabled`, or `enabled_full` |
| [`AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE`](#aws-bedrock-allow-guardrail-override) | `false` | Allow users to override global guardrail configuration via request headers (security: default off) |
| [`AWS_BEDROCK_SESSION_ENCRYPTION_KEY_ARN`](#aws-bedrock-session-encryption-key-arn) | None | KMS key ARN encrypting Amazon Bedrock session storage (Responses API `store=true`) |
| [`AWS_BEDROCK_BATCH_ROLE_ARN`](#aws-bedrock-batch-role-arn) | None | Service role Amazon Bedrock assumes to run batch inference jobs; unset disables the Batch APIs |
| [`AWS_BEDROCK_USER_ROLE_ARN`](#aws-bedrock-user-role-arn) | None | Run each end user's model calls under a role session of their own, so AWS reports their spend separately |
| [`AWS_BEDROCK_USER_ROLE_SESSION_DURATION`](#aws-bedrock-user-role-session-duration) | `3600` | Lifetime in seconds of a per-end-user role session (900–3600) |
| [`AWS_BEDROCK_USER_ROLE_TAG_KEY`](#aws-bedrock-user-role-tag-key) | `user` | Session tag key carrying the end user identity, for cost allocation and access policies |
| [`AWS_BEDROCK_USER_ROLE_REQUIRE_IDENTITY`](#aws-bedrock-user-role-require-identity) | `false` | Reject a model request that identifies no end user instead of billing it to the server |
### :material-lock: Authentication { #summary-authentication }
Configure **one** API key source. If several are set, precedence is `API_KEY` → SSM Parameter Store → Secrets Manager — see [Authentication](#authentication):
| Variable | Default | Description |
|-------------------------------------------------------------------|-----------|--------------------------------------------------------------------|
| [`API_KEY_SSM_PARAMETER`](#api-key-ssm) | None | AWS Systems Manager Parameter Store path for API key (recommended) |
| [`API_KEY_SECRETSMANAGER_SECRET`](#api-key-secretsmanager-secret) | None | AWS Secrets Manager secret name containing API key |
| [`API_KEY_SECRETSMANAGER_KEY`](#api-key-secretsmanager-key) | `api_key` | JSON key name within Secrets Manager secret |
| [`API_KEY`](#api-key) | None | Direct API key value (not recommended for production) |
| [`AUTHENTICATION_MODE`](#authentication-mode) | `any` | Accepted methods: `any`, `api_key` or `cognito` |
Amazon Cognito user pool tokens are an alternative to the API key — see [Amazon Cognito Authentication](#cognito-authentication):
| Variable | Default | Description |
|---------------------------------------------------------------------|------------|-----------------------------------------------------------------|
| [`AWS_COGNITO_USER_POOL_ID`](#aws-cognito-user-pool-id) | None | User pool whose tokens authenticate clients (enables the method) |
| [`AWS_COGNITO_CLIENT_IDS`](#aws-cognito-client-ids) | None | App client IDs whose tokens are accepted (required with a pool) |
| [`AWS_COGNITO_REQUIRED_SCOPES`](#aws-cognito-required-scopes) | None | Scopes a token must all carry |
| [`AWS_COGNITO_ACCEPT_ID_TOKEN`](#aws-cognito-accept-id-token) | `false` | Also accept identity tokens, not only access tokens |
| [`AWS_COGNITO_ISSUER_TYPE`](#aws-cognito-issuer-type) | `original` | Pool issuer configuration: `original` or `updated` |
Publishing where tokens come from lets an AI agent authenticate itself — see [Authentication Discovery](#oauth-discovery):
| Variable | Default | Description |
|-----------------------------------------------------------------------|----------------------|--------------------------------------------------------------------|
| [`OAUTH_RESOURCE_IDENTIFIER`](#oauth-resource-identifier) | None | Public URL clients dial (publishes the discovery document) |
| [`OAUTH_AUTHORIZATION_SERVERS`](#oauth-authorization-servers) | The user pool issuer | Issuer URLs of the authorization servers |
| [`OAUTH_SCOPES_SUPPORTED`](#oauth-scopes-supported) | Required scopes | Scopes a token needs, advertised to clients |
### :material-api: API Compatibility { #summary-api-compatibility }
| Variable | Default | Description |
|-------------------------------------------------------|--------------|------------------------------------------------------|
| [`OPENAI_ROUTES_PREFIX`](#openai-routes-prefix) | None (root) | Base path prefix for OpenAI-compatible API routes |
| [`ANTHROPIC_ROUTES_PREFIX`](#anthropic-routes-prefix) | `/anthropic` | Base path prefix for Anthropic-compatible API routes |
| [`COHERE_ROUTES_PREFIX`](#cohere-routes-prefix) | `/cohere` | Base path prefix for Cohere-compatible API routes |
### :material-chart-line: Logging { #summary-logging }
| Variable | Default | Description |
|-------------------------------------------------|---------|---------------------------------------------------------------------------------------|
| [`LOG_LEVEL`](#logging-level) | `info` | Minimum log severity: `info`, `warning`, `error`, `critical`, or `disabled` |
| [`LOG_REQUEST_PARAMS`](#log-request-params) | `false` | Include request/response parameters in logs (not recommended for production) |
| [`LOG_CLIENT_IP`](#client-ip-logging) | `false` | Log client IP addresses (requires `ENABLE_PROXY_HEADERS` for real IPs behind proxies) |
### :material-chart-box-outline: CloudWatch Metrics { #summary-cloudwatch-metrics }
| Variable | Default | Description |
|-----------------------------------------------------------------------|----------|--------------------------------------------------------------------|
| [`CLOUDWATCH_METRICS`](#cloudwatch-metrics) | `false` | Emit per-request AWS-billed usage as CloudWatch EMF log lines |
| [`CLOUDWATCH_METRICS_NAMESPACE`](#cloudwatch-metrics-namespace) | `stdapi` | CloudWatch namespace for the emitted usage metrics |
### :material-currency-usd: Cost Tracking { #summary-cost-tracking }
| Variable | Default | Description |
|-----------------------------------------------------------------|----------------|-----------------------------------------------------------------------|
| [`COST_TRACKING`](#cost-tracking) | `false` | Enable real-time cost computation from live AWS pricing |
| [`COST_PRICE_OVERRIDES`](#cost-price-overrides) | `{}` | JSON map of operator-supplied unit prices for models missing from the AWS catalog |
### :material-radar: Observability (OpenTelemetry) { #summary-observability }
| Variable | Default | Description |
|-----------------------------------------------------|-----------------------------------|----------------------------------------------------------------------------------------|
| [`OTEL_ENABLED`](#otel-enabled) | `false` | Enable distributed tracing via OpenTelemetry (integrates with AWS X-Ray, Jaeger, etc.) |
| [`OTEL_SERVICE_NAME`](#otel-service-name) | `stdapi.ai` | Service name identifier in trace visualizations |
| [`OTEL_EXPORTER_ENDPOINT`](#otel-exporter-endpoint) | `http://127.0.0.1:4318/v1/traces` | OTLP HTTP endpoint URL for trace export |
| [`OTEL_SAMPLE_RATE`](#otel-sample-rate) | `1.0` | Trace sampling rate from 0.0 (none) to 1.0 (all requests) |
### :material-web: HTTP/Security { #summary-http-security }
| Variable | Default | Description |
|-------------------------------------------------------------------------------------|----------|---------------------------------------------------------------------------------------|
| [`CORS_ALLOW_ORIGINS`](#cors-allow-origins) | None | JSON array of allowed origins for browser cross-origin requests |
| [`TRUSTED_HOSTS`](#trusted-hosts) | None | JSON array of trusted Host header values (prefer ALB host-based routing; see details) |
| [`ENABLE_PROXY_HEADERS`](#enable-proxy-headers) | `false` | Trust X-Forwarded-* headers from reverse proxies (only enable behind trusted proxy) |
| [`PROXY_TRUSTED_HOSTS`](#proxy-trusted-hosts) | `*` | Peer IPs/ranges whose X-Forwarded-* headers are trusted (restrict from `*` for safety) |
| [`GRANIAN_HOST`](#granian-host) | `0.0.0.0` | Listener bind address; `::` binds a dual-stack socket answering IPv4 and IPv6 clients |
| [`GRANIAN_SSL_CERTIFICATE`](#graniansslcertificate) | None | Path to SSL certificate file for end-to-end encryption |
| [`GRANIAN_SSL_KEYFILE`](#graniansslkeyfile) | None | Path to SSL private key file (PKCS#8) for end-to-end encryption |
| [`GRANIAN_SSL_KEYFILE_PASSWORD`](#graniansslkeyfilepassword) | None | Password for the SSL private key file |
| [`GRANIAN_SSL_PROTOCOL_MIN`](#graniansslprotocolmin) | `tls1.3` | Minimum supported TLS version (`tls1.2` or `tls1.3`) |
| [`GRANIAN_SSL_CA`](#graniansslca) | None | Path to CA certificate bundle for client verification (mTLS) |
| [`GRANIAN_SSL_CLIENT_VERIFY`](#graniansslclientverify) | `false` | Enable client certificate verification (mTLS) |
| [`ENABLE_GZIP`](#enable-gzip) | `false` | Enable GZip compression for responses >1KB (prefer AWS ALB/CloudFront compression) |
| [`SSRF_PROTECTION_BLOCK_PRIVATE_NETWORKS`](#ssrf-protection-block-private-networks) | `true` | Block requests to private/local networks for SSRF protection |
| [`MAX_INPUT_FILE_SIZE`](#max-input-file-size) | `0` | Maximum size in bytes of an inline input file loaded into memory (`0` disables) |
| [`MAX_CONCURRENT_INPUT_DOWNLOADS`](#max-concurrent-input-downloads) | `8` | Maximum input files fetched/resolved concurrently per request |
### :material-cog: Application Behavior { #summary-application-behavior }
| Variable | Default | Description |
|---------------------------------------------------------------------|-------------------------|--------------------------------------------------------------------------------------------|
| [`TIMEZONE`](#timezone) | `UTC` | IANA timezone identifier for request timestamps |
| [`STRICT_INPUT_VALIDATION`](#strict-input-validation) | `false` | Reject API requests with unknown/extra fields |
| [`CHAT_COMPLETIONS_REASONING_FIELD`](#chat-completions-reasoning-field) | `reasoning_content` | Field carrying reasoning text on `/v1/chat/completions`: `reasoning_content`, `reasoning`, or `none` |
| [`MODEL_ALIASES`](#model-aliases) | `{}` | JSON object mapping custom model name aliases to Bedrock model IDs, optionally with per-alias configuration |
| [`DEFAULT_TTS_MODEL`](#default-tts-model) | `amazon.polly-standard` | Default TTS model: `amazon.polly-standard`, `-neural`, `-long-form`, or `-generative` |
| [`DEFAULT_TTS_LANGUAGE`](#default-tts-language) | None | Default language for TTS (e.g., `en-US`); when set, skips Amazon Comprehend auto-detection |
| [`TOKENS_ESTIMATION`](#tokens-estimation) | `false` | Deprecated and ignored (token estimation removed) |
| [`TOKENS_ESTIMATION_DEFAULT_ENCODING`](#tokens-encoding) | `None` | Deprecated and ignored (token estimation removed) |
| [`DEFAULT_MODEL_PARAMS`](#default-model-params) | `{}` | JSON object with per-model default inference parameters (temperature, max_tokens, etc.) |
| [`DEFAULT_MODEL_SERVICE_TIERS`](#default-model-service-tiers) | `{}` | JSON object with per-model default service tiers (default, flex, priority, reserved) |
| [`AWS_BEDROCK_ALLOW_SERVICE_TIER_OVERRIDE`](#aws-bedrock-allow-service-tier-override) | `true` | Allow users to select the service tier per request, overriding the configured one (cost control) |
| [`MODEL_CACHE_SECONDS`](#model-cache-seconds) | `900` | Model list cache lifetime in seconds before lazy refresh (default: 15 minutes) |
| [`AI_RESPONSE_TIMEOUT`](#ai-response-timeout) | `600` | Maximum seconds without data from a model before the request times out (default: 10 min) |
| [`SHUTDOWN_DRAIN_TIMEOUT`](#shutdown-drain-timeout) | `10` | Maximum seconds the server waits for background work to finish after being asked to stop |
| [`DROP_UNSUPPORTED_SYSTEM_PROMPT`](#drop-unsupported-system-prompt) | `true` | Drop system prompts for unsupported models; when `false`, return error instead |
| [`ANTHROPIC_BETA_FILTER`](#anthropic-beta-filter) | `true` | Enable filtering of unsupported `anthropic_beta` flags for Claude models |
| [`ANTHROPIC_BETA_ALLOWLIST`](#anthropic-beta-allowlist) | None | Additional `anthropic_beta` flags to allow beyond built-in Bedrock defaults |
| [`EXTRA_MODEL_PARAMS_DENYLIST`](#extra-model-params-denylist) | None | Additional "extra model parameters" names to strip, beyond the built-in LiteLLM control-parameter denylist |
| [`EXTRA_MODEL_PARAMS_DROP_ALL`](#extra-model-params-drop-all) | `false` | Disable the "extra model parameters" passthrough entirely |
| [`IMAGE_GENERATION_MODEL`](#image-generation-model) | None | Default Bedrock image model ID used when the `image_generation` Responses API tool is invoked |
| [`REALTIME_CLIENT_SECRET_KEY`](#realtime-client-secret-key) | None | Secret the Realtime API's ephemeral client secrets are signed with; derived from the API key when unset |
| [`REALTIME_ALLOW_SESSION_OVERRIDE`](#realtime-allow-session-override) | `true` | Allow a client holding an ephemeral client secret to override the session configuration it carries |
### :material-file-document: API Documentation { #summary-api-documentation }
| Variable | Default | Description |
|-----------------------------------------------|---------|----------------------------------------------------------------------------------|
| [`ENABLE_DOCS`](#enable-docs) | `false` | Enable interactive Swagger UI documentation at `/docs` |
| [`ENABLE_REDOC`](#enable-redoc) | `false` | Enable ReDoc documentation UI at `/redoc` |
| [`ENABLE_OPENAPI_JSON`](#enable-openapi-json) | `false` | Enable OpenAPI schema endpoint at `/openapi.json` (auto-enabled with docs/redoc) |
### :material-connection: MCP (Model Context Protocol) { #summary-mcp }
| Variable | Default | Description |
|---------------------------------------------------------------------|---------|-----------------------------------------------------------------------------------------|
| [`ENABLE_MCP_STREAMABLE_HTTP`](#enable-mcp-streamable-http) | `false` | Enable MCP server via Streamable HTTP at `/mcp` — recommended transport |
| [`MCP_STATELESS_HTTP`](#mcp-stateless-http) | `false` | Serve `/mcp` without server-side sessions — any replica may serve any request |
| [`ENABLE_MCP_SSE`](#enable-mcp-sse) | `false` | Enable MCP server via Server-Sent Events at `/sse` — legacy transport for older clients |
| [`MCP_INCLUDE_TOOLS`](#mcp-include-tools) | None | Comma-separated tool names to expose exclusively; all others are hidden |
| [`MCP_EXCLUDE_TOOLS`](#mcp-exclude-tools) | None | Comma-separated tool names to hide; all others remain exposed |
---
## :material-aws: AWS Services and Regions
### General Configuration
#### `AWS_ADAPTIVE_RETRY` { #aws-adaptive-retry }
:octicons-package-24: **Purpose**
: Enable adaptive retry mode that adjusts retry pacing based on observed error rates across all AWS service calls
:octicons-database-24: **Type**
: Boolean (`true` / `false`)
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Behavior**
: When enabled, the retry strategy dynamically responds to real-time congestion signals. If errors are occurring frequently, retries are spaced further apart to avoid amplifying load on an already-stressed endpoint. Once conditions improve, the pacing returns to normal. When disabled, retries follow a standard exponential backoff strategy with fixed intervals. Applies to all AWS services (Bedrock, S3, Polly, Transcribe, etc.).
!!! warning "Latency Impact"
Adaptive retry paces retries based on real-time error signals, reducing the risk of retry storms when many clients share the same endpoint under sustained congestion — at the cost of increased per-request latency when throttling is detected, since the client intentionally delays retries to shed load. Prefer it under sustained high load; keep the default standard mode for latency-sensitive, low-traffic workloads.
```bash
# Default: standard exponential backoff
export AWS_ADAPTIVE_RETRY=false
# Enable adaptive retry (recommended under sustained high load)
export AWS_ADAPTIVE_RETRY=true
```
#### `AWS_MAX_POOL_CONNECTIONS` { #aws-max-pool-connections }
:octicons-package-24: **Purpose**
: Maximum number of concurrent HTTP connections per AWS service client
:octicons-database-24: **Type**
: Integer (must be > 0)
:octicons-gear-24: **Default**
: `50`
:octicons-workflow-24: **Behavior**
: Each AWS service client (one per service per region) maintains its own connection pool up to this limit. Under high concurrency, increasing this value prevents requests from queuing for an available connection. Setting it too high may exhaust system file descriptors.
```bash
# Default
export AWS_MAX_POOL_CONNECTIONS=50
# High-concurrency deployment
export AWS_MAX_POOL_CONNECTIONS=100
```
#### `AWS_CONNECT_TIMEOUT` { #aws-connect-timeout }
:octicons-package-24: **Purpose**
: Timeout in seconds for establishing a connection to an AWS service endpoint
:octicons-database-24: **Type**
: Integer (must be > 0)
:octicons-gear-24: **Default**
: `5`
:octicons-workflow-24: **Behavior**
: Limits how long the client waits when opening a new connection. It also bounds, per candidate region, the time a real-time audio session (generative-voice speech, speech-to-speech) may take to become ready: connection, initial handshake and the first response together. A short value allows fast failover to another region when an endpoint is unreachable. Increase it only if you see spurious connection timeouts on high-latency networks, or real-time audio requests failing with a `503` a few seconds after they start.
```bash
# Default: 5 seconds
export AWS_CONNECT_TIMEOUT=5
# High-latency network
export AWS_CONNECT_TIMEOUT=10
```
### Storage Configuration
#### `AWS_S3_BUCKET` { #aws-s3-bucket }
:octicons-package-24: **Purpose**
: Primary S3 bucket for storing generated files (images, audio, documents) and temporary data during processing
:octicons-gear-24: **Default**
: None (must be configured for file operations)
:octicons-check-circle-24: **Best Practice**
: The bucket must be in the first region specified in `AWS_BEDROCK_REGIONS` (your primary region where the server should be hosted) to avoid cross-region data transfer costs and reduce latency
```bash
export AWS_S3_BUCKET=my-llm-storage-us-east-1
```
!!! tip "Presigned URLs"
Files are served via presigned URLs for secure, time-limited access. Presigned URLs expire after 1 hour.
!!! info "Terraform Module"
When using the Terraform module, the main S3 bucket is created automatically — no manual configuration required.
!!! warning "Startup Warning"
If not set, a warning is logged at startup and features that require file storage (image generation, audio output, document processing) will be unavailable.
#### `AWS_S3_ACCELERATE` { #aws-s3-accelerate }
:octicons-package-24: **Purpose**
: Enable S3 Transfer Acceleration for presigned URLs to improve download performance for large files
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
:octicons-check-circle-24: **Best Practice**
: Enable when serving large files (high-resolution images, audio) to geographically distributed users
```bash
export AWS_S3_ACCELERATE=true
```
!!! info "What is S3 Transfer Acceleration?"
S3 Transfer Acceleration uses Amazon CloudFront's globally distributed edge locations to accelerate uploads and downloads to S3 buckets. When enabled, data is routed to the nearest edge location and then transferred to S3 over Amazon's optimized network paths.
**Performance Benefits:**
- :material-speedometer: **Faster downloads** for users far from your bucket's region
- :material-earth: **Global reach** via CloudFront edge locations
- :material-upload-network: **Optimized routing** over Amazon's private backbone network
- :material-chart-line: **Consistent performance** regardless of user location
Typical speed improvements: 50-500% faster for users located far from the bucket region.
!!! warning "Requirements"
1. **Enable Transfer Acceleration** on your S3 bucket before setting this option:
```bash
aws s3api put-bucket-accelerate-configuration \
--bucket my-stdapi-bucket \
--accelerate-configuration Status=Enabled
```
2. **Additional costs**: Transfer Acceleration incurs extra data transfer fees. See [Amazon S3 Transfer Acceleration pricing](https://aws.amazon.com/s3/pricing/)
!!! tip "When to Enable"
Consider enabling S3 Transfer Acceleration when:
- :material-image: Serving generated images via [Images API](api_openai_images_generations.md)
- :material-earth-arrow-right: Users are geographically distributed across multiple continents
- :material-file-image: Generating high-resolution images that are large in file size
- :material-speedometer: Download performance is critical to user experience
For small images or users close to your bucket region, the performance benefit may not justify the additional cost.
!!! info "Current Usage"
Presigned URLs with Transfer Acceleration are currently only used for the [Images API](api_openai_images_generations.md) when returning generated images as URLs.
#### `AWS_S3_TMP_PREFIX` { #aws-s3-tmp-prefix }
:octicons-package-24: **Purpose**
: S3 prefix (folder path) for temporary files used during job processing
:octicons-gear-24: **Default**
: `tmp/`
:octicons-check-circle-24: **Best Practice**
: Configure S3 lifecycle policies to automatically delete objects under this prefix after 1 day
```bash
export AWS_S3_TMP_PREFIX=tmp/
```
!!! info "What is an S3 Prefix?"
An S3 prefix is essentially a folder path within your S3 bucket. When you set `AWS_S3_TMP_PREFIX=tmp/`, all temporary files are stored under the `tmp/` folder structure in your bucket.
**Example file paths:**
- With prefix `tmp/`: `s3://my-bucket/tmp/request-id-123/output.json`
- With prefix `temporary/`: `s3://my-bucket/temporary/request-id-123/output.json`
- With empty prefix ``: `s3://my-bucket/request-id-123/output.json` (not recommended)
!!! tip "Why Use a Prefix?"
Using a dedicated prefix for temporary files provides several benefits:
- :material-auto-fix: **Easy Lifecycle Management** - Apply S3 lifecycle policies to automatically delete only temporary files
- :material-file-tree: **Better Organization** - Keep temporary files separate from permanent storage
- :material-shield-check: **Security** - Apply different IAM policies or bucket policies to the prefix
- :material-cash: **Cost Control** - Easily identify and monitor temporary storage costs
!!! warning "Trailing Slash"
Always include a trailing slash (`/`) in your prefix to create a proper folder structure. Without it, files will be stored with the prefix as part of the filename rather than in a folder.
- ✅ Correct: `tmp/` → Files stored as `tmp/file.json`
- ❌ Incorrect: `tmp` → Files stored as `tmpfile.json`
**Custom prefix examples:**
```bash
# Production environment
export AWS_S3_TMP_PREFIX=prod/tmp/
# Staging environment
export AWS_S3_TMP_PREFIX=staging/tmp/
# Organize by date (requires manual updates)
export AWS_S3_TMP_PREFIX=tmp/2025/01/
# No prefix (store at bucket root - not recommended)
export AWS_S3_TMP_PREFIX=
```
#### `AWS_S3_FILES_PREFIX` { #aws-s3-files-prefix }
:octicons-package-24: **Purpose**
: S3 prefix (folder path) for Files API objects (OpenAI and Anthropic `/v1/files` endpoints)
:octicons-gear-24: **Default**
: `files/`
:octicons-check-circle-24: **Best Practice**
: Configure an `AbortIncompleteMultipartUpload` S3 lifecycle rule on this prefix to clean up abandoned upload parts, and apply Intelligent-Tiering for cost optimisation
```bash
export AWS_S3_FILES_PREFIX=files/
```
!!! info "S3 Prefix Format"
Prefix semantics (folder-style paths, trailing-slash requirement) are explained under [`AWS_S3_TMP_PREFIX`](#aws-s3-tmp-prefix) and apply here identically.
**Custom prefix examples:**
```bash
# Production environment
export AWS_S3_FILES_PREFIX=prod/files/
# Staging environment
export AWS_S3_FILES_PREFIX=staging/files/
# No prefix (store at bucket root - not recommended)
export AWS_S3_FILES_PREFIX=
```
#### `AWS_S3_VIDEOS_PREFIX` { #aws-s3-videos-prefix }
:octicons-package-24: **Purpose**
: S3 prefix (folder path) for videos generated through the [Videos API](api_openai_videos.md)
:octicons-gear-24: **Default**
: `videos/`
:octicons-alert-24: **Requirement**
: Must be non-empty, use only S3-safe characters (alphanumerics plus `! _ . * ' ( ) -` per path segment), and end with a trailing `/` — an empty value would widen the ownership check that scopes listing/retrieval to the whole bucket
:octicons-check-circle-24: **Best Practice**
: Generated videos persist until deleted through the API — configure an S3 lifecycle rule on this prefix to cap storage costs
```bash
export AWS_S3_VIDEOS_PREFIX=videos/
```
Amazon Bedrock writes each video generation job's output (MP4 and manifest) under this prefix, in a folder named after the job. Because Amazon Bedrock requires the output bucket to be in the same region as the invocation, videos are stored in the [`AWS_S3_REGIONAL_BUCKETS`](#aws-s3-regional-buckets) bucket of the region that served the job.
#### `AWS_S3_VIDEOS_EXPIRES_AFTER` { #aws-s3-videos-expires-after }
:octicons-package-24: **Purpose**
: Retention period in seconds (minimum `3600`) for videos generated through the [Videos API](api_openai_videos.md)
:octicons-gear-24: **Default**
: Unset — videos never expire and persist until deleted through the API
:octicons-check-circle-24: **Best Practice**
: Pair with an S3 Lifecycle expiration rule on [`AWS_S3_VIDEOS_PREFIX`](#aws-s3-videos-prefix) covering the same duration (rounded up to whole days) so the objects are actually deleted
```bash
# Expire generated videos after 24 hours
export AWS_S3_VIDEOS_EXPIRES_AFTER=86400
```
When set, the `Video` object reports `expires_at` (job completion time plus this value) and downloading expired video content returns a 404. The server enforces expiry at the API level only; the paired S3 Lifecycle rule performs the physical cleanup.
#### `AWS_S3_BATCHES_PREFIX` { #aws-s3-batches-prefix }
:octicons-package-24: **Purpose**
: S3 prefix (folder path) for the data of the [Batch API](api_openai_batches.md) and the [Message Batches API](api_anthropic_batches.md) — the submitted requests, the results, and the batch records themselves
:octicons-gear-24: **Default**
: `batches/`
:octicons-alert-24: **Requirement**
: Must be non-empty, use only S3-safe characters (alphanumerics plus `! _ . * ' ( ) -` per path segment), and end with a trailing `/` — batches are addressed by prefix, so an empty value would make every stray object at the bucket root a candidate
:octicons-check-circle-24: **Best Practice**
: Batch data persists until the batch is deleted — configure an S3 lifecycle rule on this prefix to cap storage costs, and grant [`AWS_BEDROCK_BATCH_ROLE_ARN`](#aws-bedrock-batch-role-arn) read and write access under it
```bash
export AWS_S3_BATCHES_PREFIX=batches/
```
Each batch stores its data under a folder of its own below this prefix. Because the output bucket must be in the same region as the model that serves the batch, that data is stored in the [`AWS_S3_REGIONAL_BUCKETS`](#aws-s3-regional-buckets) bucket of the region that served it.
#### Vector Stores { #vector-stores-optional }
The [Vector Stores API](api_openai_vector_stores.md) needs an [Amazon S3 vector bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors.html) — a resource type of its own, created separately from a general purpose bucket — plus the general purpose bucket in [`AWS_S3_BUCKET`](#aws-s3-bucket), which holds the stores' records. The vector store endpoints answer `503` until both are configured, or until [`AWS_BEDROCK_KNOWLEDGE_BASE_IDS`](#aws-bedrock-knowledge-base-ids) names a knowledge base to serve instead.
#### `AWS_S3_VECTORS_BUCKET` { #aws-s3-vectors-bucket }
:octicons-package-24: **Purpose**
: Name of the Amazon S3 vector bucket that holds the indexed content of every vector store
:octicons-gear-24: **Default**
: None — the [Vector Stores API](api_openai_vector_stores.md) is disabled
:octicons-alert-24: **Requirement**
: Requires [`AWS_S3_BUCKET`](#aws-s3-bucket), which holds the vector store records; startup fails if only the vector bucket is set. The gateway's role needs the [Vector Stores permissions](operations_iam_permissions.md#vector-stores-optional) on it
```bash
export AWS_S3_VECTORS_BUCKET=my-llm-vectors-us-east-1
```
Create the bucket yourself, then let the gateway create and delete the indexes inside it — one per vector store, removed when the store is deleted or expires.
#### `AWS_S3_VECTORS_REGION` { #aws-s3-vectors-region }
:octicons-package-24: **Purpose**
: AWS region holding [`AWS_S3_VECTORS_BUCKET`](#aws-s3-vectors-bucket)
:octicons-gear-24: **Default**
: The first [`AWS_BEDROCK_REGIONS`](#aws-bedrock-regions) entry
:octicons-alert-24: **Requirement**
: Must be the bucket's own region. A vector bucket is a regional resource whose content is only reachable there, so this setting has no failover: if the region is unreachable, so is the [Vector Stores API](api_openai_vector_stores.md)
```bash
export AWS_S3_VECTORS_REGION=us-east-1
```
#### `AWS_S3_VECTOR_STORES_PREFIX` { #aws-s3-vector-stores-prefix }
:octicons-package-24: **Purpose**
: S3 prefix (folder path) in [`AWS_S3_BUCKET`](#aws-s3-bucket) for the [Vector Stores API](api_openai_vector_stores.md) records — the stores, their attached files and their file batches
:octicons-gear-24: **Default**
: `vector_stores/`
:octicons-check-circle-24: **Best Practice**
: Keep it distinct from the other prefixes so a lifecycle rule written for one never reaches the records of another; these records are the stores themselves, so no expiration rule belongs on this prefix
```bash
export AWS_S3_VECTOR_STORES_PREFIX=vector_stores/
```
#### `VECTOR_STORE_EMBEDDING_MODEL` { #vector-store-embedding-model }
:octicons-package-24: **Purpose**
: Model that turns the indexed files and the search queries into vectors
:octicons-gear-24: **Default**
: `amazon.titan-embed-text-v2:0`
:octicons-alert-24: **Requirement**
: Must be an embedding model available in your configured regions — see [Embeddings API](api_openai_embeddings.md)
```bash
export VECTOR_STORE_EMBEDDING_MODEL=amazon.titan-embed-text-v2:0
```
Each vector store records the model it was created with and keeps using it, so changing this setting only affects stores created afterwards. Existing stores keep answering exactly as before.
#### `VECTOR_STORE_CHUNK_SIZE_TOKENS` { #vector-store-chunk-size-tokens }
:octicons-package-24: **Purpose**
: Default chunk size, in tokens, for files indexed without an explicit `chunking_strategy`
:octicons-gear-24: **Default**
: `800` — the same default the upstream API applies
:octicons-alert-24: **Requirement**
: Between `100` and `4096`
```bash
export VECTOR_STORE_CHUNK_SIZE_TOKENS=800
```
A request's own `chunking_strategy` always wins, and a store created with one applies it to every file later attached without one. Chunk sizes are approximate — see [Chunking](api_openai_vector_stores.md#chunking).
#### `VECTOR_STORE_CHUNK_OVERLAP_TOKENS` { #vector-store-chunk-overlap-tokens }
:octicons-package-24: **Purpose**
: Default number of tokens shared between consecutive chunks, for files indexed without an explicit `chunking_strategy`
:octicons-gear-24: **Default**
: `400` — the same default the upstream API applies
:octicons-alert-24: **Requirement**
: Must not exceed half of [`VECTOR_STORE_CHUNK_SIZE_TOKENS`](#vector-store-chunk-size-tokens); startup fails otherwise
```bash
export VECTOR_STORE_CHUNK_OVERLAP_TOKENS=400
```
More overlap keeps a sentence split across two chunks findable from either one, at the cost of more chunks to embed and store.
#### `AWS_SQS_VECTOR_STORE_QUEUE_URL` { #aws-sqs-vector-store-queue-url }
:octicons-package-24: **Purpose**
: URL of the [Amazon SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) standard queue that carries vector store indexing work, so a file keeps being indexed when the server that accepted it stops
:octicons-gear-24: **Default**
: None — indexing runs in the server that accepted the request, and a file being indexed when that server stops is reported as `failed`
:octicons-alert-24: **Requirement**
: Requires [`AWS_S3_VECTORS_BUCKET`](#aws-s3-vectors-bucket). Must be a standard queue (not FIFO), with a [dead-letter queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html), and the gateway's role needs the [Durable vector store indexing permissions](operations_iam_permissions.md#durable-vector-store-indexing) on it
```bash
export AWS_SQS_VECTOR_STORE_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/stdapi-ai-indexing
```
Attaching a file records the work on the queue before answering, and every server reads from that queue: whichever one is still running finishes it. The region comes from the URL, so there is nothing else to configure. Create the queue yourself, in the same account.
The queue only ever carries identifiers — which store, which files, which batch — never file content, so no indexed data is stored a second time.
See [Durable indexing](api_openai_vector_stores.md#durable-indexing) for what a client observes, and [Resilience](operations_resilience.md#vector-store-indexing) for how it behaves during a deployment.
!!! warning "Give the queue a dead-letter queue"
A file the gateway cannot index is retried a few times and then reported as `failed`. Without a dead-letter queue its message is dropped at that point; with one it is kept, so you can see what was refused. The gateway reads the queue's own redrive policy at startup and reports a queue that has none as a startup warning.
#### `AWS_BEDROCK_KNOWLEDGE_BASE_IDS` { #aws-bedrock-knowledge-base-ids }
:octicons-package-24: **Purpose**
: Comma-separated allowlist of [Amazon Bedrock knowledge bases](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html) served through the [Vector Stores API](api_openai_vector_stores.md). Each allowlisted knowledge base is addressed as the vector store `vs_kb_` on every `/v1/vector_stores` endpoint, and is returned by `GET /v1/vector_stores` next to the stores the server owns
:octicons-gear-24: **Default**
: Empty — no knowledge base is addressable, and a `vs_kb_...` identifier is answered exactly as an unknown vector store is, so the allowlist cannot be probed
:octicons-alert-24: **Requirement**
: Each knowledge base must already exist, in the first [`AWS_BEDROCK_REGIONS`](#aws-bedrock-regions) entry, be a [Bedrock managed](https://docs.aws.amazon.com/bedrock/latest/userguide/kb-build-managed.html) (`MANAGED`) or [customer-managed](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build.html) (`VECTOR`) knowledge base, and the gateway's role needs the [Knowledge Base Vector Stores permissions](operations_iam_permissions.md#knowledge-base-vector-stores) on it
```bash
export AWS_BEDROCK_KNOWLEDGE_BASE_IDS=ABCDE12345,FGHIJ67890/KLMNO13579
```
Write each entry as ``, or as `/` when the knowledge base has more than one data source; with a single data source the server resolves it itself.
Both kinds of document knowledge base are served, and a store behaves the same on either. A knowledge base [connected to a structured data store](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build-structured.html) (`SQL`) or backed by an Amazon Kendra GenAI index (`KENDRA`) is not a vector store and must not be allowlisted.
The knowledge base itself always stays yours: the server never creates one and never deletes one. It searches it, and manages the documents of its data source. Name, description, creation time and status are read from the knowledge base, and the requests that would change them are refused — see [Knowledge Base Stores](api_openai_vector_stores.md#knowledge-base-stores).
This setting is independent of [`AWS_S3_VECTORS_BUCKET`](#aws-s3-vectors-bucket): a deployment that sets only this one serves its allowlisted knowledge bases and creates no store of its own.
!!! info "What a knowledge base costs"
A knowledge base search costs more than a search on a store the server owns, and a knowledge base backed by an always-on vector database bills whether it is queried or not. Both backends are offered so the choice is yours — see [Cost Management](operations_cost_management.md).
#### `AWS_TRANSCRIBE_S3_BUCKET` { #aws-transcribe-s3-bucket }
:octicons-package-24: **Purpose**
: Temporary S3 bucket for transcription workflows
:octicons-gear-24: **Default**
: Falls back to `AWS_S3_BUCKET` if not specified
:octicons-alert-24: **Requirement**
: Must be in the same region as `AWS_TRANSCRIBE_REGION` when that is set; with the default multi-region behavior it serves the primary Bedrock region, and [`AWS_S3_REGIONAL_BUCKETS`](#aws-s3-regional-buckets) entries serve the other candidate regions
```bash
# If AWS_TRANSCRIBE_REGION is us-east-1
export AWS_TRANSCRIBE_S3_BUCKET=my-transcribe-temp-us-east-1
# If AWS_TRANSCRIBE_REGION is eu-west-1
export AWS_TRANSCRIBE_S3_BUCKET=my-transcribe-temp-eu-west-1
```
#### `AWS_TRANSCRIBE_STREAM_LANGUAGES` { #aws-transcribe-stream-languages }
:octicons-package-24: **Purpose**
: Languages a streamed transcription ([`stream=true`](api_openai_audio_transcriptions.md#streaming)) picks between when the request names none
:octicons-gear-24: **Default**
: Empty — a request naming no language is transcribed once the whole recording has been read, and its language detected
:octicons-code-24: **Format**
: JSON array of two or more language codes; a single entry has no effect
A streamed transcription returns text before the recording has been fully read, which requires knowing which languages to expect. A request that names its `language` — or two or more expected `languages` — always gets one. Set this to extend the same behavior to requests that name neither, listing the languages your callers actually send. Listing more than five is not recommended, and two variants of the same language (`en-US` and `en-GB`) cannot both appear.
```bash
export AWS_TRANSCRIBE_STREAM_LANGUAGES='["en-US", "es-US", "fr-FR"]'
```
#### `AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN` { #aws-transcribe-output-encryption-key-arn }
:octicons-package-24: **Purpose**
: AWS KMS key encrypting the transcription output written to [`AWS_TRANSCRIBE_S3_BUCKET`](#aws-transcribe-s3-bucket)
:octicons-gear-24: **Default**
: None — output objects keep the bucket's own default encryption (SSE-S3)
:octicons-code-24: **Format**
: KMS key ARN: `arn::kms:::key/`; startup fails on any other value
:octicons-alert-24: **Requirement**
: The server's role needs `kms:GenerateDataKey` and `kms:Decrypt` on the key ([IAM Permissions](operations_iam_permissions.md#speech-to-text-optional)). With the default multi-region behavior the key must be usable from every candidate region — a [multi-Region key](https://docs.aws.amazon.com/kms/latest/developerguide/multi-region-keys-overview.html) or a single [`AWS_TRANSCRIBE_REGION`](#aws-transcribe-region)
```bash
export AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN=arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012
```
Each job also sends its request identifiers (`stdapi-ai.request_id`, `stdapi-ai.server_id`, and `stdapi-ai.user_id` when the user is known) as the [KMS encryption context](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context), so a key policy can be conditioned on them.
#### `AWS_S3_REGIONAL_BUCKETS` { #aws-s3-regional-buckets }
:octicons-package-24: **Purpose**
: Region-specific S3 buckets for Bedrock async and batch inference operations, and for staging attachments too large to travel inside a request
:octicons-gear-24: **Default**
: Empty (no regional buckets configured)
:octicons-code-24: **Format**
: JSON object with region names as keys and bucket names as values
:octicons-alert-24: **Requirement**
: Some Bedrock models require S3 buckets in the same region for async and batch inference operations
```bash
export AWS_S3_REGIONAL_BUCKETS='{"us-east-1": "my-bedrock-temp-us-east-1", "eu-west-1": "my-bedrock-temp-eu-west-1"}'
```
!!! info "When to Use"
Configure this setting when:
- Using Bedrock async inference API
- Using Bedrock batch inference API
- Working with models that require regional S3 storage
- Accepting chat, messages or responses requests with [large attachments](features.md#attachment-size), or embedding requests with large inputs — those are staged in the bucket of the region serving the request, which then serves that request alone without failing over
If not specified for a region where async/batch operations are attempted, those operations may fail. Requests carrying an attachment larger than the model reads inline are refused with `413` when no region able to serve the model has a bucket.
!!! success "Automatic Fallback"
For the first region in `AWS_BEDROCK_REGIONS` (your primary region), if no regional bucket is specified, the service automatically falls back to `AWS_S3_BUCKET`. You only need to configure regional buckets for additional regions beyond your primary one.
!!! info "Terraform Module"
When using the Terraform module, regional S3 buckets are created automatically for each region in `aws_bedrock_regions`. The bucket names are exposed via the `aws_s3_regional_buckets` output and passed to the container as `AWS_S3_REGIONAL_BUCKETS`. No manual configuration required.
!!! tip "Best Practice"
Apply the same [S3 Bucket Lifecycle Configuration](#s3-lifecycle) to these regional buckets as you would for the primary bucket to automatically clean up temporary files.
#### `AWS_S3_ACCEPTED_BUCKETS` { #aws-s3-accepted-buckets }
:octicons-package-24: **Purpose**
: Declare external S3 buckets that the application has read access to, mapped to their AWS region
:octicons-database-24: **Type**
: JSON object (keys: bucket names, values: AWS region identifiers)
:octicons-gear-24: **Default**
: `{}` (empty — only the application's own buckets are recognized)
:octicons-workflow-24: **Behavior**
: Declaring a bucket here enables input access to objects the application does not own:
- **S3 URI and S3 HTTP URL access** — `s3://` URIs and S3 HTTP URLs (including presigned URLs) pointing at these buckets are accepted as input sources; an HTTP URL is automatically converted to an `s3://` URI so Bedrock can access the object directly.
- **Declared region** — The region mapped to each bucket is used to reach that bucket in its own region when reading the input object. It does not influence model or inference region selection.
Without this setting, only the application's own buckets (`AWS_S3_BUCKET` and `AWS_S3_REGIONAL_BUCKETS`) are recognized.
```bash
export AWS_S3_ACCEPTED_BUCKETS='{"my-data-bucket": "us-east-1", "my-eu-bucket": "eu-west-1"}'
```
!!! warning "Required IAM Permissions"
The application's IAM role must have `s3:GetObject` permission on each declared bucket. Granting access at the bucket level is recommended:
```json
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::my-data-bucket/*",
"arn:aws:s3:::my-eu-bucket/*"
]
}
```
!!! tip "When to Use"
Configure this when your users provide S3 URLs from buckets outside the application's own buckets. This enables automatic HTTP-to-S3 URI conversion and optimal region routing for those objects.
#### S3 Bucket Lifecycle Configuration { #s3-lifecycle }
:octicons-package-24: **Purpose**
: Configure automatic deletion of temporary files and abandoned multipart upload parts to minimize storage costs
:octicons-clock-24: **Recommendation**
: Configure S3 lifecycle policies to automatically delete objects under the `AWS_S3_TMP_PREFIX` after 1 day, and abort incomplete multipart uploads under the `AWS_S3_FILES_PREFIX` after 1 day
stdapi.ai stores temporary files under the prefix configured by `AWS_S3_TMP_PREFIX` (default: `tmp/`). These include generated images, audio files, and transcription workflow files. Configure S3 lifecycle policies to automatically delete objects under this prefix after 1 day.
Additionally, multipart file uploads (OpenAI Uploads API) store parts under `AWS_S3_FILES_PREFIX` (default: `files/`). If a session is never completed or cancelled — for example when a client disconnects — the uploaded parts remain in S3 and accumulate costs. Add an `AbortIncompleteMultipartUpload` rule on the files prefix to clean these up automatically.
!!! info "Application Cleanup Behavior"
**Short-lived temporary files:** The application attempts to clean up short-lived temporary files (such as intermediate transcription files) after processing completes.
**Results shared with clients:** Files shared with clients using presigned URLs (such as generated images and audio) are never cleaned up automatically by the application. These files remain in S3 until removed by lifecycle policies or manual deletion.
**Why lifecycle policies are essential:** Since the application cannot determine when a client has finished using a presigned URL, S3 lifecycle policies are the recommended mechanism to clean up these files and prevent unbounded storage growth.
```json
{
"Rules": [
{
"Id": "DeleteTemporaryFiles",
"Status": "Enabled",
"Filter": {
"Prefix": "tmp/"
},
"Expiration": {
"Days": 1
},
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 1
}
},
{
"Id": "AbortIncompleteMultipartUploads",
"Status": "Enabled",
"Filter": {
"Prefix": "files/"
},
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 1
}
}
]
}
```
!!! warning "Important: Update the Prefixes"
The `"Prefix"` values in the lifecycle policy must match your `AWS_S3_TMP_PREFIX` and `AWS_S3_FILES_PREFIX` settings. If you use custom prefixes, update the policy accordingly.
**Examples:**
- If `AWS_S3_TMP_PREFIX=temporary/`, use `"Prefix": "temporary/"` in the first rule
- If `AWS_S3_FILES_PREFIX=prod/files/`, use `"Prefix": "prod/files/"` in the second rule
**Apply via AWS CLI:**
```bash
# For primary S3 bucket (AWS_S3_BUCKET)
aws s3api put-bucket-lifecycle-configuration \
--bucket my-stdapi-bucket \
--lifecycle-configuration file://lifecycle-policy.json
# For transcribe S3 bucket (AWS_TRANSCRIBE_S3_BUCKET, if different from AWS_S3_BUCKET)
aws s3api put-bucket-lifecycle-configuration \
--bucket my-transcribe-temp-bucket \
--lifecycle-configuration file://lifecycle-policy.json
# For regional buckets (AWS_S3_REGIONAL_BUCKETS)
aws s3api put-bucket-lifecycle-configuration \
--bucket my-stdapi-us-west-2-bucket \
--lifecycle-configuration file://lifecycle-policy.json
```
!!! tip "Apply to All S3 Buckets"
Apply this lifecycle policy to:
- **`AWS_S3_BUCKET`** - Primary bucket for generated files
- **`AWS_TRANSCRIBE_S3_BUCKET`** - Transcription temporary files (if different from AWS_S3_BUCKET)
- **`AWS_S3_REGIONAL_BUCKETS`** - All regional buckets for async/batch operations
All these buckets use the same `AWS_S3_TMP_PREFIX` for temporary file storage, and the same `AWS_S3_FILES_PREFIX` for multipart upload parts.
### Bedrock Configuration
#### `AWS_BEDROCK_REGIONS` { #aws-bedrock-regions }
:octicons-package-24: **Purpose**
: List of AWS regions where Bedrock models are available
:octicons-list-ordered-24: **Format**
: Comma-separated string
:octicons-gear-24: **Default**
: Current AWS SDK region if not specified
:octicons-workflow-24: **Behavior**
: Models are discovered in the same order as the listed regions. The first region is the primary region where your server should be hosted on AWS for optimal performance. Your S3 bucket (`AWS_S3_BUCKET`) must also be in this region. If a model is unavailable in the primary region, subsequent regions are checked in order
```bash
export AWS_BEDROCK_REGIONS=us-east-1,us-west-2,eu-west-1
```
!!! info "Region Selection Guide"
| Region | Description |
|--------|-------------|
| `us-east-1` | :material-star: Widest model selection, usually gets latest releases first |
| `us-west-2` | :material-rocket-launch: Good selection, often early access to new models |
| `eu-west-1` | :material-shield-check: European compliance, subset of US models available |
!!! tip "Advanced Configuration"
See [Compliance and Latency Optimization](#compliance-and-latency-optimization) for detailed configuration examples including GDPR compliance, regional optimization strategies, and best practices for multi-region deployments.
!!! warning "Startup Warning"
If any models in the configured regions fail availability checks (not enabled, unauthorized, or missing entitlement/agreement in your AWS account), a warning listing the affected models and per-region issues is logged at startup. Enable the required models in the [Amazon Bedrock console](https://console.aws.amazon.com/bedrock/home#/modelaccess) for each configured region.
!!! info "Unreachable Region Tolerance"
A configured region that cannot be reached (invalid region for the account, network issue, throttling) does not block startup: it is skipped with an `unreachable_bedrock_regions` warning and its models are served from the remaining regions. The skipped region is retried automatically on the next model list refresh (see [`MODEL_CACHE_SECONDS`](#model-cache-seconds)), so a recovered region rejoins without a restart. Startup only fails when **every** configured region fails, or when **every** per-model availability check errors (e.g. the `bedrock:GetFoundationModelAvailability` permission is denied) — which indicates broken credentials or configuration rather than a regional outage.
#### `AWS_BEDROCK_CROSS_REGION_INFERENCE` { #cross-region-inference }
:octicons-package-24: **Purpose**
: Enable automatic cross-region routing when a model isn't available in the primary region
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `true`
```bash
export AWS_BEDROCK_CROSS_REGION_INFERENCE=true
```
#### `AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL` { #cross-region-global }
:octicons-package-24: **Purpose**
: Allow global cross-region inference routing to any region worldwide
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `true`
!!! example "GDPR Compliance"
Set to `false` to comply with data residency regulations (e.g., EU GDPR) by restricting to regional inference only
```bash
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false
```
#### `AWS_BEDROCK_REGION_ROUTING` { #bedrock-region-routing }
:octicons-package-24: **Purpose**
: Automatic region routing strategy for distributing Bedrock requests across configured regions
:octicons-database-24: **Type**
: String
:octicons-gear-24: **Default**
: `ordered`
:octicons-workflow-24: **Behavior**
: When multiple regions are configured in `AWS_BEDROCK_REGIONS`, this setting controls how requests are distributed across them. The router automatically handles quota/throttling errors and regional unavailability by temporarily avoiding affected regions
:octicons-alert-24: **Requirement**
: Requires at least 2 regions in `AWS_BEDROCK_REGIONS` to take effect
**Available strategies:**
| Strategy | Description |
|----------|-------------|
| `disabled` | No routing; uses the single region where the model was discovered |
| `ordered` | Try regions in configured order, skipping temporarily blocked ones (default). Best for prompt caching compatibility |
| `lowest_latency` | Prefer the region with lowest measured latency. Latencies are measured at startup |
| `round_robin` | Distribute requests evenly across regions. Incompatible with prompt caching |
```bash
# Use ordered routing (default)
export AWS_BEDROCK_REGION_ROUTING=ordered
# Use lowest latency routing
export AWS_BEDROCK_REGION_ROUTING=lowest_latency
# Disable routing
export AWS_BEDROCK_REGION_ROUTING=disabled
```
!!! tip "Strategy Selection"
- **`ordered`** (default): Best general-purpose choice. Compatible with prompt caching since requests consistently go to the same region. Provides failover when a region hits quota limits
- **`lowest_latency`**: Best when response time is critical. Measures region latencies at startup and prefers the fastest region. Falls back to others when the preferred region is blocked
- **`round_robin`**: Best for maximizing aggregate throughput across regions. Not recommended with prompt caching as it distributes requests across all regions equally
!!! info "More Details"
For comprehensive documentation on region routing including failover behavior, S3 bucket pinning, logging, and best practices, see the [Region Routing Guide](operations_resilience.md).
#### `AWS_BEDROCK_REGION_ROUTING_QUOTA_BACKOFF_SECONDS` { #bedrock-region-routing-quota-backoff }
:octicons-package-24: **Purpose**
: Duration to temporarily avoid a region after receiving a quota or throttling error
:octicons-database-24: **Type**
: Integer (seconds, must be > 0)
:octicons-gear-24: **Default**
: `60`
:octicons-workflow-24: **Behavior**
: This is the **base** backoff value. When a Bedrock API call fails due to quota limits (`ThrottlingException`, `TooManyRequestsException`, `ServiceQuotaExceededException`), the affected region is temporarily blocked. The actual delay doubles with each consecutive quota error on the same region (exponential backoff), up to a hard ceiling of 1 hour. The counter resets after a successful request. Subsequent requests are routed to other available regions during the backoff period.
```bash
# Default: 60 seconds (base value — actual delay doubles per consecutive error)
export AWS_BEDROCK_REGION_ROUTING_QUOTA_BACKOFF_SECONDS=60
# Shorter base backoff for aggressive retry
export AWS_BEDROCK_REGION_ROUTING_QUOTA_BACKOFF_SECONDS=30
# Longer base backoff for conservative approach
export AWS_BEDROCK_REGION_ROUTING_QUOTA_BACKOFF_SECONDS=120
```
!!! tip "Tuning"
The base value controls how long the first quota error blocks a region. Subsequent consecutive errors on the same region double the delay (60 s → 120 s → 240 s → …, capped at 1 hour). Lower base values retry the region sooner but risk repeated throttling. Higher values provide more conservative avoidance at the cost of reduced region utilization.
See [Region Routing — Overview](operations_resilience.md#overview) for full backoff behavior details.
#### `AWS_BEDROCK_REGION_ROUTING_UNAVAILABLE_BACKOFF_SECONDS` { #bedrock-region-routing-unavailable-backoff }
:octicons-package-24: **Purpose**
: Duration to temporarily avoid a region after receiving an unavailability error
:octicons-database-24: **Type**
: Integer (seconds, must be > 0)
:octicons-gear-24: **Default**
: `30`
:octicons-workflow-24: **Behavior**
: When a Bedrock API call fails due to service unavailability (`ServiceUnavailableException`, `ModelNotReadyException`), the affected region is temporarily blocked for this many seconds. These errors are typically shorter-lived than quota limits, so the default is shorter
```bash
# Default: 30 seconds
export AWS_BEDROCK_REGION_ROUTING_UNAVAILABLE_BACKOFF_SECONDS=30
# Longer backoff for stability
export AWS_BEDROCK_REGION_ROUTING_UNAVAILABLE_BACKOFF_SECONDS=60
```
!!! info "More Details"
See [Region Routing — Overview](operations_resilience.md#overview) for full backoff behavior details.
#### `AWS_BEDROCK_REGION_ROUTING_MAX_QUOTA_BACKOFF_SECONDS` { #bedrock-region-routing-max-quota-backoff }
:octicons-package-24: **Purpose**
: Hard ceiling in seconds on the exponential quota backoff for a single region
:octicons-database-24: **Type**
: Integer (seconds, must be > 0)
:octicons-gear-24: **Default**
: `3600` (1 hour)
:octicons-workflow-24: **Behavior**
: Quota backoff grows exponentially with consecutive errors (base interval × 2^n). This setting caps how large that value can become, preventing a region from being blocked indefinitely. Reduce it to allow faster recovery; increase it to keep a misbehaving region sidelined for longer.
```bash
# Default: 1 hour ceiling
export AWS_BEDROCK_REGION_ROUTING_MAX_QUOTA_BACKOFF_SECONDS=3600
# More aggressive recovery
export AWS_BEDROCK_REGION_ROUTING_MAX_QUOTA_BACKOFF_SECONDS=600
```
#### `AWS_BEDROCK_REGION_ROUTING_QUOTA_STALE_FACTOR` { #bedrock-region-routing-quota-stale-factor }
:octicons-package-24: **Purpose**
: Multiplier applied to the max quota backoff to compute the stale-error reset threshold
:octicons-database-24: **Type**
: Integer (must be > 0)
:octicons-gear-24: **Default**
: `2` (threshold = 2 × max quota backoff = 2 hours with defaults)
:octicons-workflow-24: **Behavior**
: If the most recent quota error on a region occurred more than `max_quota_backoff × factor` seconds ago, the consecutive-error counter is reset and the next error is treated as a fresh start rather than an escalation. A higher value keeps memory of past errors for longer before resetting the counter.
```bash
# Default: reset counter after 2× the max backoff window
export AWS_BEDROCK_REGION_ROUTING_QUOTA_STALE_FACTOR=2
# Longer memory of past errors
export AWS_BEDROCK_REGION_ROUTING_QUOTA_STALE_FACTOR=4
```
#### `AWS_BEDROCK_MAX_RETRIES` { #bedrock-max-retries }
:octicons-package-24: **Purpose**
: Maximum number of retries per Bedrock invocation, each retry escalating to the next available region
:octicons-database-24: **Type**
: Integer (must be 0 or greater; `0` disables retries)
:octicons-gear-24: **Default**
: `9`
:octicons-workflow-24: **Behavior**
: Controls the retry budget for each Bedrock API call. When region routing is enabled, every retry escalates to the next region in priority order and each candidate region is tried at most once, so the attempts are bounded by the smaller of `AWS_BEDROCK_MAX_RETRIES` + 1 and the number of candidate regions for the model — with 3 regions and the default 9 retries, a request makes at most 3 attempts. A region that just failed is still blocked by its own backoff, and retrying it would only extend that backoff instead of recovering the request. When routing is disabled, or the region is pinned by S3 inputs, the full budget is spent as SDK retries against that single region.
```bash
# Default: 9 retries (10 total attempts)
export AWS_BEDROCK_MAX_RETRIES=9
# Fail faster (e.g. low-latency interactive use cases)
export AWS_BEDROCK_MAX_RETRIES=3
# Deeper in-region retrying for single-region or S3-pinned requests
export AWS_BEDROCK_MAX_RETRIES=18
```
!!! tip "Related setting"
See [`AWS_BEDROCK_REGION_ROUTING`](#bedrock-region-routing) and [Region Routing](operations_resilience.md) for the full retry and failover behavior.
#### `AWS_FAILOVER_MAX_RETRIES` { #failover-max-retries }
:octicons-package-24: **Purpose**
: Maximum SDK retry attempts per candidate region for the multi-region failover services (Polly, Transcribe, Translate, Comprehend)
:octicons-database-24: **Type**
: Integer (must be 0 or greater)
:octicons-gear-24: **Default**
: `2`
:octicons-workflow-24: **Behavior**
: Only applied when a service has several candidate regions (no explicit region setting): each region attempt uses this reduced retry budget (`2` retries = 3 attempts per region) before failing over, so failover across regions replaces deep in-region retrying. When a service is pinned to a single region, the standard retry budget from [`AWS_BEDROCK_MAX_RETRIES`](#bedrock-max-retries) applies instead.
```bash
# Default: 2 retries (3 attempts) per candidate region
export AWS_FAILOVER_MAX_RETRIES=2
# Fail over after a single attempt per region
export AWS_FAILOVER_MAX_RETRIES=0
```
!!! tip "Related setting"
See [Other AWS Services Failover](operations_resilience.md#other-aws-services-failover) for the full multi-region failover behavior.
#### `AWS_BEDROCK_MANTLE_ENABLED` { #bedrock-mantle-enabled }
:octicons-package-24: **Purpose**
: Expose models served by the Amazon Bedrock Mantle endpoint (OpenAI/Anthropic-compatible APIs) in addition to the classic Bedrock Converse models
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `true`
:octicons-workflow-24: **Behavior**
: Mantle-only models (e.g. OpenAI GPT, xAI Grok, Google Gemma 4) become available on the chat completions, responses, messages, and completions routes. Models available on both the classic bedrock-runtime endpoint and Mantle are served by bedrock-runtime unless listed in [`AWS_BEDROCK_MANTLE_PREFERRED_MODELS`](#bedrock-mantle-preferred-models).
Authentication requires no static secrets: short-term bearer tokens are derived automatically (SigV4-presigned) from the same AWS credential chain the server already uses, and refreshed transparently.
When Bedrock Mantle is unreachable or the IAM role lacks `bedrock-mantle` permissions, Mantle models are simply not listed and a warning is logged at startup — no configuration change required.
```bash
export AWS_BEDROCK_MANTLE_ENABLED=false
```
!!! warning "Guardrails Not Supported"
Amazon Bedrock Guardrails are not supported on Mantle-served requests. When guardrails are configured while Mantle models are exposed, a startup warning reports how many models are affected; set `AWS_BEDROCK_MANTLE_ENABLED=false` to disable them.
!!! note "Cross-Region Inference Profiles Not Available"
Bedrock cross-region inference profiles do not exist on the Mantle endpoint. Mantle relies on [multi-region failover](#bedrock-mantle-regions) and its own separate throughput quotas instead.
!!! warning "Required IAM Permissions"
Enabling this setting requires the `bedrock-mantle` IAM permissions — see [Bedrock Mantle IAM Permissions](operations_iam_permissions.md#bedrock-mantle-iam).
[:octicons-arrow-right-24: Bedrock Mantle Models feature overview](features.md#bedrock-mantle-models)
#### `AWS_BEDROCK_MANTLE_REGIONS` { #bedrock-mantle-regions }
:octicons-package-24: **Purpose**
: List of AWS regions used for Amazon Bedrock Mantle, in failover priority order
:octicons-database-24: **Type**
: Comma-separated string of AWS region identifiers
:octicons-gear-24: **Default**
: The regions of [`AWS_BEDROCK_REGIONS`](#aws-bedrock-regions) that offer Bedrock Mantle
:octicons-workflow-24: **Behavior**
: Model availability differs per region; the served model catalog is the union of all listed regions. Region failover, quota backoff, and health tracking work exactly like classic Bedrock [region routing](operations_resilience.md).
```bash
export AWS_BEDROCK_MANTLE_REGIONS=us-east-1,eu-west-1
```
!!! note "Regions Without a Mantle Endpoint"
Bedrock Mantle is offered in fewer regions than classic Bedrock — see [model availability by endpoint](https://docs.aws.amazon.com/bedrock/latest/userguide/models-endpoint-availability.html). Left unset, this setting keeps only the regions of `AWS_BEDROCK_REGIONS` known to offer it, so a deployment spanning other regions is not held up at startup by an address that does not exist.
An explicit value is used exactly as given, which is how a region AWS adds later is used without waiting for a release. A region that turns out to have no Mantle endpoint is named in a startup warning rather than retried forever. If none of your regions offers it, set `AWS_BEDROCK_MANTLE_ENABLED=false`.
#### `AWS_BEDROCK_MANTLE_ENDPOINT_URL` { #bedrock-mantle-endpoint-url }
:octicons-package-24: **Purpose**
: Override the Amazon Bedrock Mantle endpoint URL template
:octicons-database-24: **Type**
: String — URL template with a `{region}` placeholder
:octicons-gear-24: **Default**
: None (`https://bedrock-mantle.{region}.api.aws`)
:octicons-workflow-24: **Behavior**
: The `{region}` placeholder is substituted with the target region.
```bash
export AWS_BEDROCK_MANTLE_ENDPOINT_URL='https://bedrock-mantle.{region}.api.aws'
```
#### `AWS_BEDROCK_MANTLE_PREFERRED_MODELS` { #bedrock-mantle-preferred-models }
:octicons-package-24: **Purpose**
: Model IDs (or ID prefixes) served by Amazon Bedrock Mantle even when also available on the classic bedrock-runtime endpoint
:octicons-database-24: **Type**
: Comma-separated string of model IDs or ID prefixes
:octicons-gear-24: **Default**
: `[]` (empty — dual-homed models are served by bedrock-runtime)
:octicons-workflow-24: **Behavior**
: Useful to leverage Mantle's independent throughput quotas or native response storage for selected models. Mantle quotas (per-model, per-region tokens-per-minute) are independent from bedrock-runtime quotas.
```bash
export AWS_BEDROCK_MANTLE_PREFERRED_MODELS='anthropic.claude-haiku-4-5,openai.gpt-oss'
```
#### `AWS_BEDROCK_MANTLE_SERVICE_HEADER` { #bedrock-mantle-service-header }
:octicons-package-24: **Purpose**
: Honor the `x-stdapi-service: bedrock-mantle` request header to route a model available on both endpoints through Bedrock Mantle for that request instead of the default bedrock-runtime serving
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
```bash
export AWS_BEDROCK_MANTLE_SERVICE_HEADER=true
```
!!! warning "Incompatible with Bedrock Guardrails"
Requires [`AWS_BEDROCK_MANTLE_ENABLED`](#bedrock-mantle-enabled) and cannot be enabled together with Amazon Bedrock Guardrails: guardrails do not apply to Mantle-served requests, so a per-request header would allow clients to bypass them.
#### `AWS_BEDROCK_MANTLE_PROJECT` { #bedrock-mantle-project }
:octicons-package-24: **Purpose**
: Default Amazon Bedrock Project/Workspace ID attributed to Bedrock Mantle inference requests for cost tracking and observability
:octicons-database-24: **Type**
: String — a bare project ID (e.g. `proj_abc123` or `default`), not an ARN
:octicons-gear-24: **Default**
: None (requests fall to the account's `default` project)
:octicons-workflow-24: **Behavior**
: Bedrock Projects (OpenAI-compatible APIs) and Workspaces (Anthropic Messages API) are the same underlying resource; the value is sent as the `OpenAI-Project` header on the Chat Completions and Responses APIs, and as the `anthropic-workspace` header on the Anthropic Messages API. When unset, requests fall to the account's `default` project — no failure.
```bash
export AWS_BEDROCK_MANTLE_PROJECT=proj_abc123
```
!!! note "Bedrock Mantle only"
Project/Workspace attribution is honored **only** for models served by the Amazon Bedrock Mantle endpoint. Classic `bedrock-runtime` (non-Mantle) models ignore it and use application inference profiles instead.
#### `AWS_BEDROCK_ALLOW_MANTLE_PROJECT_OVERRIDE` { #bedrock-allow-mantle-project-override }
:octicons-package-24: **Purpose**
: Allow a request to override the configured Mantle project via the `OpenAI-Project` / `anthropic-workspace` header
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Behavior**
: When `true`, a request may set its own project through the `OpenAI-Project` (Chat Completions, Responses) or `anthropic-workspace` (Anthropic Messages) header. When `false` **and** [`AWS_BEDROCK_MANTLE_PROJECT`](#bedrock-mantle-project) is configured, the request header is ignored and the server default applies. When **no** default project is configured, the request header is always honored regardless of this flag. A malformed request-supplied project ID returns `400`.
```bash
export AWS_BEDROCK_ALLOW_MANTLE_PROJECT_OVERRIDE=true
```
!!! note "Bedrock Mantle only"
These headers apply **only** to models served by the Amazon Bedrock Mantle endpoint; classic `bedrock-runtime` models ignore them.
#### `AWS_BEDROCK_EXTERNAL_WEB_ACCESS` { #bedrock-external-web-access }
:octicons-package-24: **Purpose**
: Let the built-in [web search tool](api_openai_responses.md#openai-gpt-web-search) reach the public web
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Behavior**
: Controls whether the built-in web search tool may reach the external web. Searches are answered from the Amazon Bedrock web index and cache either way, and answers are current and carry source citations. AWS [documents](https://docs.aws.amazon.com/bedrock/latest/userguide/web-search.html) that retrieval is served entirely from that index and cache today, so no request data leaves the AWS boundary even when this is enabled, and that a future release may allow live external retrieval — at which point request data may leave it. Enabling it is therefore a decision taken in advance about behaviour that can change. It also requires the `bedrock-websearch:ExternalWebAccess` IAM permission on the credentials this server uses; each action is authorized only when a model actually attempts it, and a denied call does not fail the request.
```bash
export AWS_BEDROCK_EXTERNAL_WEB_ACCESS=true
```
#### `AWS_BEDROCK_ALLOW_EXTERNAL_WEB_ACCESS_OVERRIDE` { #bedrock-allow-external-web-access-override }
:octicons-package-24: **Purpose**
: Allow a request to override [`AWS_BEDROCK_EXTERNAL_WEB_ACCESS`](#bedrock-external-web-access) with the `external_web_access` extra model parameter
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Behavior**
: When `true`, a request that sends `external_web_access` as an extra model parameter decides for that request, on the models whose web search takes a web access choice per request — the OpenAI GPT-5.x family. When `false`, a request that sets it to anything other than the configured value is rejected with `400` rather than being silently overridden; a request that omits it always gets the configured value. A request asking for a value a model cannot be given is rejected with `400` as well, rather than accepted and quietly ignored. On the models that do take it, a request naming no web search tool is accepted: nothing is searched, so the value has no search to apply to.
```bash
export AWS_BEDROCK_ALLOW_EXTERNAL_WEB_ACCESS_OVERRIDE=true
```
#### `AWS_BEDROCK_MODEL_REGION_RESTRICT` { #bedrock-model-region-restrict }
:octicons-package-24: **Purpose**
: Restrict a model to specific region(s) only, useful when a model provides important features only in certain regions
:octicons-database-24: **Type**
: JSON object (keys: Bedrock model IDs or prefixes, values: ordered lists of allowed regions)
:octicons-gear-24: **Default**
: `{}` (empty — no model-specific region restriction)
:octicons-workflow-24: **Behavior**
: When set, the model is made available **only** in the listed regions (intersected with the regions where it is actually available), and the list order defines the routing priority when the default `ordered` [routing strategy](#bedrock-region-routing) is used. No fallback to other regions occurs. Keys can be exact model IDs or prefixes that match the beginning of a model ID
```bash
# Restrict Nova Pro to us-east-1 for grounding support
export AWS_BEDROCK_MODEL_REGION_RESTRICT='{"amazon.nova-pro-v1:0": ["us-east-1"]}'
```
!!! tip "Use Case: Region-Specific Features"
Some model features are only available in specific regions. For example, Nova grounding is only available in `us-east-1`. Restricting the model to that region ensures the feature is always available.
See [Region Routing — Model Region Restrict](operations_resilience.md#model-region-restrict) for more details.
!!! warning "Startup Warning"
If a key has no matching available model, a warning is logged at startup. This can happen for two reasons:
- **Typo or unknown model** — the key (exact ID or prefix) does not match any model ID returned by Bedrock.
- **No matching region** — the model exists but is not available in any of the regions listed in `AWS_BEDROCK_REGIONS` (e.g. the model is not enabled in those regions, or the restricted regions are not configured).
#### `AWS_BEDROCK_LEGACY` { #bedrock-legacy }
:octicons-package-24: **Purpose**
: Allow usage of legacy/deprecated Bedrock models
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
```bash
export AWS_BEDROCK_LEGACY=true
```
#### `AWS_BEDROCK_DEPRECATED_MODEL_FALLBACK` { #bedrock-deprecated-model-fallback }
:octicons-package-24: **Purpose**
: Transparently reroute requests using a deprecated model ID to its recommended replacement
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `true`
:octicons-workflow-24: **Behavior**
: When `true`, any request that specifies a deprecated model ID (as listed in the server's deprecation registry) is silently retried with the recommended replacement model. The replacement is fully re-evaluated — alias resolution, modality checks, and region routing all apply to the new model ID. When `false`, deprecated model IDs return a `404` error with a message indicating the replacement, forcing clients to migrate explicitly.
```bash
# Transparent fallback (default) — clients using old model IDs keep working
export AWS_BEDROCK_DEPRECATED_MODEL_FALLBACK=true
# Strict mode — deprecated model IDs return 404, clients must update their code
export AWS_BEDROCK_DEPRECATED_MODEL_FALLBACK=false
```
#### `AWS_BEDROCK_DEPRECATED_MODELS` { #bedrock-deprecated-models }
:octicons-package-24: **Purpose**
: Extend or override the built-in deprecated model registry with custom mappings
:octicons-database-24: **Type**
: JSON object — `dict[str, str]`
:octicons-gear-24: **Default**
: `{}`
:octicons-workflow-24: **Behavior**
: Merged with the built-in registry at startup. User-provided entries take precedence over built-in ones — this means it can be used both to **add** new deprecated model mappings and to **override** the fallback target of an already-defined deprecated model. Effective only when [`AWS_BEDROCK_DEPRECATED_MODEL_FALLBACK`](#bedrock-deprecated-model-fallback) is `true`.
:octicons-link-external-24: **Reference**
: [Amazon Bedrock model lifecycle](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle.html)
```bash
# Add a custom deprecated model and override an existing built-in mapping
export AWS_BEDROCK_DEPRECATED_MODELS='{"my-old-model-v1": "my-new-model-v2", "amazon.titan-text-lite-v1": "amazon.nova-lite-v1:0"}'
```
#### `AWS_BEDROCK_MARKETPLACE_AUTO_SUBSCRIBE` { #bedrock-marketplace-auto-subscribe }
:octicons-package-24: **Purpose**
: Control automatic subscription to new models in AWS Marketplace
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `true`
:octicons-workflow-24: **Behavior**
: When `true`, the server automatically subscribes to new models discovered in the AWS Marketplace, making them immediately available through the API. When `false`, only models with existing marketplace subscriptions are visible and accessible
:octicons-lock-24: **IAM Permissions Required**
: `aws-marketplace:Subscribe`, `aws-marketplace:ViewSubscriptions` — see [Marketplace Auto-Subscribe IAM](operations_iam_permissions.md#bedrock-marketplace-auto-subscribe-iam)
```bash
# Allow automatic subscription (default)
export AWS_BEDROCK_MARKETPLACE_AUTO_SUBSCRIBE=true
# Restrict to pre-subscribed models only
export AWS_BEDROCK_MARKETPLACE_AUTO_SUBSCRIBE=false
```
!!! info "What is Marketplace Auto-Subscribe?"
Amazon Bedrock requires marketplace subscription before certain models can be used. This setting controls whether stdapi.ai automatically handles the subscription process:
- :material-check: **`true` (default)**: Models are automatically subscribed when discovered, providing seamless access to new models as they become available
- :material-close: **`false`**: Only models that have already been subscribed through the AWS Marketplace are visible, providing explicit control over model access
!!! tip "When to Disable"
Set to `false` when:
- :material-shield-check: You need explicit control over which models are accessible
- :material-cash: You want to prevent automatic marketplace subscriptions that may incur costs
- :material-security: Your organization requires manual approval for new AI model usage
- :material-account-check: Compliance policies require pre-authorization of AI models
!!! info "AWS Documentation"
For more information about Bedrock model access and marketplace registration, see the [Amazon Bedrock Model Access documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html).
#### `AWS_BEDROCK_ALLOW_CROSS_REGION_INFERENCE_PROFILE_ARN` { #bedrock-allow-cross-region-profile-arn }
:octicons-package-24: **Purpose**
: Allow users to pass cross-region inference profile ARNs directly as model IDs in API requests
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Behavior**
: When enabled, users can use cross-region inference profile ARNs instead of model IDs in the `model` parameter. Cross-region inference profiles enable routing to multiple regions for better availability
:octicons-lock-24: **IAM Permissions Required**
: `bedrock:GetInferenceProfile` (see [IAM Permissions](operations_iam_permissions.md#bedrock-inference-profiles-and-prompt-routers-optional))
```bash
# Disabled (default) - users can only use standard model IDs
# No environment variable needed
# Enable cross-region inference profile ARN support
export AWS_BEDROCK_ALLOW_CROSS_REGION_INFERENCE_PROFILE_ARN=true
```
!!! warning "Additional IAM Permissions Required"
Enabling this setting requires adding the `bedrock:GetInferenceProfile` IAM permission to your role/user. Without this permission, API requests using inference profile ARNs will fail with authorization errors.
See the [Bedrock Inference Profiles and Prompt Routers IAM section](operations_iam_permissions.md#bedrock-inference-profiles-and-prompt-routers-optional) for the complete policy configuration.
!!! example "Example ARN"
```text
arn:aws:bedrock:us-east-1:123456789012:inference-profile/us.anthropic.claude-sonnet-5
```
!!! success "Automatic Cross-Region Routing (Default Behavior)"
**By default, stdapi.ai automatically determines and uses the best cross-region inference profile for each model**, based on [`AWS_BEDROCK_REGIONS`](#aws-bedrock-regions), [`AWS_BEDROCK_CROSS_REGION_INFERENCE`](#cross-region-inference), and [`AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL`](#cross-region-global). Manually passing cross-region inference profile ARNs is only needed in rare cases to override that selection — for most deployments, leave this disabled. See [Using Inference Profile and Prompt Router ARNs](#using-inference-profile-and-prompt-router-arns) for details.
#### `AWS_BEDROCK_ALLOW_APPLICATION_INFERENCE_PROFILE_ARN` { #bedrock-allow-application-profile-arn }
:octicons-package-24: **Purpose**
: Allow users to pass application inference profile ARNs directly as model IDs in API requests
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Behavior**
: When enabled, users can use application inference profile ARNs instead of model IDs in the `model` parameter. Application inference profiles are custom routing configurations for specific use cases
:octicons-lock-24: **IAM Permissions Required**
: `bedrock:GetInferenceProfile` (see [IAM Permissions](operations_iam_permissions.md#bedrock-inference-profiles-and-prompt-routers-optional))
```bash
# Disabled (default) - users can only use standard model IDs
# No environment variable needed
# Enable application inference profile ARN support
export AWS_BEDROCK_ALLOW_APPLICATION_INFERENCE_PROFILE_ARN=true
```
!!! warning "Additional IAM Permissions Required"
Enabling this setting requires adding the `bedrock:GetInferenceProfile` IAM permission to your role/user. Without this permission, API requests using application inference profile ARNs will fail with authorization errors.
See the [Bedrock Inference Profiles and Prompt Routers IAM section](operations_iam_permissions.md#bedrock-inference-profiles-and-prompt-routers-optional) for the complete policy configuration.
!!! example "Example ARN"
```text
arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123xyz
```
!!! info "What are Application Inference Profiles?"
Application inference profiles are custom routing configurations that you create in your AWS account. They allow you to define specific routing behavior, region preferences, and failover strategies tailored to your application's needs.
!!! tip "When to Enable"
Enable this setting when:
- :material-application: You have custom application inference profiles configured in your AWS account
- :material-cog: You need application-specific routing configurations
- :material-account-multiple: You want to give users access to custom profiles you've created
#### `AWS_BEDROCK_ALLOW_PROMPT_ROUTER_ARN` { #bedrock-allow-prompt-router-arn }
:octicons-package-24: **Purpose**
: Allow users to pass prompt router ARNs directly as model IDs in API requests
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Behavior**
: When enabled, users can use prompt router ARNs instead of model IDs in the `model` parameter. Prompt routers enable dynamic model selection based on prompt characteristics
:octicons-lock-24: **IAM Permissions Required**
: `bedrock:GetPromptRouter` (see [IAM Permissions](operations_iam_permissions.md#bedrock-inference-profiles-and-prompt-routers-optional))
```bash
# Disabled (default) - users can only use standard model IDs
# No environment variable needed
# Enable prompt router ARN support
export AWS_BEDROCK_ALLOW_PROMPT_ROUTER_ARN=true
```
!!! warning "Additional IAM Permissions Required"
Enabling this setting requires adding the `bedrock:GetPromptRouter` IAM permission to your role/user. Without this permission, API requests using prompt router ARNs will fail with authorization errors.
See the [Bedrock Inference Profiles and Prompt Routers IAM section](operations_iam_permissions.md#bedrock-inference-profiles-and-prompt-routers-optional) for the complete policy configuration.
!!! example "Example ARN"
```text
arn:aws:bedrock:us-east-1:123456789012:default-prompt-router/my-router
```
!!! info "What are Prompt Routers?"
Prompt routers are intelligent routing systems that analyze prompt characteristics (length, complexity, language) and dynamically select the most appropriate model. This enables cost optimization and performance tuning based on request patterns.
!!! tip "When to Enable"
Enable this setting when:
- :material-robot: You have prompt routers configured in your AWS account
- :material-cash: You want intelligent cost optimization through dynamic model selection
- :material-speedometer: You need automatic model selection based on prompt complexity
#### `AWS_BEDROCK_ALLOW_PROMPT_ARN` { #bedrock-allow-prompt-arn }
:octicons-package-24: **Purpose**
: Allow users to reference an Amazon Bedrock Prompt Management prompt ARN in the OpenAI Responses API `prompt` parameter
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Behavior**
: When enabled, `prompt.id` accepts a prompt ARN (with an optional `prompt.version`) and `prompt.variables` fill in the template. Amazon Bedrock renders the stored prompt, and the model it is bound to serves the request. When disabled, any `prompt` parameter is rejected with a `400` error
:octicons-lock-24: **IAM Permissions Required**
: `bedrock:GetPrompt` (resolve the prompt's model) and `bedrock:RenderPrompt` (invoke it)
```bash
# Disabled (default) - the Responses API `prompt` parameter returns 400
# No environment variable needed
# Enable Prompt Management prompt ARN support
export AWS_BEDROCK_ALLOW_PROMPT_ARN=true
```
!!! warning "Additional IAM Permissions Required"
Enabling this setting requires adding the `bedrock:GetPrompt` and `bedrock:RenderPrompt` IAM permissions, scoped to the prompt resources you want to expose. Without them, requests using a prompt ARN fail with authorization errors.
!!! example "Example ARN"
```text
arn:aws:bedrock:us-east-1:123456789012:prompt/ABCDE12345:1
```
!!! info "Scope and Limitations"
- Only **TEXT** prompts are supported, and the request's `model` must be the model the prompt is bound to.
- Prompt variable values must be plain strings.
- `input`, `instructions`, `tools`, `text`, `previous_response_id` and inference parameters cannot be combined with `prompt`.
See [Managed Prompt Templates](api_openai_responses.md#managed-prompt-templates) for the full request contract.
#### `AWS_BEDROCK_MODEL_ARN_MAPPING` { #bedrock-model-arn-mapping }
:octicons-package-24: **Purpose**
: Map standard model IDs to custom inference profile or prompt router ARNs for server-controlled routing
:octicons-code-24: **Format**
: JSON object with model IDs as keys and ARNs as values
:octicons-gear-24: **Default**
: `{}` (empty, no mappings)
:octicons-workflow-24: **Behavior**
: When configured, the mapped ARN is used instead of the default cross-region inference profile when clients request the model by its standard ID. This provides centralized control over model routing without requiring client changes
```bash
export AWS_BEDROCK_MODEL_ARN_MAPPING='{
"anthropic.claude-sonnet-5": "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/my-custom-profile",
"anthropic.claude-haiku-4-5-20251001-v1:0": "arn:aws:bedrock:us-east-1:123456789012:default-prompt-router/my-router"
}'
```
!!! info "What is Model ARN Mapping?"
Model ARN mapping allows server administrators to override the default routing behavior for specific models. When a client requests a model using its standard ID (e.g., `anthropic.claude-sonnet-5`), the server automatically uses the mapped ARN for routing instead.
**Supported ARN Types:**
- :material-earth: **Cross-region inference profiles** - AWS-managed multi-region routing
- :material-application: **Application inference profiles** - Custom routing configurations
- :material-robot: **Prompt routers** - Intelligent dynamic model selection
!!! success "Key Benefits"
- :material-server: **Centralized Control** - Change routing behavior without modifying client code
- :material-account-group: **Transparent to Clients** - Clients use standard model IDs, server handles routing
- :material-swap-horizontal: **Easy Migration** - Switch between routing strategies by updating server config
- :material-cog: **Environment-Specific** - Different mappings for dev/staging/production environments
!!! example "Use Cases"
**Cost Optimization with Prompt Router:**
```bash
export AWS_BEDROCK_MODEL_ARN_MAPPING='{
"anthropic.claude-sonnet-5": "arn:aws:bedrock:us-east-1:123456789012:default-prompt-router/cost-optimizer"
}'
```
Automatically route simple prompts to cheaper models, complex prompts to premium models.
**Custom Application Profile:**
```bash
export AWS_BEDROCK_MODEL_ARN_MAPPING='{
"anthropic.claude-sonnet-5": "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/production-profile"
}'
```
Use your custom inference profile with specific region preferences and failover behavior.
**Environment-Specific Routing:**
```bash
# Production: Use cost-optimized prompt router
export AWS_BEDROCK_MODEL_ARN_MAPPING='{"anthropic.claude-sonnet-5": "arn:aws:bedrock:us-east-1:123456789012:default-prompt-router/prod-router"}'
# Development: Use standard cross-region profile
export AWS_BEDROCK_MODEL_ARN_MAPPING='{}'
```
!!! tip "Best Practices"
- :material-test-tube: Test mappings in development before deploying to production
- :material-file-document: Document your ARN mappings and their purposes
- :material-update: Keep ARN mappings in version control alongside other configuration
- :material-monitor: Monitor routing behavior after updating mappings
!!! warning "Startup Warning"
If any model IDs in `AWS_BEDROCK_MODEL_ARN_MAPPING` are not found among available Bedrock models, a warning listing the affected entries is logged at startup. This typically means the model is not enabled in your configured regions or the model ID contains a typo.
### Other AWS Services
!!! note "Optional Configuration"
Each service region is optional. Left unset, the service treats every `AWS_BEDROCK_REGIONS` entry as a candidate and fails over between them; setting one pins the service to that single region, with no failover.
#### `AWS_POLLY_REGION` { #aws-polly-region }
:octicons-package-24: **Purpose**
: Region for Amazon Polly text-to-speech service
:octicons-gear-24: **Default**
: All regions in `AWS_BEDROCK_REGIONS`, with per-engine regional discovery and automatic failover
:octicons-workflow-24: **Behavior**
: When unset, voice availability is discovered per engine in every `AWS_BEDROCK_REGIONS` entry at startup: an engine (Standard, Neural, Long-form, Generative) is exposed as a model when at least one candidate region offers it, and each synthesis call routes to the regions offering the requested engine and voice, failing over on region-level errors. Setting an explicit region pins Polly to that single region — engines it does not offer are then disabled.
```bash
export AWS_POLLY_REGION=us-east-1
```
!!! warning "Amazon Polly Engine Availability"
Not all Polly engines (Standard, Neural, Long-form, Generative) are available in all AWS regions. With the default multi-region behavior, an engine missing from one region is simply served from another candidate region that offers it. See [Amazon Polly feature and region compatibility](https://docs.aws.amazon.com/polly/latest/dg/limits.html#limits-regions) for detailed information.
#### `AWS_COMPREHEND_REGION` { #aws-comprehend-region }
:octicons-package-24: **Purpose**
: Region for the Amazon Comprehend services (language detection and toxicity moderation)
:octicons-gear-24: **Default**
: All regions in `AWS_BEDROCK_REGIONS`, tried in order with automatic failover
:octicons-workflow-24: **Behavior**
: When unset, Comprehend calls try each `AWS_BEDROCK_REGIONS` entry in order and fail over to the next region on region-level errors (throttling, service unavailability, network issues, or a region that does not offer Comprehend or the requested operation). Setting an explicit region pins Comprehend to that single region with no failover.
```bash
export AWS_COMPREHEND_REGION=us-east-1
```
!!! warning "Amazon Comprehend Regional Availability"
Amazon Comprehend is not available in all AWS regions. stdapi.ai uses the `detect_dominant_language` feature for language detection and `detect_toxic_content` for [Comprehend moderation](operations_iam_permissions.md#comprehend-moderation). Verify service and feature availability in your target region (with the default multi-region behavior, a region without Comprehend simply fails over to the next one). See [Amazon Comprehend supported regions](https://docs.aws.amazon.com/comprehend/latest/dg/guidelines-and-limits.html#limits-regions) for regional availability.
#### `AWS_TRANSCRIBE_REGION` { #aws-transcribe-region }
:octicons-package-24: **Purpose**
: Region for Amazon Transcribe speech-to-text service
:octicons-gear-24: **Default**
: All regions in `AWS_BEDROCK_REGIONS` that have a co-located S3 bucket, tried in order with automatic failover
:octicons-workflow-24: **Behavior**
: Transcription jobs need an S3 bucket in the job's region. When unset, every `AWS_BEDROCK_REGIONS` entry with a usable bucket is a candidate — the primary region is served by [`AWS_TRANSCRIBE_S3_BUCKET`](#aws-transcribe-s3-bucket) (or `AWS_S3_BUCKET`), the others by their [`AWS_S3_REGIONAL_BUCKETS`](#aws-s3-regional-buckets) entry. On a region-level error while starting a job, the audio is server-side copied to the next candidate's bucket and the job restarts there. Setting an explicit region pins Transcribe to that single region with no failover.
```bash
export AWS_TRANSCRIBE_REGION=us-east-1
```
#### `AWS_TRANSLATE_REGION` { #aws-translate-region }
:octicons-package-24: **Purpose**
: Region for Amazon Translate text translation service
:octicons-gear-24: **Default**
: All regions in `AWS_BEDROCK_REGIONS`, tried in order with automatic failover
:octicons-workflow-24: **Behavior**
: When unset, translation calls try each `AWS_BEDROCK_REGIONS` entry in order and fail over to the next region on region-level errors (throttling, service unavailability, network issues, or a region that does not offer Translate). Setting an explicit region pins Translate to that single region with no failover.
```bash
export AWS_TRANSLATE_REGION=us-east-1
```
---
### Compliance and Latency Optimization
Strategic region configuration is critical for both regulatory compliance and performance optimization. This section provides best practice configurations for common scenarios.
!!! info "AWS AI Services Data Privacy"
**Amazon Bedrock**: Does not store or use user prompts and responses, and does not share them with third parties by default. Your content remains private and is not used to train models.
**Other AI Services**: AWS collects telemetry data from other AI services (Polly, Comprehend, Transcribe, Translate) by default. For enhanced data privacy and compliance, you can opt out of AWS using your content to improve AI services. Configure [AI services opt-out policies](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_ai-opt-out.html) at the AWS Organizations level to prevent your data from being used for service improvement.
#### GDPR and Data Residency Compliance
For applications serving European users, data residency regulations like GDPR may require that data processing occurs within specific geographic boundaries.
```bash title="EU-Only Configuration (Strict GDPR)"
# Use only European regions
export AWS_S3_BUCKET=my-stdapi-eu-bucket
export AWS_BEDROCK_REGIONS=eu-west-1,eu-west-3,eu-central-1
# Disable global cross-region inference to prevent data routing outside Europe
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false
# Keep cross-region inference enabled for failover within EU regions
export AWS_BEDROCK_CROSS_REGION_INFERENCE=true
```
!!! success "Key Compliance Settings"
- **`AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false`**: Prevents requests from being routed to regions outside your specified list
- **`AWS_BEDROCK_CROSS_REGION_INFERENCE=true`**: Enables cross-region inference within your specified EU regions
- **All services in EU regions**: Ensures all data processing stays within European boundaries
!!! warning "Important Considerations"
- Not all Bedrock models are available in all EU regions - verify model availability
- Some newer models may be available in US regions first; this configuration prioritizes compliance over immediate access to latest models
- S3 buckets must be created in EU regions and configured appropriately for data residency
#### Latency Optimization
For applications prioritizing low latency and high performance, configure regions closest to your users and application infrastructure.
**:flag_us: North America:**
```bash
# Primary region for lowest latency, with fallbacks
export AWS_S3_BUCKET=my-stdapi-us-east-1-bucket
export AWS_BEDROCK_REGIONS=us-east-1,us-west-2,us-east-2
# Enable all cross-region inference for maximum model availability
export AWS_BEDROCK_CROSS_REGION_INFERENCE=true
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=true
```
**:flag_jp: Asia-Pacific:**
```bash
# Use Asia-Pacific regions for lowest latency to APAC users
export AWS_S3_BUCKET=my-stdapi-ap-southeast-1-bucket
export AWS_BEDROCK_REGIONS=ap-southeast-1,ap-northeast-1,us-west-2
# Enable global inference for fallback to US regions if needed
export AWS_BEDROCK_CROSS_REGION_INFERENCE=true
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=true
```
**:earth_africa: Global Multi-Region:**
```bash
# Balanced configuration with worldwide coverage
export AWS_S3_BUCKET=my-stdapi-us-east-1-bucket
export AWS_BEDROCK_REGIONS=us-east-1,eu-west-1,ap-southeast-1,us-west-2
# Enable global inference for best availability
export AWS_BEDROCK_CROSS_REGION_INFERENCE=true
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=true
```
!!! tip "Latency Optimization Tips"
- :material-server: **Server and S3 co-location**: Deploy stdapi.ai and your `AWS_S3_BUCKET` in the first region specified in `AWS_BEDROCK_REGIONS` (your primary region)
- :material-network: **Network proximity**: Choose the first region based on low latency to your application servers and end users
- :material-cash: **Data transfer costs**: Cross-region data transfer incurs costs; co-locating server and S3 in the same region minimizes these
- :material-check-circle: **Model availability**: While `us-east-1` often has the most models, check specific model availability in your target regions
#### Hybrid Approach: Compliance with Performance
Balance compliance requirements with performance needs:
```bash title="EU Primary with US Fallback"
# EU primary with US fallback (for model availability)
export AWS_S3_BUCKET=my-stdapi-eu-bucket
export AWS_BEDROCK_REGIONS=eu-west-1,eu-central-1,us-east-1
# Allow cross-region but restrict to specific regions only
export AWS_BEDROCK_CROSS_REGION_INFERENCE=true
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false
```
!!! warning "Legal Compliance Notice"
Including `us-east-1` as a fallback region provides access to more models but may not comply with strict data residency requirements. **Consult your legal and compliance teams before using this configuration.**
---
## :material-sort-numeric-ascending: Configuration Order
When deploying stdapi.ai, configure settings in this recommended order:
1. **[IAM Permissions](operations_iam_permissions.md)** - Set up AWS access first
2. **[AWS Services and Regions](#aws-services-and-regions)** - Configure S3 buckets and Bedrock regions
3. **[Authentication](#authentication)** - Secure your API with authentication
4. **[Optional Features](#observability-opentelemetry)** - Add observability, guardrails, and other features as needed
---
## :material-shield-key: IAM Permissions { #iam-permissions }
The full IAM reference — required Amazon Bedrock permissions, per-feature policy statements, complete policy examples, and AWS tag policy requirements — has moved to the dedicated [IAM Permissions](operations_iam_permissions.md) page.
---
## :material-lock: Authentication
stdapi.ai supports three sources for API key authentication, plus Amazon Cognito user pool tokens.
!!! info "API Key Sources"
**Configure exactly one source.** If several are set, the first match in this precedence order is used and the others are ignored:
1. :material-key: **Direct API key** — `API_KEY` (highest precedence)
2. :material-database-lock: **SSM Parameter Store** — `API_KEY_SSM_PARAMETER`
3. :material-key-variant: **Secrets Manager** — `API_KEY_SECRETSMANAGER_SECRET` (lowest precedence)
The methods below are listed in that precedence order. **SSM Parameter Store remains the recommended method for production.**
:material-account-key: **Amazon Cognito user pool tokens** ([Method 4](#cognito-authentication)) are independent of the API key and can be accepted alongside it, or instead of it — [`AUTHENTICATION_MODE`](#authentication-mode) decides.
!!! warning "Conflicting Configuration"
Only one combination is rejected at startup: `API_KEY` set together with a Secrets Manager source (`API_KEY_SECRETSMANAGER_SECRET`). Every other combination starts normally and is resolved silently by the precedence order above — the lower-precedence sources are never read.
!!! danger "No Authentication Warning"
If neither an API key source nor a user pool is configured, the API accepts all requests without authentication and a security warning is logged at startup. This is suitable **only for internal/private deployments**.
### Method 1: Direct API Key
Provide the API key directly via environment variable. Intended for local development and testing; it takes precedence over both AWS-backed sources.
#### `API_KEY` { #api-key }
:octicons-package-24: **Purpose**
: Static API key value
:octicons-alert-24: **Security Warning**
: Avoid hardcoding in configuration files; use environment variables only
:octicons-person-24: **Client Usage**
: Clients must include this key in the `Authorization: Bearer ` header or `X-API-Key` header
```bash
export API_KEY=sk-1234567890abcdef...
```
### Method 2: SSM Parameter Store (Recommended)
**Recommended** - Use AWS Systems Manager Parameter Store for secure key storage with encryption, access control, and auditing. This method should be used only with **already existing** parameters.
#### `API_KEY_SSM_PARAMETER` { #api-key-ssm }
:octicons-package-24: **Purpose**
: Name of the SSM parameter containing the API key. The parameter is retrieved from the current region detected by the running container, or defaults to the first region in `AWS_BEDROCK_REGIONS`.
:octicons-shield-check-24: **Recommendation**
: Use `SecureString` type for encryption at rest
:octicons-lock-24: **IAM Permissions Required**
: `ssm:GetParameter`, `kms:Decrypt` (if encrypted)
```bash
export API_KEY_SSM_PARAMETER=/stdapi/prod/api-key
```
### Method 3: Secrets Manager
Use AWS Secrets Manager for secure key storage with automatic rotation support. This method should be used only with **already existing** secrets.
#### `API_KEY_SECRETSMANAGER_SECRET` { #api-key-secretsmanager-secret }
:octicons-package-24: **Purpose**
: Name of the Secrets Manager secret containing the API key. The secret is retrieved from the current region detected by the running container, or defaults to the first region in `AWS_BEDROCK_REGIONS`.
:octicons-code-24: **Format**
: Can be a plain string or JSON object
:octicons-lock-24: **IAM Permissions Required**
: `secretsmanager:GetSecretValue`
#### `API_KEY_SECRETSMANAGER_KEY` { #api-key-secretsmanager-key }
:octicons-package-24: **Purpose**
: JSON key name within the secret (if the secret is a JSON object)
:octicons-gear-24: **Default**
: `api_key`
**Plain String Secret:**
```bash
export API_KEY_SECRETSMANAGER_SECRET=stdapi-api-key
```
**JSON Secret:**
```bash
export API_KEY_SECRETSMANAGER_SECRET=stdapi-credentials
export API_KEY_SECRETSMANAGER_KEY=api_key
```
Example JSON secret structure:
```json
{
"api_key": "sk-1234567890abcdef...",
"other_config": "value"
}
```
### Method 4: Amazon Cognito User Pool Tokens { #cognito-authentication }
Accept the bearer tokens issued by an [Amazon Cognito user pool](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools.html) instead of, or alongside, the API key. Each caller gets its own short-lived credential, and the verified caller is the identity [per-user cost attribution](operations_cost_management.md#per-user-attribution) bills against; withdrawing a caller's access takes effect when their current token expires. Clients send the token in the `Authorization: Bearer ` or `X-API-Key` header, like an API key. What is validated on every request is described in [Authentication & Security](operations_authentication_security.md#amazon-cognito-user-pool-tokens).
```bash
export AWS_COGNITO_USER_POOL_ID=eu-west-3_a1b2c3d4e
export AWS_COGNITO_CLIENT_IDS=1example23456789abcdefghij
```
!!! warning "Incomplete configuration fails startup"
A user pool without `AWS_COGNITO_CLIENT_IDS`, a Cognito setting without a pool, or an `AUTHENTICATION_MODE` that contradicts what is configured, all stop the server at startup with an explicit message — a partially configured pool never degrades into an unauthenticated deployment.
!!! tip "The pool also configures agent discovery"
Add [`OAUTH_RESOURCE_IDENTIFIER`](#oauth-resource-identifier) and an AI agent can authenticate itself against the deployment. Nothing else is needed: the pool's issuer and required scopes are what get published — see [Authentication Discovery for Agents](#oauth-discovery).
#### `AUTHENTICATION_MODE` { #authentication-mode }
:octicons-package-24: **Purpose**
: Which client authentication methods the deployment accepts
:octicons-gear-24: **Default**
: `any` — every method that is configured
:octicons-list-unordered-24: **Values**
: - `any`: the API key and user pool tokens, whichever is configured
- `api_key`: the API key only; startup fails if a user pool is also configured
- `cognito`: user pool tokens only; startup fails if an API key source is also configured
```bash
export AUTHENTICATION_MODE=cognito
```
#### `AWS_COGNITO_USER_POOL_ID` { #aws-cognito-user-pool-id }
:octicons-package-24: **Purpose**
: Identifier of the user pool whose tokens authenticate clients. Setting it enables the method; the pool's AWS Region is read from the identifier itself, and the public signing keys are loaded from that Region at startup.
:octicons-gear-24: **Default**
: None — user pool tokens are not accepted
:octicons-alert-24: **Requirement**
: `AWS_COGNITO_CLIENT_IDS` must be set too
:octicons-lock-24: **IAM Permissions Required**
: None — the signing keys are public
```bash
export AWS_COGNITO_USER_POOL_ID=eu-west-3_a1b2c3d4e
```
#### `AWS_COGNITO_CLIENT_IDS` { #aws-cognito-client-ids }
:octicons-package-24: **Purpose**
: Comma-separated app client IDs whose tokens are accepted. A token issued to any other app client of the pool is rejected.
:octicons-gear-24: **Default**
: Empty — startup fails when a user pool is configured without it
:octicons-alert-24: **Requirement**
: Required whenever `AWS_COGNITO_USER_POOL_ID` is set
```bash
export AWS_COGNITO_CLIENT_IDS=1example23456789abcdefghij,2example3456789abcdefghijk
```
#### `AWS_COGNITO_REQUIRED_SCOPES` { #aws-cognito-required-scopes }
:octicons-package-24: **Purpose**
: Comma-separated OAuth 2.0 scopes a token must **all** carry to be accepted
:octicons-gear-24: **Default**
: None — any scope set is accepted
:octicons-alert-24: **Requirement**
: Custom scopes exist only on tokens issued by the pool's OAuth 2.0 token endpoint, which needs a [resource server](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-define-resource-servers.html) and a pool domain. Tokens obtained by signing in with a username and password carry only `aws.cognito.signin.user.admin` and are rejected when a custom scope is required.
```bash
export AWS_COGNITO_REQUIRED_SCOPES=stdapi/invoke
```
#### `AWS_COGNITO_ACCEPT_ID_TOKEN` { #aws-cognito-accept-id-token }
:octicons-package-24: **Purpose**
: Also accept identity tokens, not only access tokens
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Effect**
: Identity tokens describe the signed-in user rather than granting API access, and carry no scopes. Enable only for clients that cannot obtain an access token.
```bash
export AWS_COGNITO_ACCEPT_ID_TOKEN=true
```
#### `AWS_COGNITO_ISSUER_TYPE` { #aws-cognito-issuer-type }
:octicons-package-24: **Purpose**
: The pool's [issuer configuration](https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_IssuerConfigurationType.html), which decides the issuer URL its tokens carry
:octicons-gear-24: **Default**
: `original`
:octicons-list-unordered-24: **Values**
: - `original`: `https://cognito-idp..amazonaws.com/`
- `updated`: `https://issuer-cognito-idp..amazonaws.com/`, available on the Essentials and Plus pool tiers
:octicons-alert-24: **Requirement**
: Must match the pool's own setting; tokens whose issuer differs are rejected
```bash
export AWS_COGNITO_ISSUER_TYPE=updated
```
### Authentication Discovery for Agents { #oauth-discovery }
Publishes, at `/.well-known/oauth-protected-resource`, where clients obtain a token, and points every `401 Unauthorized` at that document. An AI agent — or any MCP client — can then authenticate against this deployment without having been configured for it first. See [Authentication Discovery for Agents](operations_authentication_security.md#authentication-discovery-for-agents) for the full flow.
Nothing is published until `OAUTH_RESOURCE_IDENTIFIER` is set. With an [Amazon Cognito user pool](#cognito-authentication) configured, that variable is the only one to set: the pool already names the issuer and the scopes, and both are published from it. The document is public and unauthenticated, since a client reads it before it has any credential.
#### `OAUTH_RESOURCE_IDENTIFIER` { #oauth-resource-identifier }
:octicons-package-24: **Purpose**
: Public URL clients use to reach this deployment, published as the identity of the protected resource
:octicons-gear-24: **Default**
: None — no discovery document is published, and `401` responses only state that a bearer token is expected
:octicons-alert-24: **Requirement**
: Must be the exact origin clients dial — scheme and host, an explicit port only when it is not the default one for the scheme, and no path, query or fragment. Clients compare it character by character against the URL they used, so `https://api.example.com` and `https://api.example.com:443` are not interchangeable. Requires `OAUTH_AUTHORIZATION_SERVERS`, unless a user pool supplies the issuer.
```bash
export OAUTH_RESOURCE_IDENTIFIER=https://api.example.com
```
#### `OAUTH_AUTHORIZATION_SERVERS` { #oauth-authorization-servers }
:octicons-package-24: **Purpose**
: Issuer URLs of the OAuth 2.0 authorization servers that issue tokens for this deployment, comma-separated
:octicons-gear-24: **Default**
: The issuer of the [`AWS_COGNITO_USER_POOL_ID`](#aws-cognito-user-pool-id) pool, when one is configured — otherwise none
:octicons-workflow-24: **Effect**
: A client reads each issuer's own metadata to find where to sign in, so this deployment never describes the sign-in flow itself. A load balancer or API gateway authenticating in front of stdapi.ai publishes the issuer of whichever provider it uses.
With a user pool configured, leave this unset: the pool issues the tokens the deployment accepts, so its own issuer is published — `https://cognito-idp..amazonaws.com/`, or `https://issuer-cognito-idp..amazonaws.com/` when [`AWS_COGNITO_ISSUER_TYPE`](#aws-cognito-issuer-type) is `updated`. `` and `` come from the pool ID itself, and the host follows the pool Region's AWS partition (`amazonaws.com.cn` in China, `amazonaws.eu` in the European Sovereign Cloud). Set the variable only to publish further issuers.
:octicons-alert-24: **Requirement**
: Each entry is an `https` URL with no query or fragment. Required when `OAUTH_RESOURCE_IDENTIFIER` is set and no user pool is configured. When one is, the list must include the pool's own issuer — a client sent anywhere else obtains a token every request refuses, so startup fails instead.
```bash
export OAUTH_AUTHORIZATION_SERVERS=https://cognito-idp.eu-west-3.amazonaws.com/eu-west-3_a1b2c3d4e
```
#### `OAUTH_SCOPES_SUPPORTED` { #oauth-scopes-supported }
:octicons-package-24: **Purpose**
: Scopes a token needs to call this API, comma-separated
:octicons-gear-24: **Default**
: [`AWS_COGNITO_REQUIRED_SCOPES`](#aws-cognito-required-scopes) — with neither set, no scope is advertised and a client asks for whatever its own configuration names
:octicons-workflow-24: **Effect**
: Advertised both in the discovery document and in the `401` challenge, so a client asks its authorization server for the right scopes on its first attempt. The scopes a token must carry to be accepted are exactly the scopes to ask for, so they are published unless this variable names others.
```bash
export OAUTH_SCOPES_SUPPORTED=stdapi/invoke
```
---
## :material-api: API Compatibility
Configure the base URL paths for OpenAI and Anthropic-compatible API routes.
#### `OPENAI_ROUTES_PREFIX` { #openai-routes-prefix }
:octicons-package-24: **Purpose**
: Base path prefix for OpenAI-compatible API routes
:octicons-gear-24: **Default**
: `` (empty, routes mounted at root)
:octicons-alert-24: **Requirement**
: Empty, or a path starting with `/` with no trailing slash, using only alphanumeric characters and `. _ ~ -` per segment; must differ from `ANTHROPIC_ROUTES_PREFIX` and `COHERE_ROUTES_PREFIX`
:octicons-workflow-24: **Effect**
: All OpenAI-compatible endpoints will be mounted under this prefix
```bash
export OPENAI_ROUTES_PREFIX=/api
```
!!! example "Example Endpoints"
With the prefix `/api`, endpoints are available at:
- `/api/v1/chat/completions`
- `/api/v1/models`
- `/api/v1/embeddings`
#### `ANTHROPIC_ROUTES_PREFIX` { #anthropic-routes-prefix }
:octicons-package-24: **Purpose**
: Base path prefix for Anthropic-compatible API routes
:octicons-gear-24: **Default**
: `/anthropic`
:octicons-alert-24: **Requirement**
: A path starting with `/` with no trailing slash, using only alphanumeric characters and `. _ ~ -` per segment; must differ from `OPENAI_ROUTES_PREFIX` and `COHERE_ROUTES_PREFIX`
:octicons-workflow-24: **Effect**
: All Anthropic-compatible endpoints will be mounted under this prefix
```bash
export ANTHROPIC_ROUTES_PREFIX=/anthropic
```
!!! example "Example Endpoints"
With the default prefix `/anthropic`, endpoints are available at:
- `/anthropic/v1/messages`
!!! tip "Custom Prefix"
You can change the prefix to match your organization's API structure:
```bash
export ANTHROPIC_ROUTES_PREFIX=/api/anthropic
```
This would mount the Messages API at `/api/anthropic/v1/messages`
#### `COHERE_ROUTES_PREFIX` { #cohere-routes-prefix }
:octicons-package-24: **Purpose**
: Base path prefix for Cohere-compatible API routes
:octicons-gear-24: **Default**
: `/cohere`
:octicons-alert-24: **Requirement**
: A path starting with `/` with no trailing slash, using only alphanumeric characters and `. _ ~ -` per segment; must differ from `OPENAI_ROUTES_PREFIX` and `ANTHROPIC_ROUTES_PREFIX`
:octicons-workflow-24: **Effect**
: All Cohere-compatible endpoints will be mounted under this prefix
```bash
export COHERE_ROUTES_PREFIX=/cohere
```
!!! example "Example Endpoints"
With the default prefix `/cohere`, endpoints are available at:
- `/cohere/v2/rerank`
---
## :material-web: CORS Configuration
Configure Cross-Origin Resource Sharing (CORS) to control which web origins can access your API from browsers.
#### `CORS_ALLOW_ORIGINS` { #cors-allow-origins }
:octicons-package-24: **Purpose**
: List of origins allowed to make cross-origin requests
:octicons-list-ordered-24: **Format**
: JSON array of origin URLs
:octicons-gear-24: **Default**
: `None` (CORS not enabled)
:octicons-shield-check-24: **Best Practice**
: Only enable if your API is accessed from web browsers; specify exact origins in production
```bash
# Not configured (default) - CORS middleware not enabled
# Browser cross-origin requests will be blocked
# No environment variable needed
# Development: Allow all origins
export CORS_ALLOW_ORIGINS='["*"]'
# Production: Specific origins only
export CORS_ALLOW_ORIGINS='["https://myapp.com", "https://app.example.com"]'
# Multiple environments
export CORS_ALLOW_ORIGINS='["https://app.example.com", "https://staging.example.com"]'
```
!!! info "What is CORS?"
Cross-Origin Resource Sharing (CORS) is a browser security mechanism that restricts web pages from making requests to a different domain than the one serving the web page.
**Without CORS enabled:**
- Browser requests from web applications will fail due to missing CORS headers
- Non-browser clients (curl, SDKs, mobile apps, server-to-server) work normally
- Most secure default - no cross-origin access from browsers
**With CORS enabled:**
- Browsers can make requests from allowed origins
- Preflight OPTIONS requests are handled automatically
- Non-browser clients continue to work normally
!!! warning "Security Consideration"
- **Default (not configured)**: CORS is disabled. Browser cross-origin requests will fail. This is the most secure default.
- **`["*"]`**: Allows requests from any web origin. Convenient for development but not recommended for production.
- **Specific origins**: Only allows requests from listed origins. Recommended for production.
!!! note "CORS Behavior"
- When `CORS_ALLOW_ORIGINS` is not configured (default), CORS is **not enabled**
- When configured with specific origins or `["*"]`, CORS is enabled with:
- Authorization headers with credentials allowed
- All HTTP methods allowed
- All request headers allowed
!!! tip "When to Configure"
Configure `CORS_ALLOW_ORIGINS` when:
- :material-web: Your API is accessed from browser-based web applications (React, Vue, Angular, etc.)
- :material-application-brackets: Building a web frontend that calls your API from a different domain
- :material-dev-to: Developing locally with web apps (browser at `localhost:3000` calling API at `localhost:8000`)
!!! tip "When NOT to Configure"
Do **not** configure CORS when:
- :material-server: Your API is only accessed from server-to-server integrations
- :material-cellphone: Your API is only accessed from mobile apps or desktop clients
- :material-console: Your API is only accessed from CLI tools or SDKs
- :material-api: Your API is only accessed from non-browser HTTP clients
**Non-browser clients don't enforce CORS**, so enabling it is unnecessary overhead.
---
## :material-server-security: Trusted Host Configuration
Configure Host header validation to protect against Host header injection attacks.
#### `TRUSTED_HOSTS` { #trusted-hosts }
:octicons-package-24: **Purpose**
: List of trusted Host header values for validation
:octicons-list-ordered-24: **Format**
: JSON array of hostnames (supports wildcards)
:octicons-gear-24: **Default**
: `None` (no Host header validation)
:octicons-shield-check-24: **Best Practice**
: Use AWS ALB host-based routing rules instead when possible for better performance and management
```bash
# Production: Specific hosts only
export TRUSTED_HOSTS='["api.example.com", "www.example.com"]'
```
!!! info "What is Host Header Validation?"
The Host header in HTTP requests specifies the domain name of the server. Validating it prevents **Host header injection attacks** (manipulated Host headers used to poison caches or exploit application logic) and **web cache poisoning**.
!!! warning "Security Consideration: prefer ALB host-based routing"
Configure **AWS ALB listener rules** to validate the Host header and forward traffic only for approved hostnames — this rejects bad requests at the load balancer, before they reach the application, and is centrally managed. See the example below.
Use `TRUSTED_HOSTS` only when you can't configure host-based routing at the load balancer level (no ALB, or you need application-level defense-in-depth).
!!! tip "Wildcard Support"
- `*.example.com` matches any subdomain (`api.example.com`, `app.example.com`, ...)
- `example.com` matches only the exact domain
- `*` matches all hosts — not recommended, equivalent to no validation
!!! example "Common Configurations"
**Multi-Domain with Subdomains:**
```bash
export TRUSTED_HOSTS='["*.example.com", "*.myapp.com", "api.production.com"]'
```
**Development and Production:**
```bash
export TRUSTED_HOSTS='["api.example.com", "localhost", "127.0.0.1"]'
```
!!! note "Host Validation Behavior"
- Not configured (default): Host header validation is **not enabled**
- Configured: requests with a non-matching Host header are rejected with **HTTP 400 Bad Request**
!!! info "Container health probe"
Validation applies to `/health` like any other path, so the container image's `HEALTHCHECK` derives its `Host` header from this setting: it requests `/health` on `127.0.0.1:$GRANIAN_PORT` announcing the **first** entry of `TRUSTED_HOSTS`. `*` or an unset value becomes `localhost`, and a leading `*.` becomes `healthcheck.` (so `*.example.com` is probed as `healthcheck.example.com`).
A correct list therefore keeps the container healthy with no extra entry to add. Do not replace the probe with a hand-written `curl` call in a Compose `healthcheck:` block or an ECS task definition `healthCheck`: it would send an untrusted `Host` and get a `400`.
!!! warning "Load balancer health checks are rejected by default"
An ALB or NLB target-group health check does **not** send your domain name: it addresses the target directly, so the `Host` header carries the target's IP address. With `TRUSTED_HOSTS` set to domain names, every one of those probes gets **HTTP 400**, the target never turns healthy, and the load balancer serves `503` — a failure that looks like a broken deployment rather than a configuration choice.
Target-group health-check settings offer no `Host` header override, so either keep the Host allow-list at the load balancer (the recommended option above, leaving `TRUSTED_HOSTS` unset) or make sure the address the health check actually sends is in the list.
!!! success "AWS ALB Host-Based Routing Example"
**Via AWS Console:** EC2 → Load Balancers → Your ALB → Listeners → add a rule on the HTTPS (443) listener with condition "Host header" is `api.example.com`, forwarding to the target group only on match.
**Via AWS CLI:**
```bash
aws elbv2 create-rule \
--listener-arn arn:aws:elasticloadbalancing:... \
--priority 1 \
--conditions Field=host-header,Values=api.example.com \
--actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:...
```
Benefits: rejected at the load balancer (better performance, reduced load on application servers), centralized policy management, and ALB metrics/logging for rejected requests.
---
## :material-swap-horizontal: Proxy Headers Configuration
Configure X-Forwarded-* header processing when running behind reverse proxies or load balancers.
#### `ENABLE_PROXY_HEADERS` { #enable-proxy-headers }
:octicons-package-24: **Purpose**
: Enable trusting X-Forwarded-* headers from reverse proxies
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false` (disabled)
:octicons-shield-check-24: **Best Practice**
: Only enable when running behind a trusted reverse proxy
```bash
# Disabled (default) - do not trust X-Forwarded-* headers
# No environment variable needed
# Enable when behind reverse proxy
export ENABLE_PROXY_HEADERS=true
```
!!! info "What are X-Forwarded Headers?"
When your application runs behind a reverse proxy (nginx, Apache, AWS ALB, CloudFront, etc.), the proxy sits between clients and your application. Without proxy header processing:
- The application sees the proxy's IP address instead of the client's real IP
- The application sees the proxy-to-app connection (e.g., HTTP) instead of the original client connection (e.g., HTTPS)
- The application cannot distinguish between different clients behind the proxy
Reverse proxies add `X-Forwarded-*` headers to preserve the original request information:
- **X-Forwarded-For** - Client's real IP address (and chain of proxies)
- **X-Forwarded-Proto** - Original protocol (http/https)
- **X-Forwarded-Port** - Original port number
!!! warning "Security Warning"
**CRITICAL**: Only enable `ENABLE_PROXY_HEADERS` when running behind a **trusted** reverse proxy that properly sets X-Forwarded-* headers.
**If enabled without a trusted proxy:**
- :material-alert: Clients can spoof their IP address by sending fake X-Forwarded-For headers
- :material-shield-alert: Security controls based on client IP (rate limiting, allowlists) can be bypassed
- :material-bug: Logging and monitoring will record incorrect client information
- :material-lock-open: Authentication and authorization decisions may be affected
**Never enable this setting if your application is directly exposed to the internet without a reverse proxy.**
!!! example "Common Deployment Scenarios"
**Scenario 1: Direct to Internet (No Proxy)**
```bash
# Do NOT enable proxy headers
# ENABLE_PROXY_HEADERS should remain false (default)
```
Your application receives requests directly from clients.
**Scenario 2: Behind AWS ALB/CloudFront**
```bash
export ENABLE_PROXY_HEADERS=true
```
AWS load balancer or CDN forwards requests to your application.
**Scenario 3: Multiple AWS Proxy Layers**
```bash
export ENABLE_PROXY_HEADERS=true
```
Example: CloudFront → ALB → Your Application
!!! note "Proxy Headers Behavior"
- When `ENABLE_PROXY_HEADERS` is `false` (default), X-Forwarded-* headers are **not trusted**
- When enabled, the server processes X-Forwarded-For, X-Forwarded-Proto, and X-Forwarded-Port headers to determine client information
- Which peers' headers are trusted is controlled by [`PROXY_TRUSTED_HOSTS`](#proxy-trusted-hosts) — the default `*` trusts every peer, so restrict it to your reverse proxy's IP range
!!! tip "When to Enable"
Enable `ENABLE_PROXY_HEADERS` when:
- :material-aws: Deployed behind AWS ALB, NLB, API Gateway, or CloudFront
- :material-network: Running behind any reverse proxy that sets X-Forwarded-* headers
!!! info "AWS Proxy Configuration"
**AWS ALB, NLB, and CloudFront** automatically set X-Forwarded-* headers - no additional configuration needed.
When you enable `ENABLE_PROXY_HEADERS=true`, your application will trust these headers to determine:
- Client's real IP address (from X-Forwarded-For)
- Original protocol (from X-Forwarded-Proto: http/https)
- Original port (from X-Forwarded-Port)
#### `PROXY_TRUSTED_HOSTS` { #proxy-trusted-hosts }
:octicons-package-24: **Purpose**
: Restrict which peer IPs may set trusted `X-Forwarded-*` headers when `ENABLE_PROXY_HEADERS` is enabled
:octicons-database-24: **Type**
: JSON array of IPs/CIDRs, or `*`
:octicons-gear-24: **Default**
: `*` (trust every peer — backward compatible)
:octicons-shield-check-24: **Best Practice**
: Restrict to your reverse proxy's IP range so direct clients cannot spoof `X-Forwarded-For`
```bash
# Trust forwarded headers only from the VPC / proxy range
export ENABLE_PROXY_HEADERS=true
export PROXY_TRUSTED_HOSTS='["10.0.0.0/8"]'
```
!!! warning "Only effective with `ENABLE_PROXY_HEADERS=true`"
This setting has no effect unless [`ENABLE_PROXY_HEADERS`](#enable-proxy-headers) is enabled. With the default `*`, any client that can reach the server directly can forge `X-Forwarded-For`, poisoning the client IP recorded in logs and OpenTelemetry spans. Restrict it to the address range of your load balancer or reverse proxy (AWS ALB/CloudFront, nginx, etc.).
!!! tip "Configured automatically by the official Terraform module"
The [stdapi-ai Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) sets this for you when the ALB is enabled with client IP logging (`alb_enabled = true`, `log_client_ip = true`): it enables proxy headers and pins `PROXY_TRUSTED_HOSTS` to the ALB's subnet CIDRs, so only the load balancer is trusted and direct clients cannot forge `X-Forwarded-For`. Override it with the module's `proxy_trusted_hosts` variable when fronting the ALB with an additional proxy (for example CloudFront).
!!! warning "On a dual-stack listener, cover the IPv4-mapped form too"
With `GRANIAN_HOST=::` the operating system reports an IPv4 peer as an IPv4-mapped IPv6 address such as `::ffff:10.0.1.5`, which belongs to no IPv4 network and therefore matches no IPv4 entry here. Add the mapped range alongside the plain one — an IPv4 `/16` becomes a `/112` once the 96-bit mapping prefix is counted:
```bash
export PROXY_TRUSTED_HOSTS='["10.0.0.0/16", "::ffff:10.0.0.0/112"]'
```
Miss it and the proxy stops being trusted: `X-Forwarded-For` is ignored and the load balancer's own address is recorded as the client IP. The Terraform module derives these entries for you, including for values passed to its `proxy_trusted_hosts` variable.
---
## :material-certificate: TLS / SSL Configuration
Configure end-to-end TLS encryption within the container. These are native [Granian](https://github.com/emmett-framework/granian) environment variables and are available with the provided container images.
#### `GRANIAN_SSL_CERTIFICATE` { #graniansslcertificate }
:octicons-package-24: **Purpose**
: Path to the SSL certificate file
:octicons-database-24: **Type**
: File path
#### `GRANIAN_SSL_KEYFILE` { #graniansslkeyfile }
:octicons-package-24: **Purpose**
: Path to the SSL private key file (PKCS#8 format only)
:octicons-database-24: **Type**
: File path
#### `GRANIAN_SSL_KEYFILE_PASSWORD` { #graniansslkeyfilepassword }
:octicons-package-24: **Purpose**
: Password for the private key file
:octicons-database-24: **Type**
: String
#### `GRANIAN_SSL_PROTOCOL_MIN` { #graniansslprotocolmin }
:octicons-package-24: **Purpose**
: Minimum supported TLS version (`tls1.2` or `tls1.3`)
:octicons-database-24: **Type**
: Enum
:octicons-gear-24: **Default**
: `tls1.3`
#### `GRANIAN_SSL_CA` { #graniansslca }
:octicons-package-24: **Purpose**
: Path to the CA certificate bundle used to verify client certificates (mTLS)
:octicons-database-24: **Type**
: File path
#### `GRANIAN_SSL_CLIENT_VERIFY` { #graniansslclientverify }
:octicons-package-24: **Purpose**
: Enable client certificate verification (mTLS)
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false` (disabled)
---
## :material-zip-box: GZip Compression
Configure automatic GZip compression for HTTP responses to reduce bandwidth usage and improve response times.
#### `ENABLE_GZIP` { #enable-gzip }
:octicons-package-24: **Purpose**
: Enable GZip compression for HTTP responses
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false` (disabled)
:octicons-zap-24: **Best Practice**
: Use AWS ALB or CloudFront compression instead when available for better performance
```bash
# Disabled (default) - no response compression
# No environment variable needed
# Enable GZip compression (responses larger than 1 KiB will be compressed)
export ENABLE_GZIP=true
```
!!! info "How GZip Compression Works"
When enabled, the server automatically:
1. :material-file-check: Checks if the response size exceeds 1 KiB (1024 bytes)
2. :material-web: Verifies the client supports compression (via `Accept-Encoding: gzip` header)
3. :material-zip-box: Compresses the response body using gzip
4. :material-arrow-down: Adds `Content-Encoding: gzip` header to the response
Typical compression ratios for JSON responses: **60-80% size reduction**
!!! success "Recommended: Use AWS Compression Services"
Instead of enabling application-level compression, enable compression at the AWS layer — it offloads the CPU cost from your application servers, at the price of managing it in AWS instead of a single environment variable:
- **AWS ALB** — enable the `compression.enabled` target group attribute ([documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#compression))
- **Amazon CloudFront** — enable "Compress Objects Automatically" in the distribution behavior settings ([documentation](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html))
!!! tip "When to Enable Application-Level Compression"
Enable `ENABLE_GZIP` only when:
- :material-server-off: You're **not** using AWS ALB or CloudFront
- :material-wan: Your API returns large JSON responses and you want to reduce bandwidth
- :material-dev-to: Local development or non-AWS deployments
!!! warning "When NOT to Enable"
Do **not** enable when:
- :material-aws: You're behind AWS ALB with compression enabled
- :material-cloud: You're using CloudFront with compression enabled
- :material-speedometer-slow: CPU usage is a concern (compression adds CPU overhead)
**Enabling compression at multiple layers is redundant and wastes CPU resources.**
!!! note "Compression Behavior"
- When `ENABLE_GZIP` is `false` (default), compression is **not enabled**
- When enabled, only responses meeting these criteria are compressed:
- Response size ≥ 1 KiB (1024 bytes)
- Client sends `Accept-Encoding: gzip` header
- Response does not already have `Content-Encoding` header
- Streaming responses are compressed on-the-fly
---
## :material-connection: MCP (Model Context Protocol)
When enabled, stdapi.ai exposes its API endpoints as MCP tools, allowing AI clients and agents to call them directly using the Model Context Protocol. The full list of available tool names is documented in [API Overview → MCP Tools](api_overview.md#mcp-model-context-protocol).
Both transport types can be enabled independently or simultaneously.
#### `ENABLE_MCP_STREAMABLE_HTTP` { #enable-mcp-streamable-http }
:octicons-package-24: **Purpose**
: Enable the MCP server using Streamable HTTP transport — the recommended method
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Behavior**
: Exposes an MCP-compatible endpoint at `/mcp`. AI clients connect using standard HTTP requests following the MCP Streamable HTTP specification.
```bash
# Disabled (default)
# No environment variable needed
# Enable MCP Streamable HTTP transport
export ENABLE_MCP_STREAMABLE_HTTP=true
```
#### `MCP_STATELESS_HTTP` { #mcp-stateless-http }
:octicons-package-24: **Purpose**
: Serve the Streamable HTTP transport without server-side sessions
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Behavior**
: Each request to `/mcp` is handled by a fresh transport that keeps no state. Clients may call `tools/list` and `tools/call` without an `initialize` handshake, an `Mcp-Session-Id` the server never issued is accepted rather than rejected, and any replica may serve any request.
:octicons-alert-24: **Requires**
: `ENABLE_MCP_STREAMABLE_HTTP=true`. Ignored otherwise.
```bash
# Sessions enabled (default)
# No environment variable needed
# Stateless transport
export ENABLE_MCP_STREAMABLE_HTTP=true
export MCP_STATELESS_HTTP=true
```
#### `ENABLE_MCP_SSE` { #enable-mcp-sse }
:octicons-package-24: **Purpose**
: Enable the MCP server using Server-Sent Events (SSE) transport
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Behavior**
: Exposes MCP endpoints at `/sse` for AI clients that require the SSE transport protocol.
```bash
# Disabled (default)
# No environment variable needed
# Enable MCP SSE transport
export ENABLE_MCP_SSE=true
```
!!! info "Transport Recommendation"
**HTTP transport (`ENABLE_MCP_STREAMABLE_HTTP`) is the recommended method.** It implements the latest MCP Streamable HTTP specification and provides better session management and more robust connection handling.
**SSE transport (`ENABLE_MCP_SSE`)** is maintained for backwards compatibility with older MCP client implementations. Prefer HTTP for new deployments.
Both transports can be enabled simultaneously to support clients with different requirements:
```bash
export ENABLE_MCP_STREAMABLE_HTTP=true
export ENABLE_MCP_SSE=true
```
The MCP server card (`/.well-known/mcp/server-card.json`) declares a single transport: Streamable HTTP (`/mcp`) whenever it is enabled, otherwise SSE (`/sse`). When both are enabled, `/sse` is therefore not listed in the card, but it remains fully functional for clients configured with it explicitly.
#### `MCP_INCLUDE_TOOLS` { #mcp-include-tools }
:octicons-package-24: **Purpose**
: Expose only a specific subset of MCP tools; all others are hidden
:octicons-code-24: **Format**
: Comma-separated list of tool names (duplicates are automatically removed)
:octicons-gear-24: **Default**
: None (all tools exposed)
```bash
# All tools exposed by default
# No environment variable needed
# Expose only specific tools
export MCP_INCLUDE_TOOLS="openai_chat_completion,openai_embedding,search_models"
# When both MCP_INCLUDE_TOOLS and MCP_EXCLUDE_TOOLS are specified,
# tools in MCP_EXCLUDE_TOOLS are removed from MCP_INCLUDE_TOOLS:
export MCP_INCLUDE_TOOLS="openai_chat_completion,openai_embedding,search_models"
export MCP_EXCLUDE_TOOLS="openai_files_delete,anthropic_files_delete"
# Result: only openai_chat_completion, openai_embedding, search_models are exposed
```
See [API Overview → MCP Tools](api_overview.md#mcp-model-context-protocol) for the full list of available tool names.
!!! warning "Token Usage for Complex API Tools"
`anthropic_message`, `openai_chat_completion`, and `openai_response` map to large, complex APIs that may use many tokens (prompt, completion, and tool definitions). Select these tools only if your workflow requires the full API capabilities.
#### `MCP_EXCLUDE_TOOLS` { #mcp-exclude-tools }
:octicons-package-24: **Purpose**
: Hide specific MCP tools from clients; all others remain exposed
:octicons-code-24: **Format**
: Comma-separated list of tool names (duplicates are automatically removed)
:octicons-gear-24: **Default**
: None (no tools excluded)
!!! note "Behavior with `MCP_INCLUDE_TOOLS`"
When both `MCP_INCLUDE_TOOLS` and `MCP_EXCLUDE_TOOLS` are specified, tools in `MCP_EXCLUDE_TOOLS` are removed from `MCP_INCLUDE_TOOLS`. The remaining tools in `MCP_INCLUDE_TOOLS` are what get exposed.
```bash
# No tools excluded by default
# No environment variable needed
# Exclude destructive tools
export MCP_EXCLUDE_TOOLS="openai_files_delete,anthropic_files_delete"
```
See [API Overview → MCP Tools](api_overview.md#mcp-model-context-protocol) for the full list of available tool names.
### Tool Selection Best Practices { #tool-selection-best-practices }
stdapi.ai exposes a fixed set of tools derived from its API surface — you can include or exclude them by name, but cannot modify or rename them. See [API Overview → MCP Tools](api_overview.md#mcp-model-context-protocol) for the full catalog.
**Start from the minimum, not the maximum**
By default all tools are exposed. It is safer and more effective to begin with a narrow `MCP_INCLUDE_TOOLS` list covering only what the workflow needs, then expand it deliberately. LLMs perform better with fewer choices, and many AI providers cap the number of active tools per session.
**Always include `search_models` for agent model discovery**
`search_models` is the recommended tool for agents to discover available model IDs — it supports capability-based filtering (by modality, route, region, streaming support) and returns richer metadata than `openai_model_list` or `anthropic_model_list`. Include it in every agent configuration so the agent can resolve the right model dynamically rather than relying on hardcoded IDs:
```bash
export MCP_INCLUDE_TOOLS="openai_chat_completion,search_models,openai_embedding"
```
**Always exclude file deletion tools unless required**
Uploaded files are the only durable, stateful data managed by stdapi.ai — deletion is permanent and cannot be undone. Unless your workflow explicitly needs to delete files, always suppress these tools:
```bash
export MCP_EXCLUDE_TOOLS="openai_files_delete,anthropic_files_delete"
```
**Exclude high-cost tools unless the workflow requires them**
Image generation (`openai_image_generation`, `openai_image_edit`, `openai_image_variation`) and speech synthesis (`openai_audio_speech`) incur a per-call cost that accumulates quickly if an agent invokes them speculatively. Only include them when the use case calls for it and the agent's decision to generate images or audio is intentional.
**Use `MCP_INCLUDE_TOOLS` for the tightest control**
For predictable, well-defined workflows, listing tools explicitly with `MCP_INCLUDE_TOOLS` is more reliable than maintaining an exclusion list. For example, a workflow limited to text generation and model discovery needs only:
```bash
export MCP_INCLUDE_TOOLS="openai_chat_completion,search_models"
```
!!! note
Health and metadata endpoints are never exposed as MCP tools, so they do not need to be listed in `MCP_EXCLUDE_TOOLS`.
---
## :material-shield-alert: SSRF Protection
Configure Server-Side Request Forgery (SSRF) protection to prevent unauthorized access to internal networks.
#### `SSRF_PROTECTION_BLOCK_PRIVATE_NETWORKS` { #ssrf-protection-block-private-networks }
:octicons-package-24: **Purpose**
: Enable SSRF protection by blocking requests to private/local networks
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `true` (enabled for security)
:octicons-shield-check-24: **Best Practice**
: Keep enabled in production to protect against SSRF attacks
```bash
# Enabled (default) - block private networks
# No environment variable needed
# Disable only in controlled environments that need local network access
export SSRF_PROTECTION_BLOCK_PRIVATE_NETWORKS=false
```
!!! info "What is SSRF Protection?"
Server-Side Request Forgery (SSRF) is an attack where an attacker can make the server send requests to unintended destinations, including internal network resources.
**SSRF protection has two layers:**
1. **Baseline Protection (Always Enabled)** - Cannot be disabled:
- :material-loopback: **Loopback Addresses** - 127.0.0.0/8, ::1
- :material-network-off: **Unspecified Addresses** - 0.0.0.0, ::
- :material-link: **Link-Local Addresses** - 169.254.0.0/16, fe80::/10
- :material-network-off: **Reserved IP Ranges** - IETF reserved addresses
- :material-network-off: **Multicast Addresses** - Multicast IP ranges
2. **Private Network Protection (Controlled by this setting):**
- :material-ip: **Every Non-Globally-Reachable Address** - anything outside the public Internet address space, in both families and in IPv4-mapped IPv6 form
- :material-ip: **Examples** - RFC 1918 (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), IPv6 unique local (fc00::/7), RFC 6598 shared address space (100.64.0.0/10), benchmarking (198.18.0.0/15) and documentation ranges
!!! warning "Security Warning"
**CRITICAL**: Only disable `SSRF_PROTECTION_BLOCK_PRIVATE_NETWORKS` in controlled environments where accessing internal networks is explicitly required and safe.
**If disabled, private network protection is removed:**
- :material-alert: Attackers may be able to reach any non-globally-reachable address (private networks, shared address space, and the other special-purpose ranges) through your API
- :material-shield-alert: Internal services on private networks (databases, admin panels, internal APIs) may be exposed
- :material-lock-open: Internal APIs without authentication may be exploited
**Important**: Even when disabled, baseline protection remains active and prevents access to:
- :material-check: Loopback addresses (127.0.0.1, localhost) - **always blocked**
- :material-check: Link-local addresses (169.254.x.x) including AWS EC2 metadata endpoint - **always blocked**
- :material-check: Reserved and multicast addresses - **always blocked**
!!! tip "When to Disable"
Disable `SSRF_PROTECTION_BLOCK_PRIVATE_NETWORKS` only when:
- :material-lan: Your application legitimately needs to access internal network resources
- :material-dev-to: Local development environment where accessing localhost services is required
- :material-shield-check: You have other security controls in place (network segmentation, firewall rules)
- :material-docker: Running in isolated Docker/container environments with restricted network access
!!! success "Defense in Depth"
Even with SSRF protection enabled, implement additional security measures:
- :material-network-strength-4: **Network Segmentation** - Isolate application servers from sensitive internal networks
- :material-firewall: **Firewall Rules** - Restrict outbound connections from application servers
- :material-security: **Security Groups** - Use AWS security groups to limit network access
- :material-monitor: **Monitoring** - Log and monitor outbound requests for suspicious patterns
---
## :material-speedometer-slow: Request Limits
Bound per-request resource usage to protect the server when the API is exposed to untrusted clients.
#### `MAX_INPUT_FILE_SIZE` { #max-input-file-size }
:octicons-package-24: **Purpose**
: Cap the size of an inline input file loaded into memory to protect against memory-exhaustion (DoS)
:octicons-database-24: **Type**
: Integer (bytes)
:octicons-gear-24: **Default**
: `0` (disabled — no limit)
:octicons-shield-check-24: **Best Practice**
: Set a limit aligned with your largest expected inline input (e.g. `26214400` for 25 MiB) when the API is exposed to untrusted clients
```bash
# Disabled (default) - no size limit
# No environment variable needed
# Reject inline inputs larger than 25 MiB
export MAX_INPUT_FILE_SIZE=26214400
```
!!! info "What is limited"
The limit applies to file content that is **loaded into memory** for model input:
- :material-file-code: Base64 and `data:` URI inputs
- :material-download: HTTP(S) and S3 sources downloaded and read for model input
- :material-database-arrow-up: [Attachments too large to travel inside a request](features.md#attachment-size), on the size their source declares, before they are staged
Requests exceeding the limit are rejected with **HTTP 413** before the content is fully decoded or downloaded. For downloads, the body is streamed and aborted as soon as the limit is exceeded, so a spoofed `Content-Length` cannot bypass it.
**Streaming uploads are not affected**, so large file transfers remain possible:
- :material-cloud-upload: Multipart form uploads
- :material-file-move: Files API ingest from HTTP(S) URLs and S3-to-S3 copies
#### `MAX_CONCURRENT_INPUT_DOWNLOADS` { #max-concurrent-input-downloads }
:octicons-package-24: **Purpose**
: Bound the number of input files fetched or resolved concurrently within a single request
:octicons-database-24: **Type**
: Integer (> 0)
:octicons-gear-24: **Default**
: `8`
:octicons-shield-check-24: **Best Practice**
: Keep a modest value so a single request with many remote inputs cannot exhaust sockets/memory or amplify outbound requests against a target
```bash
# Allow up to 4 concurrent input downloads per request
export MAX_CONCURRENT_INPUT_DOWNLOADS=4
```
!!! info "Behaviour"
Each remote input (image, document, or audio referenced by URL or S3 URI) is fetched in parallel, capped at this many at a time. Excess inputs queue and run as slots free up, so requests still complete — they are only paced. This prevents a request carrying thousands of URLs from opening thousands of simultaneous connections (socket/memory exhaustion and SSRF amplification).
---
## :material-radar: Observability (OpenTelemetry)
Configure distributed tracing for debugging and performance monitoring. stdapi.ai integrates with AWS X-Ray, Jaeger, DataDog, and other OTLP-compatible systems.
#### `OTEL_ENABLED` { #otel-enabled }
:octicons-package-24: **Purpose**
: Enable or disable OpenTelemetry tracing
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
```bash
export OTEL_ENABLED=true
```
!!! note "Performance Consideration"
Disable in performance-critical deployments where observability is not needed.
#### `OTEL_SERVICE_NAME` { #otel-service-name }
:octicons-package-24: **Purpose**
: Service identifier in trace visualizations
:octicons-gear-24: **Default**
: `stdapi.ai`
:octicons-check-circle-24: **Best Practice**
: Use descriptive names with environment information
```bash
export OTEL_SERVICE_NAME=stdapi-production-us-east-1
```
#### `OTEL_EXPORTER_ENDPOINT` { #otel-exporter-endpoint }
:octicons-package-24: **Purpose**
: OTLP HTTP endpoint URL for sending traces
:octicons-gear-24: **Default**
: `http://127.0.0.1:4318/v1/traces`
:octicons-plug-24: **Protocol**
: Must support OTLP HTTP format
**AWS X-Ray (via ADOT):**
```bash
export OTEL_EXPORTER_ENDPOINT=http://127.0.0.1:4318/v1/traces
```
**Jaeger:**
```bash
export OTEL_EXPORTER_ENDPOINT=http://jaeger:14268/api/traces
```
**Cloud Provider OTLP:**
```bash
# Use provider-specific OTLP endpoints
export OTEL_EXPORTER_ENDPOINT=https://your-provider-otlp-endpoint.com/v1/traces
```
#### `OTEL_SAMPLE_RATE` { #otel-sample-rate }
:octicons-package-24: **Purpose**
: Percentage of requests to trace (controls cost vs. observability)
:octicons-database-24: **Type**
: Float (0.0 to 1.0)
:octicons-gear-24: **Default**
: `1.0` (100%)
**Development:**
```bash
# Trace everything for debugging
export OTEL_SAMPLE_RATE=1.0
```
**Production (Moderate Traffic):**
```bash
# Sample 10% of requests
export OTEL_SAMPLE_RATE=0.1
```
**Production (High Traffic):**
```bash
# Sample 1% of requests
export OTEL_SAMPLE_RATE=0.01
```
!!! tip "Sampling Recommendations"
| Sample Rate | Use Case |
|-------------|----------|
| `1.0` (100%) | :material-bug: Development, debugging, low-traffic services |
| `0.1` (10%) | :material-chart-line: Production with moderate traffic |
| `0.01` (1%) | :material-rocket: High-traffic production services |
| `0.0` (0%) | :material-close: Equivalent to disabling tracing |
---
## :material-file-document: API Documentation Routes
stdapi.ai provides automatic API documentation routes, which are **disabled by default** for security in production environments.
!!! warning "Security Consideration"
Exposing API documentation routes in production can reveal internal API structure, available endpoints, and request/response schemas to potential attackers. Only enable these routes in development/testing environments or when absolutely necessary.
!!! info "Agent Discovery"
The machine-readable API catalog at `/.well-known/api-catalog` (RFC 9727 Linkset) is always served, regardless of the settings below. Enabling a route adds its entry to the catalog:
- `ENABLE_OPENAPI_JSON` — adds the `service-desc` link to `/openapi.json`
- `ENABLE_DOCS` or `ENABLE_REDOC` — adds the `service-doc` link to `/docs` or `/redoc` (Swagger UI takes precedence when both are enabled)
- [`ENABLE_MCP_STREAMABLE_HTTP`](#enable-mcp-streamable-http) or [`ENABLE_MCP_SSE`](#enable-mcp-sse) — adds the `mcp-server-card` link
The same links are also advertised as RFC 8288 `Link` headers on the root endpoint (`/`). That header is only emitted when at least one of these routes is enabled; with all of them disabled, the catalog is still reachable but carries no links.
#### `ENABLE_DOCS` { #enable-docs }
:octicons-package-24: **Purpose**
: Enable interactive Swagger UI documentation at `/docs`
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false` (disabled)
```bash
# Enable for development
export ENABLE_DOCS=true
```
!!! info "Interactive Documentation Features"
The `/docs` endpoint provides an interactive interface to:
- Browse all available API endpoints
- Test API requests directly from the browser
- View request/response schemas
- Understand parameter requirements
#### `ENABLE_REDOC` { #enable-redoc }
:octicons-package-24: **Purpose**
: Enable ReDoc documentation UI at `/redoc`
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false` (disabled)
```bash
# Enable for development
export ENABLE_REDOC=true
```
!!! info "ReDoc Features"
The `/redoc` endpoint provides a clean, responsive documentation interface with:
- Three-panel layout for easy navigation
- Enhanced schema visualization
- Better rendering for complex APIs
- Export to OpenAPI specification
!!! success "Works with no outbound access"
Both pages are served entirely by the gateway: the container image ships Swagger UI and ReDoc itself, pinned to an exact release and verified against a recorded SHA-256 during the build, alongside the icon and the schema. A browser that can reach the gateway renders them, with no request to any CDN, font host or other third party — so they work unchanged in an air-gapped VPC, behind an egress allow-list, or under a strict content security policy.
!!! tip "Static Documentation Available"
ReDoc API documentation is also available as static documentation at [API Reference](api_reference.md) without requiring this endpoint to be enabled.
#### `ENABLE_OPENAPI_JSON` { #enable-openapi-json }
:octicons-package-24: **Purpose**
: Enable OpenAPI schema JSON endpoint at `/openapi.json`
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false` (disabled)
```bash
# Enable for development
export ENABLE_OPENAPI_JSON=true
```
!!! info "OpenAPI Schema"
The `/openapi.json` endpoint provides the raw OpenAPI 3.0 specification, useful for:
- Generating API clients in various languages
- Import into API testing tools (Postman, Insomnia)
- API documentation generation
- Contract testing and validation
!!! note "Automatic Enablement"
If either `ENABLE_DOCS` or `ENABLE_REDOC` is set to `true`, the `/openapi.json` endpoint will be automatically enabled since both documentation UIs require the OpenAPI schema to function. You only need to explicitly set `ENABLE_OPENAPI_JSON=true` if you want to expose the schema endpoint without enabling the documentation UIs.
### Development Configuration
**Enable all documentation routes for local development:**
```bash
export ENABLE_DOCS=true
export ENABLE_REDOC=true
# ENABLE_OPENAPI_JSON is automatically enabled when ENABLE_DOCS or ENABLE_REDOC is true
```
**Or enable only Swagger UI:**
```bash
export ENABLE_DOCS=true
# ENABLE_OPENAPI_JSON is automatically enabled
```
**Or enable only ReDoc:**
```bash
export ENABLE_REDOC=true
# ENABLE_OPENAPI_JSON is automatically enabled
```
### Production Best Practice
```bash
# Keep all routes disabled in production (default)
# No environment variables needed - defaults to false
```
!!! danger "Production Warning"
**Never enable these routes in production** unless you have specific security controls in place (e.g., IP allowlisting, VPN-only access, or additional authentication layer).
---
## :material-chart-line: Validation and Logging
For comprehensive logging and monitoring information, see the [Logging and Monitoring](operations_logging_monitoring.md) guide.
#### `STRICT_INPUT_VALIDATION` { #strict-input-validation }
:octicons-package-24: **Purpose**
: Reject API requests containing unknown/extra fields instead of ignoring them
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
```bash
# Returns HTTP 400 for requests with unexpected fields
export STRICT_INPUT_VALIDATION=true
```
#### `CHAT_COMPLETIONS_REASONING_FIELD` { #chat-completions-reasoning-field }
:octicons-package-24: **Purpose**
: Choose which field carries a reasoning model's thinking text on `/v1/chat/completions`
:octicons-database-24: **Type**
: String
:octicons-gear-24: **Default**
: `reasoning_content`
:octicons-list-ordered-24: **Options**
: `reasoning_content`, `reasoning`, `none`
:octicons-workflow-24: **Behavior**
: The OpenAI Chat Completions API returns no thinking text of its own — it reports only a `reasoning_tokens` count — so the providers that do return it have settled on two different names. `reasoning_content` is the DeepSeek spelling, which most clients that read reasoning at all look for first. `reasoning` is the name used by OpenRouter and vLLM. `none` emits neither, keeping responses strictly OpenAI-shaped.
: The setting applies to both the completed message and the streamed deltas, so a client never sees one name while streaming and another at the end. Callers can also suppress reasoning per request with `include_reasoning: false` or `reasoning: {"exclude": true}`, whatever this is set to.
```bash
# Default: the name most clients read
export CHAT_COMPLETIONS_REASONING_FIELD=reasoning_content
# For clients written against OpenRouter or vLLM
export CHAT_COMPLETIONS_REASONING_FIELD=reasoning
# Strict OpenAI shape: never return thinking text
export CHAT_COMPLETIONS_REASONING_FIELD=none
```
#### `LOG_LEVEL` { #logging-level }
:octicons-package-24: **Purpose**
: Control the minimum severity of log events written to STDOUT
:octicons-gear-24: **Default**
: `info`
:octicons-list-ordered-24: **Options**
: `info`, `warning`, `error`, `critical`, `disabled`
:octicons-workflow-24: **Behavior**
: Only log events at or above the configured level are output. Log levels are ordered by severity: **info < warning < error < critical**
```bash
# Default: Output all log events
export LOG_LEVEL=info
# Production: Suppress info logs, show only warnings and higher
export LOG_LEVEL=warning
# Critical only: Show only critical errors
export LOG_LEVEL=critical
# Disable logging: Suppress all log output (not recommended)
export LOG_LEVEL=disabled
```
!!! info "Log Level Examples"
| Level | Outputs | Use Case |
|-------|---------|----------|
| `info` | info, warning, error, critical | :material-bug: Development, debugging, full visibility |
| `warning` | warning, error, critical | :material-check: Production (recommended for most deployments) |
| `error` | error, critical | :material-alert: High-traffic production, reduce log volume |
| `critical` | critical only | :material-alert-octagon: Minimal logging, only show fatal errors |
| `disabled` | none | :material-close: Not recommended - disables all logging |
!!! tip "Production Recommendation"
For production deployments, `warning` is recommended to reduce log volume while maintaining visibility into issues. The `info` level can generate significant log volume in high-traffic environments.
For detailed information about log events, structure, and monitoring strategies, see the [Logging and Monitoring](operations_logging_monitoring.md) guide.
#### `LOG_REQUEST_PARAMS` { #log-request-params }
:octicons-package-24: **Purpose**
: Include request and response parameters (JSON body, form, query) in logs for integration debugging
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
```bash
# Enable for debugging (NOT recommended for production)
export LOG_REQUEST_PARAMS=true
```
!!! danger "Security and Cost Warning"
Enabling `LOG_REQUEST_PARAMS` may expose sensitive data in logs. Use only in development/debugging environments.
Logging full request/response payloads can also significantly increase log ingestion and storage costs, especially for large LLM prompts, tool calls, and generated outputs. If you must enable it, prefer short log retention, targeted sampling, and temporary use only.
#### `LOG_CLIENT_IP` { #client-ip-logging }
:octicons-package-24: **Purpose**
: Enable logging of client IP addresses for each request and add IP to OpenTelemetry spans
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false` (disabled for privacy)
```bash
# Disabled (default) - no client IP logging
# No environment variable needed
# Enable client IP logging
export LOG_CLIENT_IP=true
```
!!! info "Client IP Behavior"
When enabled, client IP addresses are:
- Included in log output for each request
- Added as the `client.address` attribute to OpenTelemetry spans (when `OTEL_ENABLED=true`)
The IP address depends on your proxy configuration:
**With `ENABLE_PROXY_HEADERS=true` (behind reverse proxy):**
- Logs the real client IP address from the `X-Forwarded-For` header
- Shows the actual end-user IP, not the proxy IP
- Requires your reverse proxy (ALB, CloudFront, etc.) to set the header correctly
**With `ENABLE_PROXY_HEADERS=false` (default):**
- Logs the direct connection IP address
- Typically shows your reverse proxy or load balancer IP, not the end-user IP
- Limited usefulness unless application is directly exposed to clients
!!! tip "When to Enable"
Enable `LOG_CLIENT_IP` when:
- :material-shield-check: You need client IP addresses for security auditing or compliance
- :material-chart-line: Analyzing traffic patterns and geographic distribution
- :material-alert: Investigating abuse, fraud, or suspicious activity
- :material-bug: Debugging client-specific issues
**Important**: Also enable `ENABLE_PROXY_HEADERS=true` when behind AWS ALB, CloudFront, or other reverse proxies to log the real client IP instead of the proxy IP.
!!! warning "Privacy Consideration"
Client IP addresses are considered personal data under privacy regulations like GDPR. When logging IP addresses:
- :material-clock: Consider shorter log retention periods
- :material-file-document: Document the purpose in your privacy policy
- :material-shield-lock: Ensure logs are stored securely
- :material-delete: Implement log deletion procedures aligned with your data retention policy
!!! example "Configuration for AWS Deployments"
**Behind AWS ALB or CloudFront:**
```bash
# Enable proxy headers to get real client IPs
export ENABLE_PROXY_HEADERS=true
# Enable client IP logging
export LOG_CLIENT_IP=true
```
**Direct exposure (not recommended for production):**
```bash
# Only enable client IP logging
export LOG_CLIENT_IP=true
# ENABLE_PROXY_HEADERS remains false (default)
```
#### `TIMEZONE` { #timezone }
:octicons-package-24: **Purpose**
: IANA timezone identifier used for request date and time
:octicons-database-24: **Type**
: String (IANA timezone identifier)
:octicons-gear-24: **Default**
: `UTC`
```bash
# UTC (default)
export TIMEZONE=UTC
# North America
export TIMEZONE=America/New_York
# Europe
export TIMEZONE=Europe/London
```
---
## :material-chart-box-outline: CloudWatch Metrics and Cost Tracking
The behavior of these settings — EMF line structure, cost log format, pricing accuracy, regional price fallback, known limitations, and the price override format with examples — is documented in [CloudWatch Metrics (EMF)](operations_logging_monitoring.md#cloudwatch-metrics-emf) and [Cost Tracking](operations_cost_management.md#cost-tracking-real-time-aws-pricing) in the Logging and Monitoring guide.
#### `CLOUDWATCH_METRICS` { #cloudwatch-metrics }
:octicons-package-24: **Purpose**
: Emit per-request AWS-billed usage as CloudWatch Embedded Metric Format (EMF) log lines
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
```bash
export CLOUDWATCH_METRICS=true
```
#### `CLOUDWATCH_METRICS_NAMESPACE` { #cloudwatch-metrics-namespace }
:octicons-package-24: **Purpose**
: CloudWatch namespace under which the emitted usage metrics are grouped
:octicons-database-24: **Type**
: String
:octicons-gear-24: **Default**
: `stdapi`
:octicons-alert-24: **Requirement**
: 1-255 characters, alphanumeric plus `. - _ / # :`, must not start with the reserved `AWS/` prefix
```bash
export CLOUDWATCH_METRICS_NAMESPACE=my-app-metrics
```
#### `COST_TRACKING` { #cost-tracking }
:octicons-package-24: **Purpose**
: Enable real-time cost computation from live AWS pricing ([details and accuracy caveats](operations_cost_management.md#cost-tracking-real-time-aws-pricing)). Disabled by default: it requires the extra `pricing:GetProducts` IAM permission — see [Cost Tracking IAM Permissions](operations_iam_permissions.md#cost-tracking-iam).
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
```bash
export COST_TRACKING=true
```
#### `COST_PRICE_OVERRIDES` { #cost-price-overrides }
:octicons-package-24: **Purpose**
: Operator-supplied unit price overrides for models not covered by the AWS Price List API ([format and example](operations_cost_management.md#override-map-for-missing-models))
:octicons-database-24: **Type**
: JSON object — keys are model IDs, values are dicts mapping dimension name to price per one unit
:octicons-gear-24: **Default**
: `{}`
---
## :material-shield-check: Bedrock Guardrails
Amazon Bedrock Guardrails add content filtering and safety controls to model inputs and outputs. The configured guardrail also powers the [OpenAI-compatible Moderations API](api_openai_moderations.md) (`POST /v1/moderations`); without one, that API falls back to [inline guardrail checks](api_openai_moderations.md#model-support) in supported regions, then Amazon Comprehend.
!!! info "Configuration Options"
Guardrails can be configured in three ways:
1. :material-cog: **Global** - Via environment variables
2. :material-web: **Per-request** - Via HTTP headers
3. :material-code-json: **Request body** - Via `amazon-bedrock-guardrailConfig` object
### Route Coverage
The configured guardrail applies to every route that serves a request directly. The [Batch API](api_openai_batches.md) is the exception: Amazon Bedrock batch inference cannot apply a guardrail, so a batch a configured guardrail would cover is refused rather than run unchecked. Chat routes use the native Bedrock integration; routes whose AWS backend has no guardrail mechanism enforce it through the [ApplyGuardrail API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ApplyGuardrail.html): client-supplied text is checked as `INPUT` before the backend call and generated text as `OUTPUT` after it. On the Realtime API, where content streams both ways for as long as the session is open, the check runs per turn and an intervention ends the session — see [Guardrail coverage](api_openai_realtime.md#guardrail-coverage) for what that does and does not catch.
| Routes | Mechanism | Checked content |
|------------------------------------------------------------|--------------------------------------|--------------------------------------------------------|
| Chat Completions, Responses, Completions, Anthropic Messages | :material-link: Native (Converse `guardrailConfig` / InvokeModel) | Model input and output |
| Moderations | :material-shield-search: ApplyGuardrail (classification) | Submitted text and images |
| Embeddings (OpenAI and Cohere v1/v2) | :material-shield-check: ApplyGuardrail | `INPUT` — each text input |
| Rerank (Cohere v1/v2) | :material-shield-check: ApplyGuardrail | `INPUT` — query and each document text |
| Images Generations / Edits | :material-shield-check: ApplyGuardrail | `INPUT` — prompt (Variations has no text to check) |
| Videos | :material-shield-check: ApplyGuardrail | `INPUT` — prompt |
| Audio Speech | :material-shield-check: ApplyGuardrail | `INPUT` — text to synthesize |
| Audio Transcriptions (including streaming) | :material-shield-check: ApplyGuardrail | `OUTPUT` — transcript |
| Audio Translations | :material-shield-check: ApplyGuardrail | `OUTPUT` — translated text |
| [Realtime](api_openai_realtime.md#guardrail-coverage) | :material-shield-check: ApplyGuardrail | `INPUT` — each written item before it reaches the model, and each transcribed caller turn; `OUTPUT` — each completed answer |
!!! warning "Cost Tracking"
AWS bills the guardrail on **every** route it applies to, but only the ApplyGuardrail-enforced ones report the units consumed. The mechanism a route uses therefore decides whether its guardrail cost is visible.
| Mechanism | Guardrail cost in [usage logs](operations_logging_monitoring.md) |
|-----------|-----------------------------------------------------------------|
| :material-shield-check: ApplyGuardrail | :material-check-circle:{ .success role="img" aria-label="Tracked" } **Tracked** — the response returns the units each policy consumed |
| :material-link: Native (Converse / InvokeModel) | :material-close-circle:{ .unsupported role="img" aria-label="Not tracked" } **Not tracked** — the response reports no guardrail units |
On ApplyGuardrail routes, the units AWS reports appear as `text_units` and `input_images` under one `amazon.bedrock-runtime-guardrail-*` model per applied policy, each priced at that policy's own rate; see [Moderations billing](api_openai_moderations.md#billing). A route that checks both `INPUT` and `OUTPUT` calls the API twice, so it records two sets of units for one request.
On native routes the guardrail still runs and AWS still charges for it, but the Converse and InvokeModel responses carry no unit counts for the gateway to record. **Reported costs on these routes are lower than the AWS bill by the guardrail's share.** Deriving the units from text length instead would be a guess, not a measurement, so none is made.
!!! info "Intervention Behavior"
On ApplyGuardrail-enforced routes, a blocking intervention fails the request with HTTP 400 and error code `content_filter` (the same code chat routes report as their finish reason), carrying the guardrail's configured blocked messaging. A masking-only intervention (sensitive-information anonymization) substitutes the masked text — input masking reaches the backend model, and a masked transcript or translation is returned on the plain `json`/`text` formats. Response formats that cannot carry masked text (`srt`, `vtt`, `verbose_json`, `diarized_json`) fail with the same `content_filter` error instead of leaking the unmasked content.
### Global Configuration
#### `AWS_BEDROCK_GUARDRAIL_IDENTIFIER` { #aws-bedrock-guardrail-identifier }
:octicons-package-24: **Purpose**
: ID of the Bedrock Guardrail to apply
:octicons-alert-24: **Required**
: Yes (together with `AWS_BEDROCK_GUARDRAIL_VERSION`)
```bash
export AWS_BEDROCK_GUARDRAIL_IDENTIFIER=abc123def456
```
#### `AWS_BEDROCK_GUARDRAIL_VERSION` { #aws-bedrock-guardrail-version }
:octicons-package-24: **Purpose**
: Version of the Bedrock Guardrail
:octicons-alert-24: **Required**
: Yes (together with `AWS_BEDROCK_GUARDRAIL_IDENTIFIER`)
```bash
export AWS_BEDROCK_GUARDRAIL_VERSION=1
```
#### `AWS_BEDROCK_GUARDRAIL_TRACE` { #aws-bedrock-guardrail-trace }
:octicons-package-24: **Purpose**
: Trace level for guardrail evaluation
:octicons-gear-24: **Options**
: `disabled`, `enabled`, `enabled_full`
:octicons-gear-24: **Default**
: None (optional)
```bash
export AWS_BEDROCK_GUARDRAIL_TRACE=enabled
```
#### `AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE` { #aws-bedrock-allow-guardrail-override }
:octicons-package-24: **Purpose**
: Control whether users can override the global guardrail configuration at request level via HTTP headers
:octicons-gear-24: **Default**
: `false` (disabled for security)
:octicons-shield-24: **Security Consideration**
: When set to `false` (default) and a global guardrail is configured, only the global configuration is enforced, preventing users from bypassing or modifying safety controls. Set to `true` if you need to allow per-request guardrail customization to override the global configuration.
:octicons-info-24: **Auto-Enable Behavior**
: If no guardrail is configured at all — both `AWS_BEDROCK_GUARDRAIL_IDENTIFIER` and `AWS_BEDROCK_GUARDRAIL_VERSION` unset, and no [model alias](#model-aliases-configuration) carrying one — this setting is automatically set to `true` at startup, allowing per-request guardrails when no policy is enforced.
```bash
export AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE=true
```
!!! tip "Per-Alias Guardrails"
A [model alias](#model-aliases-configuration) can carry its own guardrail, applied to the requests naming it and overriding the global one. That is how a single deployment publishes the same model under a strictly guarded name and an unguarded one.
!!! example "Complete Guardrail Configuration"
```bash
export AWS_BEDROCK_GUARDRAIL_IDENTIFIER=abc123def456
export AWS_BEDROCK_GUARDRAIL_VERSION=1
export AWS_BEDROCK_GUARDRAIL_TRACE=enabled
export AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE=false # Default: prevent overrides
```
### Per-Request Guardrail Configuration
!!! info "Header Usage Behavior"
Request headers can be used when [`AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE`](#aws-bedrock-allow-guardrail-override) is `true`:
- **No global guardrail configured**: Setting is automatically `true` at startup, enabling per-request guardrails
- **Global guardrail configured**: Setting defaults to `false` for security; set to `true` to allow overrides
This prevents users from bypassing configured safety controls while still allowing flexibility when no global policy exists.
Use HTTP headers to specify guardrail settings per request:
| Header | Purpose | Valid Values |
|-------------------------------------------------|-----------------------------------------------------------------------|----------------------------------------|
| `X-Amzn-Bedrock-GuardrailIdentifier` | Guardrail ID | Your guardrail identifier |
| `X-Amzn-Bedrock-GuardrailVersion` | Guardrail version | Version number (e.g., `1`) |
| `X-Amzn-Bedrock-Trace` | Trace level | `disabled`, `enabled`, `enabled_full` |
| `X-Amzn-Bedrock-GuardrailStreamProcessingMode` | Guardrail assessment timing for streaming requests (stripped from non-streaming requests) | `sync`, `async` |
```bash title="Example cURL Request"
curl -X POST https://api.example.com/v1/chat/completions \
-H "Authorization: Bearer sk-..." \
-H "X-Amzn-Bedrock-GuardrailIdentifier: abc123def456" \
-H "X-Amzn-Bedrock-GuardrailVersion: 1" \
-H "X-Amzn-Bedrock-Trace: enabled" \
-d '{"model": "anthropic.claude-sonnet-5", "messages": [...]}'
```
### Request Body Configuration
The `amazon-bedrock-guardrailConfig` object in the request body is supported for OpenAI Chat Completions compatibility.
!!! warning "Compatibility Note"
Only fields compatible with Bedrock Converse API are honored. The `tagSuffix` field is documented in AWS but **not supported** in this implementation.
---
## :material-database-lock: Bedrock Session Storage { #bedrock-session-storage-optional }
Requests with `store=true` on the [Responses](api_openai_responses.md#stored-responses) and [Chat Completions](api_openai_chat_completions.md#stored-chat-completions) APIs persist generations in Amazon Bedrock sessions. No environment variable is needed to enable this — it requires the [Bedrock Session Storage IAM permissions](operations_iam_permissions.md#bedrock-session-storage-optional).
!!! warning "Not available in every region"
Amazon Bedrock session storage covers fewer regions than model inference. When the primary Bedrock region — the first entry of [`AWS_BEDROCK_REGIONS`](#aws-bedrock-regions), which is where all sessions are created — does not provide it, `store=true` is **ignored**: the generation is still returned, and a warning is recorded in the request log stating that the session storage endpoint was unreachable or timed out and that session storage is offered in fewer regions than model inference. Retrieving a stored object then returns `404`. A missing `bedrock:CreateSession` permission produces a distinct `AccessDenied` warning pointing at the IAM permissions instead.
Nothing fails and no request is lost, but stored responses and stored chat completions are simply unavailable. To rely on them, make the primary Bedrock region one that offers session storage — check the [Amazon Bedrock session management endpoints](https://docs.aws.amazon.com/general/latest/gr/bedrock.html) for current coverage.
#### `AWS_BEDROCK_SESSION_ENCRYPTION_KEY_ARN` { #aws-bedrock-session-encryption-key-arn }
:octicons-package-24: **Purpose**
: KMS key ARN encrypting the Amazon Bedrock sessions that back [stored responses](api_openai_responses.md#stored-responses) and [stored chat completions](api_openai_chat_completions.md#stored-chat-completions) (`store=true`)
:octicons-gear-24: **Default**
: None — sessions are encrypted with the AWS-managed key
:octicons-check-circle-24: **Validation**
: Checked at startup: must be a KMS key ARN (`arn::kms:::key/`).
```bash
export AWS_BEDROCK_SESSION_ENCRYPTION_KEY_ARN=arn:aws:kms:us-east-1:123456789012:key/abcd-...
```
!!! warning "Shared Visibility Across Deployments"
Stored responses and chat completions are namespaced by AWS account and region, not by stdapi.ai deployment. Multiple deployments sharing the same account and region can list, retrieve, and delete each other's stored objects. Use a dedicated AWS account per deployment when isolation matters, or accept this shared visibility as a deliberate trade-off.
!!! info "Orphaned Session Cleanup"
A session is created independently of the generation it will hold — before it for the Responses API, concurrently with it for Chat Completions — so a crash before the generation is written leaves an empty, orphaned session. Bedrock sessions have no TTL and persist until deleted, so periodically clean up stale sessions (`aws bedrock-agent-runtime list-sessions` plus `delete-session`, or an operator-managed lifecycle policy).
#### `AWS_BEDROCK_BATCH_ROLE_ARN` { #aws-bedrock-batch-role-arn }
:octicons-package-24: **Purpose**
: AWS IAM service role that [Amazon Bedrock batch inference](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html) assumes to read a batch's requests and write its results — required to enable the [Batch API](api_openai_batches.md) and the [Message Batches API](api_anthropic_batches.md)
:octicons-database-24: **Type**
: String — an IAM role ARN
:octicons-gear-24: **Default**
: None — the batch endpoints answer `503` (`529` on the Anthropic-compatible routes) and the server reports the disabled feature in its startup log
:octicons-workflow-24: **Behavior**
: The server passes this role when it submits a batch; Amazon Bedrock then reads the requests and writes the results with it. The role must be able to read and write every bucket configured with [`AWS_S3_BUCKET`](#aws-s3-bucket) and [`AWS_S3_REGIONAL_BUCKETS`](#aws-s3-regional-buckets), under [`AWS_S3_BATCHES_PREFIX`](#aws-s3-batches-prefix).
:octicons-check-circle-24: **Validation**
: Checked at startup: must be an IAM role ARN (`arn::iam:::role/`). Leaving it unset is reported as a startup warning, never a failure.
```bash
export AWS_BEDROCK_BATCH_ROLE_ARN=arn:aws:iam::123456789012:role/stdapi-ai-batch
```
!!! warning "The role and two IAM policies come first"
The role's trust policy must allow `bedrock.amazonaws.com` to assume it, and the server's own role needs `iam:PassRole` on this ARN. See [IAM Permissions](operations_iam_permissions.md#batch-inference) for copyable policies.
#### `AWS_BEDROCK_USER_ROLE_ARN` { #aws-bedrock-user-role-arn }
:octicons-package-24: **Purpose**
: Run each end user's model calls under an AWS IAM role session of their own, so AWS reports Amazon Bedrock model usage [per end user](operations_cost_management.md#per-user-attribution) in Cost Explorer and in the Cost and Usage Report
:octicons-database-24: **Type**
: String — an IAM role ARN
:octicons-gear-24: **Default**
: None — every request runs under the server's own identity, and AWS reports all model usage under it
:octicons-workflow-24: **Behavior**
: The server opens one short-lived session of this role per end user, caches it, and signs that user's model invocations with it. The identity is the authenticated caller when [authentication](operations_authentication_security.md) is enabled, otherwise the identifier the request declares (`safety_identifier` or `user` on the OpenAI-compatible APIs, `metadata.user_id` on the Anthropic Messages API). Only model invocations are covered — guardrail evaluations, video generation, speech, transcription and translation keep the server's identity. A session that cannot be opened fails the request rather than falling back to the server's identity.
:octicons-check-circle-24: **Validation**
: Checked at startup: must be an IAM role ARN (`arn::iam:::role/`). The server also tries to assume it at startup and reports a warning — not a failure — when it cannot.
```bash
export AWS_BEDROCK_USER_ROLE_ARN=arn:aws:iam::123456789012:role/stdapi-ai-end-user
```
!!! warning "The role and two IAM policies come first"
The role's trust policy must allow this server's own role to call both `sts:AssumeRole` **and** `sts:TagSession` on it, and the server's role needs the same two actions on this role ARN. See [IAM Permissions](operations_iam_permissions.md#per-user-cost-attribution) for copyable policies, including the model ARNs a cross-region inference profile requires.
#### `AWS_BEDROCK_USER_ROLE_SESSION_DURATION` { #aws-bedrock-user-role-session-duration }
:octicons-package-24: **Purpose**
: Lifetime of a per-end-user role session, in seconds
:octicons-database-24: **Type**
: Integer — 900 to 3600
:octicons-gear-24: **Default**
: `3600`
:octicons-workflow-24: **Behavior**
: Sessions are cached per end user and reopened shortly before they expire, so a longer lifetime means fewer AWS STS calls. The upper bound is imposed by AWS: the server itself runs under an assumed role, and a role session obtained from another role session cannot last longer than one hour, whatever the role's maximum session duration.
```bash
export AWS_BEDROCK_USER_ROLE_SESSION_DURATION=1800
```
#### `AWS_BEDROCK_USER_ROLE_TAG_KEY` { #aws-bedrock-user-role-tag-key }
:octicons-package-24: **Purpose**
: Session tag key carrying the end user identity on each per-end-user role session
:octicons-database-24: **Type**
: String, or null to send no session tag
:octicons-gear-24: **Default**
: `user`
:octicons-workflow-24: **Behavior**
: Activate this key as a cost allocation tag — in the AWS Billing console, under **Cost allocation tags** filtered by type **IAM principal** — to group Bedrock costs by end user in Cost Explorer. The same tag is testable in IAM policies as `aws:PrincipalTag/`, so the role can be restricted per user. With no tag, end users are still distinguished by their role session name in the Cost and Usage Report.
:octicons-check-circle-24: **Validation**
: Checked at startup: 1 to 128 characters over letters, digits, spaces and `_ . : / = + - @`; keys beginning with `aws:` are reserved by AWS and rejected.
```bash
export AWS_BEDROCK_USER_ROLE_TAG_KEY=end-user
```
#### `AWS_BEDROCK_USER_ROLE_REQUIRE_IDENTITY` { #aws-bedrock-user-role-require-identity }
:octicons-package-24: **Purpose**
: Reject a model request that identifies no end user, instead of running it under the server's own identity
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false` — such requests run under the server's identity, and their usage is reported under it
:octicons-workflow-24: **Behavior**
: Enable it so that no model usage escapes per-user attribution: a request carrying neither an authenticated caller nor an end user identifier is answered `400`. Clients that never send one stop working, so enable it only once every client identifies its user — and note that some APIs, audio transcription among them, have no end user field at all, so on those it takes an authenticated caller. A real-time speech-to-speech session keeps for its whole life the identity it opened with, so while this is enabled it is refused rather than attributed to the server. Requires [`AWS_BEDROCK_USER_ROLE_ARN`](#aws-bedrock-user-role-arn).
```bash
export AWS_BEDROCK_USER_ROLE_REQUIRE_IDENTITY=true
```
---
## :material-speedometer: Bedrock Service Tier and Performance Configuration
Amazon Bedrock service tiers and performance configurations allow you to optimize AI workload performance and cost trade-offs. Configure latency optimization and throughput priority for your inference requests.
!!! info "AWS Documentation"
For detailed information about service tiers, see:
- [Amazon Bedrock Service Tiers](https://aws.amazon.com/blogs/aws/new-amazon-bedrock-service-tiers-help-you-match-ai-workload-performance-with-cost/)
### Service Tiers
Service tiers help you match AI workload performance with cost by selecting the appropriate throughput and latency characteristics:
- **`priority`** - Highest priority processing with guaranteed capacity and fastest response times. Best for latency-sensitive applications.
- **`default`** - Standard processing with balanced performance and cost. Suitable for most production workloads.
- **`flex`** - Cost-optimized processing with flexible scheduling. Best for batch jobs and non-time-sensitive workloads.
### Performance Configuration
Performance configuration allows you to optimize for latency:
- **`standard`** - Standard latency profile with balanced performance
- **`optimized`** - Optimized for lowest possible latency
### Per-Request Service Tier Configuration { #service-tier-per-request }
Configure service tier and performance settings per request using HTTP headers. These headers are available on all Bedrock-based routes (Chat Completions, Embeddings, Images). Server-side per-model defaults can be set with [`DEFAULT_MODEL_SERVICE_TIERS`](#default-model-service-tiers).
| Header | Purpose | Valid Values |
|--------------------------------------------|------------------------|-------------------------------|
| `X-Amzn-Bedrock-Service-Tier` | Service tier selection | `priority`, `default`, `flex` |
| `X-Amzn-Bedrock-PerformanceConfig-Latency` | Latency optimization | `standard`, `optimized` |
!!! warning "The tier header is subject to the override gate"
When [`AWS_BEDROCK_ALLOW_SERVICE_TIER_OVERRIDE`](#aws-bedrock-allow-service-tier-override) is `false`, `X-Amzn-Bedrock-Service-Tier` — like the `service_tier` request parameter — is ignored for any model that has a tier configured, by `DEFAULT_MODEL_SERVICE_TIERS` or by the [alias](#model-aliases-configuration) the request names. A model with no configured tier honors the header in either case. The response's `service_tier` field keeps echoing the request's own value; [usage and cost reporting](operations_cost_management.md) record the tier that actually served the call.
Configured tiers, the header and this gate all apply to models served through the Bedrock Converse and InvokeModel APIs. On a [Bedrock Mantle](#summary-bedrock-mantle)-served model, the request's own `service_tier` parameter is what applies — the header is not read, no configured tier is added, and the response reports the tier that model returns.
```bash title="Example: Chat Completions with Priority Tier and Optimized Latency"
curl -X POST https://api.example.com/v1/chat/completions \
-H "Authorization: Bearer sk-..." \
-H "X-Amzn-Bedrock-Service-Tier: priority" \
-H "X-Amzn-Bedrock-PerformanceConfig-Latency: optimized" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-sonnet-5",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
```bash title="Example: Embeddings with Flex Tier for Batch Processing"
curl -X POST https://api.example.com/v1/embeddings \
-H "Authorization: Bearer sk-..." \
-H "X-Amzn-Bedrock-Service-Tier: flex" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-2-multimodal-embeddings-v1:0",
"input": ["text 1", "text 2", "text 3"]
}'
```
```bash title="Example: Image Generation with Default Tier"
curl -X POST https://api.example.com/v1/images/generations \
-H "Authorization: Bearer sk-..." \
-H "X-Amzn-Bedrock-Service-Tier: default" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-canvas-v1:0",
"prompt": "A serene mountain landscape"
}'
```
!!! tip "When to Use Each Tier"
**Priority Tier:**
- Real-time customer-facing applications
- Interactive chatbots and assistants
- Applications requiring guaranteed low latency
- Production workloads with strict SLAs
**Default Tier:**
- Standard production workloads
- General-purpose API usage
- Applications with moderate latency requirements
**Flex Tier:**
- Batch processing and bulk operations
- Offline content generation
- Data processing pipelines
- Non-time-sensitive workloads
- Cost-optimized inference at scale
---
## :material-account-voice: Audio and Text-to-Speech
#### `DEFAULT_TTS_MODEL` { #default-tts-model }
:octicons-package-24: **Purpose**
: Default text-to-speech model when not specified in requests
:octicons-gear-24: **Default**
: `amazon.polly-standard`
| Model | Description | Quality |
|-------|-------------|---------|
| `amazon.polly-standard` | Standard Polly voices | :material-star: Classic quality |
| `amazon.polly-neural` | Neural Polly voices | :material-star-circle: Higher quality, more natural |
| `amazon.polly-long-form` | Long-form content | :material-text-long: Optimized for long content |
| `amazon.polly-generative` | Generative AI voices | :material-sparkles: Latest technology |
```bash
export DEFAULT_TTS_MODEL=amazon.polly-neural
```
#### `DEFAULT_TTS_LANGUAGE` { #default-tts-language }
:octicons-package-24: **Purpose**
: Default language code for text-to-speech synthesis when using OpenAI voice names
:octicons-gear-24: **Default**
: None (automatic language detection via Amazon Comprehend)
:octicons-check-circle-24: **Behavior**
: When specified, this language is used instead of automatic detection. When not set, Amazon Comprehend detects the language automatically from the input text.
**Valid Language Codes**: Any Amazon Polly language code (e.g., `en-US`, `fr-FR`, `es-ES`, `de-DE`, `ja-JP`)
```bash
# Use English (US) for all TTS requests
export DEFAULT_TTS_LANGUAGE=en-US
# Use French for all TTS requests
export DEFAULT_TTS_LANGUAGE=fr-FR
```
!!! tip "Performance Benefits"
Setting a default language improves performance by:
- **Faster responses**: Skips language detection API call to Amazon Comprehend
- **Reduced costs**: No Amazon Comprehend charges for language detection
- **Predictable voice selection**: Always uses voices from the specified language
!!! info "When to Use"
Consider setting a default language when:
- Your application primarily serves content in a single language
- You want to optimize response times and reduce AWS service calls
- You prefer predictable voice selection over automatic language matching
!!! note "Interaction with Voice Selection"
This setting only affects automatic language detection when using OpenAI voice names (like `alloy`, `echo`, `nova`). If you specify a Polly voice ID directly (like `Joanna`, `Matthew`), language detection is already skipped.
---
## :material-headset: Realtime API
#### `REALTIME_CLIENT_SECRET_KEY` { #realtime-client-secret-key }
:octicons-package-24: **Purpose**
: Secret the [Realtime API](api_openai_realtime.md#ephemeral-client-secrets)'s ephemeral client secrets (`POST /v1/realtime/client_secrets`) are signed with
:octicons-database-24: **Type**
: String (any value)
:octicons-gear-24: **Default**
: None — a signing key is derived from the configured API key instead
:octicons-workflow-24: **Behavior**
: Ephemeral client secrets are stateless: nothing is stored server-side, so any instance behind a load balancer verifies a secret minted by any other, as long as they all sign with the same key. By default that key is derived from the deployment's own API key, which is already shared across every instance — so this setting has nothing to add on a deployment that already configures one.
Set it explicitly on a deployment that runs with **no API key configured at all**: without either one, each instance falls back to a random key generated **per process**, and a client secret minted by one instance then fails to verify on any other — the symptom is an ephemeral secret rejected intermittently on a multi-instance deployment. Any value works, as long as every instance shares it.
```bash
export REALTIME_CLIENT_SECRET_KEY=a-value-shared-by-every-instance
```
!!! warning "Changing it invalidates outstanding secrets"
A secret minted with one key does not verify against another. Rotating this setting (or the API key it would otherwise derive from) invalidates every client secret minted before the change — they simply stop working once their bearer tries to open a session, the same as if they had expired.
#### `REALTIME_ALLOW_SESSION_OVERRIDE` { #realtime-allow-session-override }
:octicons-package-24: **Purpose**
: Whether a client connecting with an [ephemeral client secret](api_openai_realtime.md#ephemeral-client-secrets) may override the session configuration that secret carries
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `true` — the upstream behavior: the carried configuration is a default the client may change
:octicons-workflow-24: **Behavior**
: A minted secret carries a session configuration, and by default a client opening a session with it may name another model on the `?model=` query string and replace any of that configuration with its own `session.update` — exactly as it can against the upstream API.
Set to `false` on a multi-tenant deployment, where the secret is the only thing constraining an untrusted browser or mobile client. The `model`, the `instructions` and `max_output_tokens` the secret was minted with are then final: connecting with a `?model=` naming a different model is refused before the session opens, and a `session.update` changing any of the three answers an `error` event. Everything else — voice, audio formats, turn detection, transcription — stays under the client's control.
```bash
export REALTIME_ALLOW_SESSION_OVERRIDE=false
```
---
## :material-archive: Deprecated Settings
!!! warning "Deprecated and Ignored"
`TOKENS_ESTIMATION` (default: `false`) and `TOKENS_ESTIMATION_DEFAULT_ENCODING` (default: `None`) are deprecated and ignored: tiktoken-based token estimation has been removed from the project. Token counts are now sourced directly from AWS billing data when available. Remove these variables from existing configurations.
---
## :material-cached: Model Cache
stdapi.ai automatically discovers and caches available Bedrock models from configured regions. The cache is refreshed on-demand when expired, not via background tasks.
#### `MODEL_CACHE_SECONDS` { #model-cache-seconds }
:octicons-package-24: **Purpose**
: Cache lifetime for the Bedrock models list before refresh
:octicons-database-24: **Type**
: Integer (seconds)
:octicons-gear-24: **Default**
: `900` (15 minutes)
:octicons-workflow-24: **Behavior**
: When a request needs the model list (e.g., model lookup, `/models` endpoint) and the cache has expired, the server queries Amazon Bedrock to discover newly available models, check for model access changes, and update inference profile configurations. This cache also applies to application inference profile and prompt router information when users pass ARNs directly (if enabled via [`AWS_BEDROCK_ALLOW_APPLICATION_INFERENCE_PROFILE_ARN`](#bedrock-allow-application-profile-arn) or [`AWS_BEDROCK_ALLOW_PROMPT_ROUTER_ARN`](#bedrock-allow-prompt-router-arn))
```bash
# Default: 15 minutes
export MODEL_CACHE_SECONDS=900
# More frequent updates (5 minutes)
export MODEL_CACHE_SECONDS=300
# Less frequent updates (1 hour)
export MODEL_CACHE_SECONDS=3600
```
!!! info "Lazy Refresh Behavior"
The model cache uses **lazy (on-demand) refresh**, not background tasks:
- Cache is refreshed only when a request needs it **and** the cache has expired
- Common triggers: model lookup failures, `/v1/models` API calls, inference requests with unknown models
- The **first request after expiration** experiences additional latency (typically 2-5 seconds) while the cache refreshes; the AWS calls (`ListFoundationModels`, `GetFoundationModelAvailability`, `ListInferenceProfiles`) run in parallel across regions, so the penalty scales with the slowest region rather than the number of regions
- Subsequent requests use the fresh cache until it expires again
!!! tip "Tuning Recommendations"
| Interval | Use Case | Trade-offs |
|----------|----------|------------|
| `300` (5 min) | :material-rocket: Development, testing new models | More frequent refresh latency, faster model discovery |
| `900` (15 min) | :material-check: Production (default, balanced) | Balanced refresh frequency and latency impact |
| `3600` (1 hour) | :material-cash: Stable production, cost optimization | Rare refresh latency, slower model discovery |
Lower cache lifetimes increase the frequency of the per-region discovery calls; very frequent refreshes in high-traffic deployments may approach API rate limits.
---
## :material-timer-sand: AI Response Timeout { #ai-response-timeout-section }
#### `AI_RESPONSE_TIMEOUT` { #ai-response-timeout }
:octicons-package-24: **Purpose**
: Maximum time in seconds to wait without receiving any data from an AI model
:octicons-database-24: **Type**
: Integer (seconds, must be greater than 0)
:octicons-gear-24: **Default**
: `600` (10 minutes)
:octicons-workflow-24: **Behavior**
: Inactivity (per-read) timeout on the upstream model connection, applied to both streaming and non-streaming requests. The timer resets every time data is received, so it fires only when the model stalls for longer than this value — it does **not** bound the total duration of a response: a stream that keeps producing chunks can run well past it. On a non-streaming request, where the whole response arrives at once, it effectively bounds the wait for that single response. When it fires, the connection is closed and the request fails with a timeout error
```bash
# Default (10 minutes) - suitable for extended thinking models
export AI_RESPONSE_TIMEOUT=600
# Shorter timeout for standard models (2 minutes)
export AI_RESPONSE_TIMEOUT=120
# Longer timeout for very long documents or high reasoning budgets (15 minutes)
export AI_RESPONSE_TIMEOUT=900
```
!!! tip "When to Adjust"
- **Increase** if you see timeout errors with models that use extended thinking/reasoning, large document analysis, or high token budgets
- **Decrease** to fail fast and free resources if your workload only uses standard models where long waits indicate a problem
!!! info "Extended Thinking Models"
Models with extended reasoning capabilities (such as Claude with `thinking` enabled or high `reasoning_effort`) may spend significant time generating internal reasoning steps before producing output. The default of 600 seconds accommodates these use cases. Standard models without extended thinking typically respond within 60 seconds.
---
## :material-power-plug-off: Shutdown Drain { #shutdown-drain-section }
#### `SHUTDOWN_DRAIN_TIMEOUT` { #shutdown-drain-timeout }
:octicons-package-24: **Purpose**
: Maximum time in seconds the server waits for background work to finish after it has been asked to stop
:octicons-database-24: **Type**
: Number (seconds, `0` or greater)
:octicons-gear-24: **Default**
: `10`
:octicons-workflow-24: **Behavior**
: Some work is deliberately started outside the request that asked for it, so the caller is answered without waiting for it: temporary file cleanups, vector store file indexing, and the release of live audio sessions. On a stop signal the server waits up to this long for that work to finish, then cancels whatever is still running. The wait is a single deadline shared by all of it, not a budget per item, and a server with nothing outstanding stops immediately
```bash
# Default: comfortably inside a 30-second container stop timeout
export SHUTDOWN_DRAIN_TIMEOUT=10
# Longer wait, with the container stop timeout raised to match
export SHUTDOWN_DRAIN_TIMEOUT=20
# No wait: cancel background work immediately and stop as fast as possible
export SHUTDOWN_DRAIN_TIMEOUT=0
```
!!! warning "Best effort, not a delivery guarantee"
A container runtime sends `SIGKILL` a fixed delay after the stop signal — 30 seconds by default on [Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_timeout) — so the server can be killed before the wait ends, and a deployment may run under an orchestrator that stops it sooner still. Keep this value comfortably below your container stop timeout, and raise it only together with that timeout. Never rely on this wait for anything whose completion matters: retry the operation instead.
!!! tip "When Work Is Lost"
Anything cancelled at the deadline is counted in the server's `stop` log event, which is then emitted at `warning` level with one count per kind of work. Deployments that see those counts regularly are stopping the server faster than its work can finish: raise this value and the container stop timeout together, or reduce what each request defers.
---
## :material-tune: Default Model Parameters
Configure default inference parameters applied automatically to specific models.
!!! success "What You Can Do"
- :material-thermometer: Set consistent temperature/creativity levels per model
- :material-flask: Enable provider-specific features (e.g., Anthropic beta features)
- :material-cash: Configure default token limits for cost control
- :material-stop: Apply model-specific stop sequences
!!! info "Parameter Precedence"
Request parameters **always take precedence** over defaults.
#### `DEFAULT_MODEL_PARAMS` { #default-model-params }
:octicons-package-24: **Purpose**
: Per-model default parameters
:octicons-code-24: **Format**
: JSON object with model IDs as keys
**Supported Parameters:**
| Parameter | Type | Range | Description |
|-----------|------|-------|-------------|
| `temperature` | Float | ≥ 0 | Sampling temperature |
| `top_p` | Float | ≥ 0 | Nucleus sampling |
| `max_tokens` | Integer | ≥ 1 | Maximum response tokens |
| `stop_sequences` | String/Array | - | Stop generation tokens |
| Provider-specific | Various | - | e.g., `anthropic_beta` |
Only the outer JSON shape (an object of per-model objects) is validated at startup. The parameter values above are validated lazily, the first time a model with configured defaults is used: a wrong type, or a value below the lower bounds shown in the table, fails that request with HTTP `400`. The numeric ceilings (for example the usual `top_p` maximum of `1.0`) are enforced by Amazon Bedrock and the target model.
### Configuration Examples { #default-model-params-examples }
**Basic Parameters:**
```bash
export DEFAULT_MODEL_PARAMS='{
"amazon.nova-micro-v1:0": {
"temperature": 0.3,
"max_tokens": 800
}
}'
```
**Provider-Specific Features:**
```bash
export DEFAULT_MODEL_PARAMS='{
"anthropic.claude-sonnet-5": {
"anthropic_beta": ["Interleaved-thinking-2025-05-14"]
}
}'
```
**Multiple Models:**
```bash
export DEFAULT_MODEL_PARAMS='{
"amazon.nova-micro-v1:0": {
"temperature": 0.3,
"max_tokens": 500
},
"amazon.nova-lite-v1:0": {
"temperature": 0.7,
"max_tokens": 2000
},
"anthropic.claude-sonnet-5": {
"temperature": 0.5,
"top_p": 0.9,
"anthropic_beta": ["Interleaved-thinking-2025-05-14"]
}
}'
```
**Advanced Configuration:**
```bash
export DEFAULT_MODEL_PARAMS='{
"amazon.nova-pro-v1:0": {
"temperature": 0.7,
"top_p": 0.95,
"max_tokens": 4096,
"stop_sequences": ["Human:", "Assistant:"]
}
}'
```
### Parameter Merging
```mermaid
graph LR
A[Default Parameters] --> B[Merged Config]
C[Request Parameters] --> B
B --> D[Final Configuration]
```
1. :material-numeric-1-circle: **Default parameters** are applied first (from `DEFAULT_MODEL_PARAMS`)
2. :material-numeric-2-circle: **Request parameters** override defaults if both are specified
3. :material-numeric-3-circle: **Provider-specific fields** are forwarded to Bedrock as additional model request fields
4. :material-numeric-4-circle: **Unsupported fields** reach Bedrock as-is, and a field the model rejects surfaces as a `ValidationException` returned to the client as HTTP `400`. Three cases are handled before that: `anthropic_beta` flags are filtered individually against an allowlist (see [`ANTHROPIC_BETA_FILTER`](#anthropic-beta-filter)); a system prompt sent to a model that does not support one is dropped when [`DROP_UNSUPPORTED_SYSTEM_PROMPT`](#drop-unsupported-system-prompt) is enabled (the default); and Amazon Nova 2 drops `max_tokens` when reasoning effort is `high`, logging a warning
---
## :material-layers-triple: Default Model Service Tiers { #default-model-service-tiers-section }
Configure default service tiers applied automatically to specific Bedrock models.
!!! success "What You Can Do"
- :material-layers: Set cost-efficient tiers for batch and agentic workloads by default
- :material-speedometer: Configure priority tiers for latency-sensitive models
- :material-cash: Optimize compute costs without modifying client requests
!!! info "Available Service Tiers"
| Tier | Description |
|------------|-------------------------------------------------------------------|
| `default` | Standard compute tier (default) |
| `flex` | Flexible compute tier for cost optimization |
| `priority` | Priority compute tier for lower latency |
| `reserved` | Reserved capacity for dedicated resources (requires AWS contract) |
!!! tip "When to Use Each Tier"
- **Default**: Everyday AI tasks like content generation and text analysis
- **Flex**: Cost-sensitive workloads like model evaluations, summarization, and agentic workflows
- **Priority**: Mission-critical applications requiring lowest latency
- **Reserved**: Predictable workloads needing 99.5% uptime guarantee (requires AWS contact)
!!! warning "Model Support"
Not all models support all service tiers. Check the [official AWS documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/service-tiers-inference.html) for each model's supported tiers.
**Examples:**
- `amazon.nova-pro-v1:0` supports: `default`, `flex`, `priority` (not `reserved`)
- `amazon.nova-premier-v1:0` (legacy) supports: `default`, `flex`, `priority`, `reserved`
!!! info "Tier Precedence"
Explicit request parameters take precedence over the tier configured for the model, unless [`AWS_BEDROCK_ALLOW_SERVICE_TIER_OVERRIDE`](#aws-bedrock-allow-service-tier-override) is disabled.
#### `DEFAULT_MODEL_SERVICE_TIERS` { #default-model-service-tiers }
:octicons-package-24: **Purpose**
: Per-model default service tier
:octicons-code-24: **Format**
: JSON object with model IDs as keys and tier string as value
:octicons-gear-24: **Default**
: `{}`
**Supported Values:**
| Value | Description |
|------------|---------------------------------------------------|
| `default` | Standard compute (Bedrock default) |
| `flex` | Cost-optimized flexible compute |
| `priority` | Lower-latency priority compute |
| `reserved` | Dedicated reserved capacity (requires AWS contract) |
#### `AWS_BEDROCK_ALLOW_SERVICE_TIER_OVERRIDE` { #aws-bedrock-allow-service-tier-override }
:octicons-package-24: **Purpose**
: Control whether clients can select the service tier at request level
:octicons-gear-24: **Default**
: `true` (clients may select a tier)
:octicons-cash-24: **Cost Consideration**
: Service tiers are billed at different rates. Set to `false` on a shared deployment to pin every model to the tier you configured, so a client cannot move its traffic to a more expensive tier. A model with no configured tier still honors the request in either case.
:octicons-alert-24: **Scope**
: Applies to models served through the Bedrock Converse and InvokeModel APIs. A [Bedrock Mantle](#summary-bedrock-mantle)-served model carries no configured tier, so its requests always run on the tier they name.
```bash
export AWS_BEDROCK_ALLOW_SERVICE_TIER_OVERRIDE=false
```
### Configuration Examples { #service-tier-examples }
**Single Model:**
```bash
export DEFAULT_MODEL_SERVICE_TIERS='{
"amazon.nova-pro-v1:0": "flex"
}'
```
**Multiple Models:**
```bash
export DEFAULT_MODEL_SERVICE_TIERS='{
"amazon.nova-pro-v1:0": "flex",
"amazon.nova-premier-v1:0": "priority"
}'
```
### Service Tier Merging
For models served through the Bedrock Converse and InvokeModel APIs:
1. :material-numeric-1-circle: **Explicit request parameter** takes highest priority
2. :material-numeric-2-circle: **HTTP header** (`X-Amzn-Bedrock-Service-Tier`, see [Per-Request Service Tier Configuration](#service-tier-per-request)) overrides defaults
3. :material-numeric-3-circle: **Tier configured on the requested alias** (see [Model Aliases](#model-aliases-section)) applies if the request sets none
4. :material-numeric-4-circle: **Default from** `DEFAULT_MODEL_SERVICE_TIERS` applies if neither does
5. :material-numeric-5-circle: **No service tier** passed to Bedrock if unset
Steps 1 and 2 are skipped when [`AWS_BEDROCK_ALLOW_SERVICE_TIER_OVERRIDE`](#aws-bedrock-allow-service-tier-override) is `false` and a tier is configured.
On a [Bedrock Mantle](#summary-bedrock-mantle)-served model, only step 1 applies: the request's own `service_tier` is forwarded as sent, and neither the header, nor an alias' tier, nor `DEFAULT_MODEL_SERVICE_TIERS` takes part.
---
## :material-label: Model Aliases { #model-aliases-section }
Configure custom aliases to map user-friendly model names to actual model IDs. This enables OpenAI API compatibility and simplifies model references.
!!! success "What You Can Do"
- :material-label: Create custom aliases for frequently used models
- :material-api: Enable OpenAI-compatible model names by default
- :material-swap-horizontal: Simplify model ID references in API requests
- :material-transition: Seamlessly migrate between model versions
!!! info "Default Aliases"
stdapi.ai includes default aliases for OpenAI compatibility:
- `tts-1` → `amazon.polly-standard`
- `tts-1-hd` → `amazon.polly-neural`
- `whisper-1` → `amazon.transcribe`
stdapi.ai also supports dynamic model name aliases matching official provider APIs (OpenAI, Anthropic). You can use model names from provider documentation (e.g., `claude-sonnet-5`, `gpt-oss-20b`) which are automatically resolved to their corresponding Amazon Bedrock model identifiers.
#### `MODEL_ALIASES` { #model-aliases }
:octicons-package-24: **Purpose**
: Map alias names to actual model IDs or ARNs
:octicons-code-24: **Format**
: JSON object with alias names as keys, and as values either a model ID or ARN, or an object carrying that model plus the configuration to apply to it
:octicons-gear-24: **Default**
: `{}` (empty, uses built-in defaults only)
!!! tip "Advanced Routing with ARNs"
Model aliases can also reference ARNs for Application Inference Profiles or Prompt Routers, enabling advanced routing strategies through friendly alias names. See [Using Inference Profile and Prompt Router ARNs](#using-inference-profile-and-prompt-router-arns) for more details.
### Aliases That Carry Configuration { #model-aliases-configuration }
An alias may map to an object instead of a model name. Every request naming that alias then gets the configuration attached to it, so one deployment can publish the same model under several names with different tiers, safeguards or defaults.
Every field below is designed around an **Amazon Bedrock** model call. An alias pointing at a model served by another AWS service — Amazon Polly, Amazon Transcribe, Amazon Comprehend — still resolves the name, and two fields keep working there: `extra_params` applies wherever the route accepts model parameters, and `guardrail_id` is enforced on the routes that check content through [Amazon Bedrock Guardrails](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html) (speech input, transcripts) — a billed guardrail evaluation. `service_tier` and `metadata` configure the Bedrock call itself and are ignored on those services.
| Field | Purpose |
|----------------------|-----------------------------------------------------------------------------------------------|
| `model` | **Required.** Model ID or ARN the alias resolves to |
| `service_tier` | Service tier for requests naming the alias, on a model served through the Bedrock Converse or InvokeModel APIs — see [Default Model Service Tiers](#default-model-service-tiers-section) |
| `guardrail_id` | ID of an [Amazon Bedrock Guardrail](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html) to apply, requires `guardrail_version`; `guardrail_identifier` is accepted as the same field |
| `guardrail_version` | Version of that guardrail |
| `guardrail_trace` | Guardrail trace level: `disabled`, `enabled` or `enabled_full` |
| `metadata` | Key-value metadata attached to the model call, for audit reporting — it reaches [Amazon Bedrock model invocation logs](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html), which you enable and deliver yourself, and nothing else: it is not a cost allocation tag, see [AWS Cost Attribution](operations_cost_management.md#aws-cost-attribution) |
| `extra_params` | Model parameters, in the format of [`DEFAULT_MODEL_PARAMS`](#default-model-params) |
```bash
export MODEL_ALIASES='{
"support-assistant": {
"model": "amazon.nova-lite-v1:0",
"service_tier": "flex",
"guardrail_id": "abc123def456",
"guardrail_version": "1",
"metadata": {"team": "support"},
"extra_params": {"temperature": 0.2}
}
}'
```
!!! info "Precedence"
Each field resolves in one order: **the request**, then **the alias**, then the **server-wide setting** for that field. A field the alias leaves unset falls through to the server-wide value, and a client that sends nothing gets the alias' configuration.
The two settings that decide whether a request may override an administrator's value apply to the alias layer as well:
- [`AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE`](#aws-bedrock-allow-guardrail-override) — when `false`, the alias' guardrail holds and request headers cannot replace it
- [`AWS_BEDROCK_ALLOW_SERVICE_TIER_OVERRIDE`](#aws-bedrock-allow-service-tier-override) — when `false`, the alias' service tier holds and the request cannot select another
!!! warning "Startup Validation"
An alias object is validated when the server starts: an unknown field, a missing `model`, a guardrail ID without its version, or an out-of-range `extra_params` value stops startup with an error naming the alias. A typo never becomes a silently ignored setting.
An alias whose `guardrail_id` targets a model served through [Bedrock Mantle](#summary-bedrock-mantle) also stops startup: Amazon Bedrock Guardrails do not apply to those models, and serving them unfiltered while a guardrail is configured would be a silent gap. Point the alias at another model, or — when the model is also available on the classic endpoint — remove it from [`AWS_BEDROCK_MANTLE_PREFERRED_MODELS`](#bedrock-mantle-preferred-models) so it is served where guardrails apply.
!!! note "Scope on Bedrock Mantle models"
On a [Bedrock Mantle](#summary-bedrock-mantle)-served model, `guardrail_id` is rejected at startup as above, and `service_tier`, `metadata` and `extra_params` — like the server-wide [`DEFAULT_MODEL_SERVICE_TIERS`](#default-model-service-tiers) and [`DEFAULT_MODEL_PARAMS`](#default-model-params) — do not apply. Such a request runs on the tier it names itself, and on that model's default tier when it names none.
### Configuration Examples { #model-aliases-examples }
**Basic Alias:**
```bash
export MODEL_ALIASES='{
"my-tts": "amazon.polly-neural",
"my-stt": "amazon.transcribe"
}'
```
**Override Default Aliases:**
```bash
# Override the default tts-1 mapping
export MODEL_ALIASES='{
"tts-1": "amazon.polly-generative"
}'
```
**Multiple Custom Aliases:**
```bash
export MODEL_ALIASES='{
"fast-model": "amazon.nova-micro-v1:0",
"balanced-model": "amazon.nova-lite-v1:0",
"quality-model": "amazon.nova-pro-v1:0",
"claude": "anthropic.claude-sonnet-5"
}'
```
**Map OpenAI Models to Bedrock:**
```bash
# Make OpenAI model names work with Amazon Bedrock models
export MODEL_ALIASES='{
"gpt-5": "anthropic.claude-sonnet-5",
"gpt-4o": "anthropic.claude-sonnet-5",
"gpt-4o-mini": "anthropic.claude-haiku-4-5-20251001-v1:0",
"dall-e-3": "amazon.nova-canvas-v1:0",
"dall-e-2": "stability.stable-image-ultra-v1:1"
}'
```
**Override Deprecated Models:**
```bash
# Redirect deprecated model IDs to their newer replacements
export MODEL_ALIASES='{
"amazon.titan-image-generator-v1": "amazon.nova-canvas-v1:0",
"amazon.titan-text-express-v1": "amazon.nova-lite-v1:0",
"anthropic.claude-3-5-sonnet-20240620-v1:0": "anthropic.claude-sonnet-5",
"stability.stable-image-ultra-v1:0": "stability.stable-image-ultra-v1:1"
}'
```
**Advanced Routing with ARNs:**
```bash
# Map friendly names to Application Inference Profiles or Prompt Routers
export MODEL_ALIASES='{
"my-router": "arn:aws:bedrock:us-east-1:123456789012:default-prompt-router/cost-optimizer",
"my-profile": "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123xyz",
}'
```
### Using Aliases in API Requests
Once configured, aliases can be used anywhere a model ID is expected:
```bash
# Using the default tts-1 alias
curl https://api.example.com/v1/audio/speech \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "tts-1",
"input": "Hello world",
"voice": "alloy"
}'
# Using a custom alias
curl https://api.example.com/v1/chat/completions \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "fast-model",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### Alias Resolution
```mermaid
graph LR
A[API Request] --> B{Alias Exists?}
B -->|Yes| C[Resolve to Model ID]
B -->|No| D[Use as Model ID]
C --> E[Model Validation]
D --> E
E --> F[Execute Request]
```
1. :material-numeric-1-circle: **User-configured aliases** override default aliases
2. :material-numeric-2-circle: **Default aliases** apply if not overridden
3. :material-numeric-3-circle: **Non-aliased names** pass through unchanged
4. :material-numeric-4-circle: **Resolved model ID** is validated and used for the request
---
## :material-message-text: System Prompt Handling
Control how system prompts are handled for models that don't support them.
#### `DROP_UNSUPPORTED_SYSTEM_PROMPT` { #drop-unsupported-system-prompt }
:octicons-package-24: **Purpose**
: Control system prompt behavior for models that don't support system prompts
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `true`
```bash
# Default: silently drop system prompts for unsupported models
export DROP_UNSUPPORTED_SYSTEM_PROMPT=true
# Strict mode: return error when system prompt is used with unsupported model
export DROP_UNSUPPORTED_SYSTEM_PROMPT=false
```
!!! info "Models Without System Prompt Support"
Some Bedrock models don't support system prompts, including:
- `mistral.mistral-7b-instruct-v0:2`
- `mistral.mixtral-8x7b-instruct-v0:1`
- Other older or specialized models
!!! success "Use Cases"
**Enable (true, default)** for:
- :material-check: **Backward compatibility** - Existing applications continue working
- :material-swap-horizontal: **Model flexibility** - Switch between models without code changes
- :material-shield-check: **Graceful degradation** - System prompts are ignored instead of failing
- :material-application: **Global system prompts** - Applications that set system prompts globally for all models work seamlessly
**Disable (false)** for:
- :material-alert: **Strict validation** - Catch configuration errors early
- :material-bug: **Debugging** - Identify when system prompts aren't being used
- :material-shield-alert: **Security requirements** - Ensure system prompts are always applied
## :material-flask: Anthropic Beta Flag Filtering
Anthropic-compatible clients like Claude Code send `anthropic-beta` headers with experimental beta flags. Many of these flags (such as `files-api-2025-04-14`, `prompt-caching-2024-07-31`) are **not supported by Amazon Bedrock** and cause `ValidationException` errors (HTTP 400).
stdapi.ai automatically filters out unsupported flags while preserving supported ones, so clients work without any special configuration. Previously, the workaround was to set `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1` on the client side, but this also disabled Bedrock-supported flags like `Interleaved-thinking-2025-05-14` and `token-efficient-tools-2025-02-19`, degrading capabilities. This workaround is no longer needed.
Filtering is controlled by two settings: [`ANTHROPIC_BETA_FILTER`](#anthropic-beta-filter) to enable or disable it, and [`ANTHROPIC_BETA_ALLOWLIST`](#anthropic-beta-allowlist) to extend the built-in set of allowed flags.
#### `ANTHROPIC_BETA_FILTER` { #anthropic-beta-filter }
:octicons-package-24: **Purpose**
: Enable or disable filtering of unsupported `anthropic_beta` flags for Anthropic Claude models
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `true`
:octicons-workflow-24: **Behavior**
: When enabled, `anthropic_beta` flags not in the allowlist are silently removed from requests before they reach Bedrock. A warning is logged when flags are filtered. When disabled, all flags are passed through to Bedrock as-is
```bash
# Enabled (default) - filter unsupported flags automatically
# No environment variable needed
# Disable filtering entirely (pass all flags through to Bedrock)
export ANTHROPIC_BETA_FILTER=false
```
!!! tip "When to Disable"
Set to `false` only when:
- :material-test-tube: **Testing** - You want to verify Bedrock behavior with specific flags directly
- :material-cog: **Custom setups** - You manage flag compatibility at the client level
#### `ANTHROPIC_BETA_ALLOWLIST` { #anthropic-beta-allowlist }
:octicons-package-24: **Purpose**
: Add extra `anthropic_beta` flags to the built-in set of Bedrock-supported flags
:octicons-code-24: **Format**
: Comma-separated string of additional beta flag names
:octicons-gear-24: **Default**
: Empty (only the built-in Bedrock defaults are used)
:octicons-workflow-24: **Behavior**
: The flags specified here are **merged with** the built-in set of Bedrock-supported flags. You only need to specify extra flags beyond the defaults (e.g., newly added Bedrock flags). Only effective when [`ANTHROPIC_BETA_FILTER`](#anthropic-beta-filter) is `true`
```bash
# Use built-in defaults only (recommended) - no environment variable needed
# Add newly supported Bedrock flags without waiting for a stdapi.ai update
export ANTHROPIC_BETA_ALLOWLIST='new-feature-2026-03-01,another-flag-2026-04-01'
```
**Built-in Allowed Flags:**
| Flag | Feature |
|------------------------------------|-------------------------------|
| `computer-use-2024-10-22` | Computer use (Claude 3.5) |
| `computer-use-2025-01-24` | Computer use (Claude 3.7) |
| `computer-use-2025-11-24` | Computer use (Claude 4.5/4.6) |
| `token-efficient-tools-2025-02-19` | Token efficient tools |
| `Interleaved-thinking-2025-05-14` | Interleaved thinking |
| `output-128k-2025-02-19` | 128K output |
| `dev-full-thinking-2025-05-14` | Raw thinking dev mode |
| `context-1m-2025-08-07` | 1M context |
| `context-management-2025-06-27` | Context management (memory) |
| `effort-2025-11-24` | Effort control |
| `tool-search-tool-2025-10-19` | Tool search |
| `tool-examples-2025-10-29` | Tool use examples |
!!! success "Use Cases"
**Filtering enabled (default)** for:
- :material-robot: **Claude Code via Bedrock** - Clients work without `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1`
- :material-shield-check: **Production stability** - Prevent unsupported flags from causing request failures
- :material-swap-horizontal: **Drop-in compatibility** - Clients configured for direct Anthropic API work through stdapi.ai without changes
#### `EXTRA_MODEL_PARAMS_DENYLIST` { #extra-model-params-denylist }
:octicons-package-24: **Purpose**
: Add extra parameter names to strip from the "extra model parameters" passthrough (any undeclared top-level JSON field on a chat or non-chat route, forwarded to Bedrock as a provider-specific inference field)
:octicons-code-24: **Format**
: Comma-separated string of additional parameter names
:octicons-gear-24: **Default**
: Empty (only the built-in denylist is used)
:octicons-workflow-24: **Behavior**
: The names specified here are **merged with** the built-in denylist of LiteLLM client-control parameters (such as `drop_params`, `api_key`, `custom_llm_provider`) that some OpenAI-SDK-based clients leak into `extra_body` and that are never legitimate Bedrock model parameters — for example RAGFlow hardcodes `extra_body={"drop_params": True}` on every embeddings call, which previously reached Bedrock as an unrecognized inference field and failed with `ValidationException`. Every other extra parameter keeps being forwarded as before. Only effective when [`EXTRA_MODEL_PARAMS_DROP_ALL`](#extra-model-params-drop-all) is `false`
```bash
# Use the built-in denylist only (recommended) - no environment variable needed
# Also strip a project-specific control field some client leaks into requests
export EXTRA_MODEL_PARAMS_DENYLIST='x_internal_debug_flag,x_proxy_trace_id'
```
#### `EXTRA_MODEL_PARAMS_DROP_ALL` { #extra-model-params-drop-all }
:octicons-package-24: **Purpose**
: Disable the "extra model parameters" passthrough entirely
:octicons-database-24: **Type**
: Boolean
:octicons-gear-24: **Default**
: `false`
:octicons-workflow-24: **Behavior**
: When enabled, no undeclared request field is ever forwarded to Bedrock as a provider-specific inference parameter, on every route that supports the passthrough (chat completions/responses/messages, and embeddings/images/audio/rerank/etc.). This overrides [`EXTRA_MODEL_PARAMS_DENYLIST`](#extra-model-params-denylist): with drop-all enabled, denylist filtering no longer matters because nothing is forwarded. Per-model defaults configured through [`DEFAULT_MODEL_PARAMS`](#default-model-params) are unaffected — only request-supplied extras are dropped
```bash
# Keep the passthrough (default) - no environment variable needed
# Lock the deployment down to only declared API fields
export EXTRA_MODEL_PARAMS_DROP_ALL=true
```
!!! tip "When to Enable"
Set to `true` only when you need to guarantee that no undeclared client field ever reaches Bedrock, for example a strict multi-tenant deployment where provider-specific knobs must go through an explicit allowlisted mechanism instead of the passthrough.
---
## :material-image: Image Generation
#### `IMAGE_GENERATION_MODEL` { #image-generation-model }
:octicons-package-24: **Purpose**
: Default Bedrock image model ID used when the [`image_generation`](api_openai_responses.md#image-generation) integrated tool is invoked via the Responses API. The tool intercepts requests from any text model, generates the image against this Bedrock image model, and returns an `image_generation_call` output item.
:octicons-database-24: **Type**
: String (Bedrock image model ID)
:octicons-gear-24: **Default**
: None — the tool returns HTTP 400 if no model is configured and the request does not specify one
:octicons-workflow-24: **Behavior**
: The tool definition in the request may include a `model` field to override this default per call. Priority: request `model` field > this env var. Any available Bedrock image generation model can be used — for example `amazon.nova-canvas-v1:0`, `amazon.titan-image-generator-v2:0`, or the Stability AI Stable Image / Stable Diffusion family. Legacy models (such as `amazon.titan-image-generator-v1` and `stability.stable-diffusion-xl-v1`) are hidden unless [`AWS_BEDROCK_LEGACY`](#bedrock-legacy) is enabled. Use the [Search Models API](api_search_models.md) to list the image models available in your deployment.
```bash
export IMAGE_GENERATION_MODEL='amazon.nova-canvas-v1:0'
```
With this set, any text model can generate images via the Responses API:
```bash
curl -X POST "$BASE/v1/responses" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "amazon.nova-micro-v1:0",
"input": "Generate a sunset over the ocean.",
"tools": [{"type": "image_generation"}],
"tool_choice": "required"
}'
```
---
## :material-directions-fork: Using Inference Profile and Prompt Router ARNs { #using-inference-profile-and-prompt-router-arns }
stdapi.ai supports passing ARNs directly as model IDs in API requests, enabling advanced routing capabilities beyond standard model selection.
!!! tip "Simplify ARNs with Model Aliases"
Instead of using long ARNs directly in API requests, you can create [Model Aliases](#model-aliases) that map friendly names to ARNs. This provides shorter, easier-to-use naming for your API users.
### Overview
Instead of using standard model IDs like `anthropic.claude-sonnet-5`, you can pass ARNs that reference:
- **Cross-Region Inference Profiles** - AWS-managed multi-region routing
- **Application Inference Profiles** - Your custom routing configurations
- **Prompt Routers** - Intelligent dynamic model selection
!!! info "Automatic Cross-Region Routing"
**stdapi.ai automatically handles cross-region routing by default.** When you use standard model IDs, the application automatically selects and uses the optimal AWS-managed cross-region inference profile based on your configured `AWS_BEDROCK_REGIONS`.
You typically **do not need to manually pass cross-region inference profile ARNs**. The automatic selection handles routing across your configured regions for best availability and latency.
Manual ARN passing is primarily useful for:
- :material-application: **Application inference profiles** - Your custom routing configurations
- :material-robot: **Prompt routers** - Intelligent cost optimization and dynamic model selection
- :material-cog: **Rare cases** - When you need to override automatic cross-region profile selection
### Enabling ARN Support
By default, users can only pass standard model IDs. To allow ARN usage, enable the appropriate settings:
```bash
# Allow cross-region inference profile ARNs
export AWS_BEDROCK_ALLOW_CROSS_REGION_INFERENCE_PROFILE_ARN=true
# Allow application inference profile ARNs
export AWS_BEDROCK_ALLOW_APPLICATION_INFERENCE_PROFILE_ARN=true
# Allow prompt router ARNs
export AWS_BEDROCK_ALLOW_PROMPT_ROUTER_ARN=true
```
!!! warning "Security Consideration"
These settings are disabled by default. Only enable them when you want to give users explicit control over ARN-based routing. For centralized server-controlled routing, use [`AWS_BEDROCK_MODEL_ARN_MAPPING`](#bedrock-model-arn-mapping) instead.
### Using ARNs in API Requests
Once enabled, users can pass ARNs directly in the `model` parameter:
**Cross-Region Inference Profile Example:**
```bash
curl -X POST https://api.example.com/v1/chat/completions \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{
"model": "arn:aws:bedrock:us-east-1:123456789012:inference-profile/us.anthropic.claude-sonnet-5",
"messages": [
{"role": "user", "content": "Hello!"}
]
}'
```
**Application Inference Profile Example:**
```bash
curl -X POST https://api.example.com/v1/chat/completions \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{
"model": "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/my-custom-profile",
"messages": [
{"role": "user", "content": "Hello!"}
]
}'
```
**Prompt Router Example:**
```bash
curl -X POST https://api.example.com/v1/chat/completions \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{
"model": "arn:aws:bedrock:us-east-1:123456789012:default-prompt-router/my-router",
"messages": [
{"role": "user", "content": "Hello!"}
]
}'
```
### Use Case Comparison
| Approach | Best For | Configuration |
|-----------------------------|---------------------------------------------|---------------------------------------------------------------|
| **Standard Model IDs** | Most common use case, simple routing | No special configuration needed |
| **Server-Side ARN Mapping** | Centralized control, transparent to clients | [`AWS_BEDROCK_MODEL_ARN_MAPPING`](#bedrock-model-arn-mapping) |
| **Client-Side ARN Passing** | User-controlled routing, advanced use cases | Enable `AWS_BEDROCK_ALLOW_*_ARN` settings |
### Best Practices
!!! success "Recommended Approach"
**For most deployments, use server-side ARN mapping** ([`AWS_BEDROCK_MODEL_ARN_MAPPING`](#bedrock-model-arn-mapping)):
- :material-server: Centralized control over routing behavior
- :material-account-group: Transparent to API clients
- :material-cog: Easy to change routing without modifying client code
- :material-shield-check: Better security (server controls which ARNs are used)
!!! info "When to Allow Client-Side ARNs"
Enable `AWS_BEDROCK_ALLOW_*_ARN` settings when:
- :material-api: Clients need fine-grained control over routing
- :material-cog: Different clients require different routing strategies
- :material-dev-to: Advanced users managing their own inference profiles
- :material-test-tube: Testing and comparing different routing configurations
!!! warning "Security and Governance"
When enabling client-side ARN passing:
- :material-shield-alert: Clients can bypass server-configured routing
- :material-cash: Monitor usage to prevent unexpected costs
- :material-account-check: Ensure appropriate IAM permissions are in place
- :material-chart-line: Track ARN usage through logs and monitoring
### Required IAM Permissions
When using ARN-based routing, ensure your IAM role/user has the appropriate permissions:
```json
{
"Sid": "BedrockARNRouting",
"Effect": "Allow",
"Action": [
"bedrock:GetInferenceProfile",
"bedrock:GetPromptRouter"
],
"Resource": "*"
}
```
See the [IAM Permissions](operations_iam_permissions.md) page for complete policy examples.
---
## :material-arrow-right: Next Steps
- :material-shield-key: [**IAM Permissions**](operations_iam_permissions.md) — Complete IAM policy reference
- :material-lock: [**Authentication & Security**](operations_authentication_security.md) — Secure your deployment
- :material-shield-check: [**Resilience & Failover**](operations_resilience.md) — Region routing and failover behavior
- :material-chart-line: [**Logging & Monitoring**](operations_logging_monitoring.md) — Observability and metrics
---
## Authentication & Security
Source: https://stdapi.ai/md/operations_authentication_security.md
# :material-lock: Authentication & Security
stdapi.ai provides flexible authentication options and built-in security mechanisms to protect your API. Choose from API key authentication for simple deployments to enterprise-grade identity management via AWS services — all backed by built-in SSRF protection, host header validation, and configurable encryption in transit.
- :material-key: __API Key Authentication__
Securely stored in AWS SSM Parameter Store or Secrets Manager. Mimics OpenAI and Anthropic auth.
- :material-account-key: __Amazon Cognito User Pool Tokens__
Per-user and per-application bearer tokens, validated on every request. No key to rotate.
- :material-compass-outline: __Authentication Discovery for Agents__
Publish where tokens come from, so an agent authenticates itself with no prior configuration.
- :material-account-check: __OIDC, Cognito & IAM Identity Center__
Offload user and workforce identity management to AWS ALB or API Gateway.
- :material-shield-key: __AWS IAM__
Use AWS native access control via API Gateway.
- :material-lock-open: __No Authentication__
Optional mode for private/internal VPC deployments where security is handled at the network level.
---
## :material-key-chain: Authentication Methods
| Method | Best for | AWS infrastructure | Enforced by |
|---|---|---|---|
| **API Key** | Server-to-server, simple deployments | Any | stdapi.ai |
| **Amazon Cognito user pool token** | Per-user access, autonomous agents, revocable credentials | Cognito user pool | stdapi.ai |
| **OIDC / Cognito / IAM Identity Center** | User-facing apps, SSO, workforce identity | ALB or API Gateway | AWS (before request reaches stdapi.ai) |
| **AWS IAM** | Service-to-service within AWS | API Gateway | AWS (SigV4) |
| **No Authentication** | Local development, trusted VPC | Any | Network controls only |
The API key and Cognito tokens can be used together: [`AUTHENTICATION_MODE`](operations_configuration.md#authentication-mode) selects which of them a deployment accepts, and defaults to accepting every method that is configured.
When stdapi.ai enforces the credential itself — the API key and Cognito user pool tokens — a rejected request is answered with `401 Unauthorized`, a `WWW-Authenticate: Bearer` challenge, and a body that states nothing beyond `Unauthorized` — the reason is written to the server log only, so the response cannot be used to work out which half of a credential was wrong. That challenge is also where an AI agent starts: see [Authentication Discovery for Agents](#authentication-discovery-for-agents). The edge-enforced methods answer an unauthenticated request themselves, before it reaches stdapi.ai, with whatever their own configuration says — commonly a redirect to the identity provider.
### :material-key-star: API Key Authentication
The API key authentication method uses a single API key that mimics the behavior of upstream OpenAI and Anthropic authentication. It must be provided by clients in the `Authorization: Bearer ` or `X-API-Key` header.
Three sources are supported — configure exactly one. If more than one is set, the first match in this precedence order is used (the others are ignored): **Direct value → SSM Parameter Store → Secrets Manager**.
- **Direct value** (`API_KEY`) — for local development and testing only; not recommended for production.
- **SSM Parameter Store** (`API_KEY_SSM_PARAMETER`) — the parameter must already exist before startup.
- **Secrets Manager** (`API_KEY_SECRETSMANAGER_SECRET`) — the secret must already exist; supports a configurable key within the JSON secret via `API_KEY_SECRETSMANAGER_KEY`.
For the full list of environment variables and required IAM permissions for each method, see the [Configuration Guide](operations_configuration.md#authentication).
!!! abstract "In-Memory Key Protection"
stdapi.ai never stores API keys in plain text. At startup, the key is retrieved, salted, and hashed using an industry-standard cryptographic function; only the hash is retained in memory — even a full memory dump cannot reconstruct the original key. Verification uses constant-time comparison to prevent timing-based side-channel attacks.
!!! info "Terraform Module"
The Terraform module offers additional options for providing the API key:
- **`api_key`** — provide the key directly; it is injected as an **ECS Secret** (sourced from an encrypted SSM parameter with KMS), never exposed in task definitions or logs.
- **`api_key_create`** — auto-generate a secure 64-character random key; the generated value is returned as a sensitive Terraform output.
- **`api_key_ssm_parameter`** / **`api_key_secretsmanager_secret`** — reference an existing SSM parameter or Secrets Manager secret; the module does not create these resources.
### :material-account-key: Amazon Cognito User Pool Tokens
stdapi.ai can accept the bearer tokens issued by an [Amazon Cognito user pool](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools.html) as an alternative to the static API key, validating each one on every request. Each caller — a person or an application — gets its own short-lived credential, so there is no shared key to rotate, and the verified caller becomes the identity [per-user cost attribution](operations_cost_management.md#per-user-attribution) bills against.
Clients send the token exactly as they would send an API key, in either header:
```bash
curl https://your-gateway.example.com/v1/models \
-H "Authorization: Bearer eyJraWQiOi..."
# Anthropic-compatible routes, whose SDK uses its own header
curl https://your-gateway.example.com/anthropic/v1/models \
-H "x-api-key: eyJraWQiOi..." -H "anthropic-version: 2023-06-01"
```
#### What you configure in AWS
1. **A user pool**, in any Region. Its ID (for example `eu-west-3_a1b2c3d4e`) goes into [`AWS_COGNITO_USER_POOL_ID`](operations_configuration.md#aws-cognito-user-pool-id); the Region is read from the ID itself, and so is the issuer URL — nothing else names it, here or in [agent discovery](#authentication-discovery-for-agents).
2. **One or more app clients** in that pool. Their IDs go into [`AWS_COGNITO_CLIENT_IDS`](operations_configuration.md#aws-cognito-client-ids) — a token issued to any other app client is rejected, so this list is required.
3. **A [pool domain](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-assign-domain.html)**, which is what gives the pool its OAuth 2.0 authorization and token endpoints. It is needed in two cases: to demand a scope such as `stdapi/invoke` through [`AWS_COGNITO_REQUIRED_SCOPES`](operations_configuration.md#aws-cognito-required-scopes), since custom scopes exist only on tokens those endpoints issue and on a resource server that defines them; and for the [agent discovery flow](#authentication-discovery-for-agents), scope or no scope, since an agent has nowhere to obtain a token otherwise. Machine-to-machine clients use the `client_credentials` grant there.
No IAM permission is involved: the pool's signing keys are public. The task must be able to **reach** them, though — it reads them over HTTPS from `cognito-idp..amazonaws.com` in the pool's Region at startup, and fails to start if it cannot. Give the task outbound HTTPS to that host: NAT or internet egress, or an [interface VPC endpoint](https://docs.aws.amazon.com/cognito/latest/developerguide/vpc-interface-endpoints.html) for `com.amazonaws..cognito-idp` in that Region — note that AWS declares a pool with a domain assigned incompatible with that endpoint, so a pool used for scopes or for agent discovery needs the egress path.
!!! danger "The pool decides who can call the gateway"
Any identity that can obtain a token from an app client listed in [`AWS_COGNITO_CLIENT_IDS`](operations_configuration.md#aws-cognito-client-ids) can call the API, and its model usage is billed to your account. A Cognito user pool **allows self sign-up by default**, so pointing the gateway at a customer-facing pool without further restriction turns it into an open, self-service one.
Before enabling it: disable self-registration on the pool (`AllowAdminCreateUserOnly`), or dedicate an app client to the gateway and require one of its resource-server scopes.
#### What stdapi.ai validates
Every request is checked against all of the following, and any failure returns the same `401 Unauthorized` with no detail (the reason is recorded in the server log only):
| Check | Requirement |
|---|---|
| **Signature** | RS256, against the pool's published keys. Unsigned (`alg=none`) and symmetric (`HS*`) tokens are always rejected. |
| **Issuer** | Exactly the configured pool's issuer — see [`AWS_COGNITO_ISSUER_TYPE`](operations_configuration.md#aws-cognito-issuer-type). A token from another pool is rejected. |
| **Token use** | An access token. Identity tokens are rejected unless [`AWS_COGNITO_ACCEPT_ID_TOKEN`](operations_configuration.md#aws-cognito-accept-id-token) is enabled. |
| **Application** | The token's app client is in [`AWS_COGNITO_CLIENT_IDS`](operations_configuration.md#aws-cognito-client-ids). |
| **Validity period** | Not expired, and not used before its start time, with a one-minute tolerance for clock drift. |
| **Scopes** | All of [`AWS_COGNITO_REQUIRED_SCOPES`](operations_configuration.md#aws-cognito-required-scopes) are present. |
!!! abstract "Signing keys are loaded once, at startup"
The pool's public keys are read at startup and kept in memory, so validation adds no network call and no measurable latency to a request. When a pool rotates its keys, the first request carrying a token signed by the new key reloads them, at most once every five minutes — a forged key identifier cannot turn requests into outbound traffic. A deployment that cannot read the keys at startup **fails to start** rather than serving requests it could not authenticate.
!!! warning "Revocation takes effect when the token expires"
A token is validated against the pool's published keys and its own claims, without calling the pool, so nothing is checked back with it once it has been issued. Disabling a user, deleting them or signing them out therefore stops the **next** token, not the one already in their hands: that one keeps working until its `exp`, up to the app client's access-token validity (one hour by default, 24 hours at most). Keep that validity short — it is the upper bound on how long a revoked credential stays usable.
!!! warning "Username and password sign-in yields no custom scope"
Tokens obtained by signing in directly against the user pool API carry the single scope `aws.cognito.signin.user.admin`. Custom scopes only exist on tokens issued by the pool's OAuth 2.0 token endpoint. Requiring `stdapi/invoke` therefore rejects every client that signs in with a username and password — leave [`AWS_COGNITO_REQUIRED_SCOPES`](operations_configuration.md#aws-cognito-required-scopes) empty unless all your clients use the OAuth 2.0 endpoints.
!!! tip "Both methods, or one"
With both a user pool and an API key configured, either credential is accepted: a bearer value shaped like a signed token is validated against the pool, anything else is compared to the API key. Set [`AUTHENTICATION_MODE`](operations_configuration.md#authentication-mode) to `cognito` or `api_key` to accept only one of them — the deployment then refuses to start if the other one is configured too, so a credential is never accepted by accident.
### :material-compass-outline: Authentication Discovery for Agents
An AI agent that meets an API it has never been configured for has one thing to go on: the `401 Unauthorized` it just received. stdapi.ai answers that request with everything the agent needs to obtain a credential on its own — the standard discovery flow behind MCP client authentication.
**Name the public URL and the token issuer, and the discovery surface turns on:**
| Setting | Value |
|---|---|
| [`OAUTH_RESOURCE_IDENTIFIER`](operations_configuration.md#oauth-resource-identifier) | The public URL clients dial, exactly as they dial it — for example `https://api.example.com` |
| [`OAUTH_AUTHORIZATION_SERVERS`](operations_configuration.md#oauth-authorization-servers) | The issuer URL of whatever issues your tokens. Leave it unset with a user pool configured: its issuer is published for you |
| [`OAUTH_SCOPES_SUPPORTED`](operations_configuration.md#oauth-scopes-supported) | Leave it unset too — [`AWS_COGNITO_REQUIRED_SCOPES`](operations_configuration.md#aws-cognito-required-scopes) is published, since an agent that is not told which scope to ask for obtains a token without it and is refused on every retry |
!!! success "With a user pool, one variable turns discovery on"
The pool named by [`AWS_COGNITO_USER_POOL_ID`](operations_configuration.md#aws-cognito-user-pool-id) issues the tokens this deployment accepts, so it *is* the authorization server clients must be sent to, and the scopes it requires are the scopes they must ask for. Both are published from it, and the issuer is exactly what the pool signs into `iss`:
```text
https://cognito-idp..amazonaws.com/
https://issuer-cognito-idp..amazonaws.com/ # AWS_COGNITO_ISSUER_TYPE=updated
```
`` and `` are read from the pool ID itself, and the host follows that Region's AWS partition — `amazonaws.com.cn` in China, `amazonaws.eu` in the European Sovereign Cloud. Set the two variables only for an issuer no pool can supply, such as an identity-aware proxy in front of the deployment; the pool's own issuer must stay in the list, or startup fails rather than sending clients to an authorization server whose tokens are refused.
**What an agent then sees:**
```console
$ curl -i https://api.example.com/v1/models
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer resource_metadata="https://api.example.com/.well-known/oauth-protected-resource", scope="stdapi/invoke"
$ curl https://api.example.com/.well-known/oauth-protected-resource
{
"resource": "https://api.example.com",
"authorization_servers": ["https://cognito-idp.eu-west-3.amazonaws.com/eu-west-3_a1b2c3d4e"],
"scopes_supported": ["stdapi/invoke"],
"bearer_methods_supported": ["header"],
"resource_name": "stdapi.ai (Community Edition)",
"resource_documentation": "https://stdapi.ai/api_reference/"
}
```
From there the agent reads the authorization server's own metadata, signs in, and retries the request with a bearer token. The document is public and unauthenticated by design — it is read by a client that has no credential yet — and contains nothing beyond the issuer URL you configured. It is also cacheable, and it is advertised from the [API catalog and the root `Link` header](features.md#agent-discovery), so an agent that starts from either entry point finds it without a 401.
!!! info "Both the API and the MCP server are covered"
The identifier names the deployment's origin, which every surface shares: an MCP client connecting to `/mcp` or `/sse`, and an SDK calling `/v1/…`, all match the same published resource. Only one document is served, at the root.
!!! warning "`/.well-known/openid-configuration` is deliberately not served"
stdapi.ai is a **resource server**, not an authorization server: it validates tokens, it does not issue them. Mirroring or redirecting the authorization server's own discovery document would create a second, staler copy of something the issuer already publishes authoritatively. Agents reach it through `authorization_servers` instead — which is exactly what the standard flow does.
!!! warning "Amazon Cognito needs a pre-registered app client"
A Cognito user pool publishes neither a dynamic client registration endpoint nor client-id metadata document support, so an agent cannot register itself. Create the app client in the pool and give the agent its client ID (and secret, for confidential clients) ahead of time. Discovery still saves the agent everything else: the issuer, the endpoints, and the scopes.
The pool also needs a [domain](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-assign-domain.html): its authorization and token endpoints exist only once one is assigned, so without it the issuer's own discovery document names no endpoint the agent can obtain a token from, and the flow dead-ends after the 401.
!!! tip "Browser-hosted clients need CORS"
A client running in a browser reads this document cross-origin. Add its origin to [`CORS_ALLOW_ORIGINS`](operations_configuration.md#cors-allow-origins), which it also needs for the API calls that follow.
With a CORS origin configured, the gateway also marks `WWW-Authenticate` as readable cross-origin, so a page-hosted client reads the metadata URL and the advertised scope straight off the `401`. Without one, the browser hides that header and the client falls back to the default well-known path, losing the scope the challenge carries.
### :material-account-check: OIDC, Cognito & IAM Identity Center
For user-facing applications or enterprise SSO, you can offload authentication to an OpenID Connect (OIDC) provider, Amazon Cognito, or AWS IAM Identity Center (the AWS-native workforce SSO service for centralized employee and partner access).
When authentication is handled at this layer, requests are fully validated **before they reach stdapi.ai** — the application receives only authenticated, pre-authorized requests. This remains a valid alternative to [validating Cognito tokens in stdapi.ai](#amazon-cognito-user-pool-tokens): choose the edge when you need a hosted sign-in flow or session cookies, and choose in-application validation when clients already hold a bearer token and you want the verified caller to drive [per-user cost attribution](operations_cost_management.md#per-user-attribution).
!!! tip "Trusted by AWS teams — no custom implementation risk"
Cognito and IAM Identity Center are the same identity systems AWS teams already rely on for console access and internal applications. By delegating authentication to these services, you get MFA, SSO, and fine-grained permission sets without building or maintaining custom user management logic — and without the security risk of a home-grown implementation. See [Eliminate key rotation with AWS native auth](#security-best-practices) for the operational payoff.
!!! info "Terraform Module"
The Terraform module does not configure the ALB or API Gateway integrations described in this section, nor the identity provider itself — these are set up directly on the ALB or API Gateway. Validating Cognito tokens **in stdapi.ai** needs no infrastructure beyond the pool: see [Amazon Cognito User Pool Tokens](#amazon-cognito-user-pool-tokens).
#### via Application Load Balancer (ALB)
The AWS ALB can authenticate users before forwarding requests to stdapi.ai. This is the most common way to add OIDC or Cognito authentication.
- **Capabilities**: Integrates with any OIDC-compliant identity provider — including **AWS IAM Identity Center** (workforce SSO), **Amazon Cognito** User Pools, or third-party providers such as Okta, Auth0, and Google.
- **IAM Identity Center**: Expose the Identity Center OIDC application endpoints as the ALB OIDC configuration. This gives employees and partners seamless SSO using their AWS-managed identity, with support for MFA and permission sets defined in your AWS Organization.
- **Documentation**: [Authenticate users using an Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-authenticate-users.html)
#### via API Gateway
Both REST and HTTP APIs support OIDC and Cognito integration.
- **HTTP API (JWT)**: Native support for JWT-based authorization — any OIDC provider (including IAM Identity Center configured as a trusted token issuer) can be used.
- [Controlling access to HTTP APIs with JWT](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-access-control.html)
- **REST API**: Supports Cognito User Pool authorizers and Lambda Authorizers for custom authentication logic.
- [Control access to a REST API in API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html)
### :material-shield-key: AWS IAM
For service-to-service communication within AWS or when using IAM roles for access control, you can use AWS IAM authentication via API Gateway. Clients sign their requests using the AWS Signature Version 4 (SigV4) process, enabling fine-grained access control using IAM policies attached to users or roles.
#### via API Gateway
Amazon API Gateway provides native IAM authentication for both REST and HTTP APIs.
- **Documentation**:
- [IAM authorization for REST APIs](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html)
- [IAM authorization for HTTP APIs](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-access-control.html)
### :material-lock-open: No Authentication
In certain environments, you may choose to disable application-level authentication and rely entirely on network security.
#### When to Use
- Local development or testing using [Docker](operations_getting_started_local.md).
- Internal VPC deployments where traffic is trusted.
- When an upstream proxy handles all security and identity.
#### Configuration
To disable application-level authentication, do **not** configure any API key environment variable (`API_KEY`, `API_KEY_SSM_PARAMETER`, or `API_KEY_SECRETSMANAGER_SECRET`) and no user pool (`AWS_COGNITO_USER_POOL_ID`).
When no authentication method is configured, stdapi.ai will:
1. Accept all incoming requests without validating an API key.
2. Log a security warning at startup.
!!! danger "Network Security Requirement"
When API key authentication is disabled, you **must** ensure strict network security. Use Security Groups to restrict access so that the stdapi.ai service only accepts traffic from your trusted Load Balancer or API Gateway.
---
## :material-shield-outline: Application Security
stdapi.ai includes built-in security mechanisms that are active regardless of the authentication method chosen.
!!! abstract "Hardened Container Image"
The container images provided via the AWS Marketplace are security-hardened with a minimal base containing no unnecessary system tools or packages. The Terraform module further enforces a **read-only root filesystem** and **drops all Linux capabilities** from the ECS task definition, reducing the attack surface to the minimum required for operation.
[:octicons-arrow-right-24: Subscribe on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-su2dajk5zawpo)
### :material-package-variant-closed: Supply Chain Security
Supply chain attacks — where malicious code is injected into a software package before it reaches your infrastructure — represent one of the most dangerous threat vectors for self-hosted AI gateways. A compromised package can silently exfiltrate API keys and credentials from every deployment that installs it, and attacks of this kind have targeted the AI tooling ecosystem.
stdapi.ai's distribution model eliminates this attack surface entirely:
- **Container-only distribution** — stdapi.ai is never distributed as a pip package or PyPI dependency. There is no `pip install stdapi.ai`, no transitive dependency chain to compromise, and no package registry to hijack. The only distribution channels are the AWS Marketplace ECR registry (commercial) and GHCR (community image).
- **AWS Marketplace security validation** — the commercial container image is scanned and validated by AWS before being made available in the Marketplace. The image you deploy is exactly the image AWS validated — nothing can be inserted between validation and deployment.
- **Minimal base image** — the container has no shell, no package manager, and no unnecessary system tools. There is no mechanism to install additional packages or execute arbitrary commands at runtime.
- **Immutable deployment** — the container image is pulled once at task start and run as-is. There are no runtime `pip install`, `apt install`, or dependency resolution steps that could fetch and execute untrusted code.
- **Read-only root filesystem** — when deployed via the Terraform module, the ECS task definition enforces a read-only root filesystem, preventing any modification to the container files at runtime.
- **Temporary credentials with least privilege** — CI/CD pipelines that build and publish the container image use short-lived, role-scoped credentials with only the permissions required for that specific job. No long-lived access keys are used in the build or release process. Jobs that do not require AWS access — such as linters and security scanners — run on isolated runners with no AWS credentials at all, limiting the blast radius of any compromised job.
### :material-shield-alert-outline: SSRF Protection
Users can pass URLs to the API as multimodal content references (images, documents, audio). The application fetches the content at those URLs before forwarding it to the model. Without protection, a malicious user could supply URLs pointing to internal services — such as the AWS EC2/ECS metadata endpoint or private network resources — and read sensitive data through the model response.
stdapi.ai validates every user-supplied URL via DNS resolution before fetching its content — each hostname is resolved and every resulting IP is checked against the blocklist. This protects against DNS rebinding attacks where a seemingly-safe domain resolves to a blocked address.
- **Baseline Protection**: Always-active blocking of loopback (127.0.0.1, localhost), link-local (169.254.169.254), and reserved address ranges. This prevents access to the AWS EC2/ECS metadata service from within the container.
- **Private Network Blocking**: By default, the service also blocks every address that is not globally reachable on the public Internet — RFC 1918 networks (10.x.x.x, 172.16-31.x.x, 192.168.x.x), IPv6 unique local addresses, RFC 6598 shared address space (100.64.0.0/10, used by EKS custom networking and Hybrid Nodes) and the remaining special-purpose ranges, in their IPv4-mapped IPv6 form as well. Disable only in controlled environments where accessing local networks is required (`SSRF_PROTECTION_BLOCK_PRIVATE_NETWORKS=false`).
- **Defense in Depth**: Even with SSRF protection enabled, restrict outbound Security Group rules to only the necessary AWS service endpoints.
!!! tip "Network-layer complement: Route 53 Resolver DNS Firewall"
SSRF protection blocks by IP range — it doesn't know whether a public IP belongs to a malicious domain (malware, phishing, botnet C2). The Terraform module can close that gap: set `dns_firewall_enabled = true` to block outbound DNS resolution of known-malicious domains before a fetch ever happens. See [AWS Security Hub, GuardDuty & DNS Firewall Integration](#aws-security-hub-guardduty-dns-firewall-integration).
### :material-web-check: Host Header Validation
To protect against Host header injection and web cache poisoning, stdapi.ai can validate the `Host` header of incoming requests. When `TRUSTED_HOSTS` is not configured, no Host header validation is performed.
- **Recommended Approach**: Use **AWS ALB host-based routing rules** to reject invalid Host headers before they reach the application. This is more performant and centrally managed.
- **Application Validation**: Use `TRUSTED_HOSTS` to define a list of approved hostnames (supports wildcards like `*.example.com`). Requests with non-matching headers are rejected with an HTTP 400 error.
- **Probe Impact**: Validation covers `/health` too. The container image's health probe derives its `Host` header from `TRUSTED_HOSTS` and stays green on its own, but a load balancer health check sends the target's IP address as the `Host` and is rejected — see [`TRUSTED_HOSTS`](operations_configuration.md#trusted-hosts) before enabling it behind an ALB or NLB.
### :material-web: Browser Security (CORS)
If your API is accessed directly from web browsers, Cross-Origin Resource Sharing (CORS) must be configured to prevent unauthorized cross-site requests. When `CORS_ALLOW_ORIGINS` is not configured, the CORS middleware is not enabled and browser cross-origin requests are blocked by default.
- **Principle of Least Privilege**: Only allow specific, trusted domains using `CORS_ALLOW_ORIGINS`.
- **Offload to Infrastructure**: For production, consider managing CORS at the **AWS ALB** or **API Gateway** level for better performance and to keep your application logic clean.
### :material-swap-horizontal: Proxy Headers
When running behind a load balancer (ALB) or reverse proxy, stdapi.ai can be configured to trust `X-Forwarded-*` headers (`X-Forwarded-For`, `X-Forwarded-Proto`, `X-Forwarded-Port`) to correctly identify the original client IP and protocol. When enabled, the real client IP appears in request logs instead of the proxy's IP.
- **Security Warning**: Only enable `ENABLE_PROXY_HEADERS` when the service is deployed behind a **trusted proxy**. Enabling this in a publicly exposed environment without a proxy allows attackers to spoof their IP address by sending custom headers.
### :material-resize: Request Size & Resource Limits
Large or numerous inputs are a denial-of-service vector: a single request can carry hundreds of megabytes of inline base64 content, reference many remote files, or stream for a long time. stdapi.ai provides application-level controls, but the most effective size limit is enforced at the edge, before the request reaches the application.
**Application-level controls:**
- **Inline & downloaded input size**: [`MAX_INPUT_FILE_SIZE`](operations_configuration.md#max-input-file-size) caps the bytes of any single file loaded into memory for model input (base64, `data:` URIs, and HTTP(S)/S3 sources read for the model). Requests over the limit are rejected with HTTP 413 before the content is fully decoded or downloaded — a spoofed `Content-Length` cannot bypass it. Disabled by default; set a value aligned with your largest expected input when exposed to untrusted clients.
- **Concurrent downloads**: [`MAX_CONCURRENT_INPUT_DOWNLOADS`](operations_configuration.md#max-concurrent-input-downloads) bounds how many remote inputs a single request fetches at once, preventing socket/memory exhaustion and SSRF amplification from a request carrying many URLs.
!!! warning "Application limits do not cap the total request body"
`MAX_INPUT_FILE_SIZE` applies to individual **file** inputs. It does not bound the overall request body: a large prompt sent as plain text (for example a huge `messages` array rather than a file) is still received and parsed in full before any application limit applies. Cap the total body size at the edge (below).
**Edge body-size limit (recommended):**
The overall request body should be capped at the network edge, before it is buffered by the application — this is the primary protection against memory exhaustion from oversized bodies.
- **AWS WAF**: the mechanism for an ALB-fronted deployment. Add a rule with a `SizeConstraintStatement` on the request `BODY` and set **oversize handling to block**, so bodies larger than the inspectable size are rejected outright. The Terraform module's WAF (`alb_waf_enabled=true`) is the place to add this rule. AWS WAF inspects only a portion of the body (8 KB by default, up to 64 KB on ALB), so rely on the oversize-handling block action rather than attempting to inspect the full payload.
- **ALB**: does not enforce a request body size limit on its own — pair it with AWS WAF for that control.
- **Amazon API Gateway**: if you front the service with API Gateway instead of an ALB, note its hard **10 MB** payload limit, which may be too small for large multimodal or long-context requests.
!!! note "Sizing the edge limit is inherently hard for AI workloads"
There is no universally correct body-size cap: legitimate prompts are large and keep growing as model context windows expand, and multimodal inputs (images, audio, documents) push request bodies higher still. Set the edge limit **generously** — large enough for your largest legitimate request, small enough to stop obvious abuse — and rely on the finer-grained controls below for cost and abuse management rather than a single tight byte limit.
**Duration & throughput controls:**
- **Output length**: bound generated output with the API's `max_tokens` / `max_output_tokens` parameters; large client-supplied defaults can be constrained at your application or gateway layer.
- **Upstream timeout**: [`AI_RESPONSE_TIMEOUT`](operations_configuration.md#ai-response-timeout) closes stalled model connections so a slow or hung generation does not hold resources indefinitely.
- **Rate limiting & concurrency**: use WAF rate limiting (`alb_waf_rate_limit`, see [Security Best Practices](#security-best-practices)) to bound requests per client, complementing the per-request download concurrency cap above.
---
## :material-shield-star: AWS Security Hub, GuardDuty & DNS Firewall Integration
!!! success "Recommended: deploy via the Terraform module"
The [stdapi-ai Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) is built against the **AWS Security Hub Foundational Security Best Practices (FSBP)** standard and passes a large share of its applicable controls **by default** — no extra configuration required. A set of opt-in variables closes the remaining gaps for organizations with stricter compliance requirements, and the same dedicated VPC also enables native **GuardDuty Runtime Monitoring** and **Route 53 Resolver DNS Firewall** (see below). This makes the module the fastest path to a Security Hub-compliant, threat-monitored deployment; manual (non-Terraform) deployments must implement each control yourself.
The module composes three child modules — [VPC](https://github.com/JGoutin/terraform-aws-vpc), [KMS](https://github.com/JGoutin/terraform-aws-kms-key), and [ECS Fargate](https://github.com/JGoutin/terraform-aws-ecs-fargate) — plus its own ALB/WAF/S3/IAM resources. Each repository's README documents every relevant control — pass, fail, conditional (depends on your configuration), or not applicable — with remediation notes where relevant.
| Module | Key controls covered by default | Extra options to improve compliance |
|---------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------|
| [stdapi-ai (root)](https://github.com/stdapi-ai/terraform-aws-stdapi-ai#security-hub-controls) | S3 encryption/versioning/public-access block, least-privilege IAM, ALB access logging | `alb_waf_enabled`, `alb_certificate_arn` / `alb_domain_name` (HTTPS redirect), `deletion_protection` |
| [VPC](https://github.com/JGoutin/terraform-aws-vpc#security-hub-controls) | EC2.2 (default security group lockdown), EC2.6 (flow logs, 365-day retention), EC2.15/21/48, IAM.24 | `compliance_vpc_endpoints_enabled`, `guardduty_vpc_endpoint_enabled`, `dns_firewall_enabled` — **dedicated VPC only, see below** |
| [KMS](https://github.com/JGoutin/terraform-aws-kms-key#security-hub-controls) | KMS.3 (deletion window), KMS.4 (automatic rotation, hardcoded on), KMS.5 (key policy) | None — rotation and deletion window are hardcoded, not configurable |
| [ECS Fargate](https://github.com/JGoutin/terraform-aws-ecs-fargate#security-hub-controls) | ECS.\*, EFS.1-8 (POSIX user enforcement, native backups), CloudWatch.15-17, Backup.1-5, IAM.1/24 | `mount_points_efs_backup_enable`, `mount_points[].efs_posix_user`, `alarms_enabled` |
All four modules also accept a `tags` variable to propagate custom resource tags — itself relevant to Security Hub controls IAM.24 and EC2.48 when a tagging policy is enforced on your account.
### :material-radar: GuardDuty Runtime Monitoring
stdapi.ai's ECS tasks support **[Amazon GuardDuty Runtime Monitoring](https://docs.aws.amazon.com/guardduty/latest/ug/runtime-monitoring.html)** natively. Set `guardduty_vpc_endpoint_enabled = true` and the Terraform module creates the dedicated `guardduty-data` interface VPC endpoint the GuardDuty agent needs to report findings — keeping that traffic off the internet and off any NAT gateway. This isn't a Security Hub control, so it stays disabled by default; enable it whenever GuardDuty Runtime Monitoring is turned on for your account, since provisioning the endpoint through the module guarantees correct subnet placement.
### :material-dns: Route 53 Resolver DNS Firewall
stdapi.ai fetches user-supplied URLs as multimodal content references (images, documents, audio) — see [SSRF Protection](#ssrf-protection) for the application-level defense (IP-based blocklisting of loopback/link-local/private ranges). **Route 53 Resolver DNS Firewall** adds a network-layer control on top of that: set `dns_firewall_enabled = true` and the Terraform module creates a DNS Firewall rule group on the dedicated VPC that blocks or alerts on outbound DNS queries to known-malicious domains, closing the gap SSRF protection doesn't cover — a public IP address that resolves from a domain associated with malware, phishing, or botnet command-and-control.
- `dns_firewall_managed_domain_list_ids` (default: `null`, which resolves automatically to the [AWS Managed](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-dns-firewall-managed-domain-lists.html) Aggregate Threat List ID for your deployment region — no AWS CLI call or extra IAM permissions needed) selects which managed domain lists to enforce; `dns_firewall_action` controls whether matches are `BLOCK`ed, `ALERT`ed on, or explicitly `ALLOW`ed.
- `dns_firewall_advanced_enabled` turns on **DNS Firewall Advanced** (additional AWS cost), which detects domain generation algorithm (DGA) and DNS tunneling activity — patterns a static domain list can't catch. Tune sensitivity with `dns_firewall_advanced_confidence_threshold` (`LOW` / `MEDIUM` / `HIGH`).
!!! warning "`compliance_vpc_endpoints_enabled`, `guardduty_vpc_endpoint_enabled`, and `dns_firewall_enabled` require the module's dedicated VPC"
All three variables attach resources (interface endpoints, or a DNS Firewall rule group) to the VPC the Terraform module creates for you. They have **no effect** — `dns_firewall_enabled` cannot even be set to `true` and fails validation — if you instead pass your own `subnet_ids` to integrate with existing network infrastructure (see [Integration with Existing Infrastructure](operations_deploy_advanced.md#integration-with-existing-infrastructure)). In that case the module never creates a VPC, and you are responsible for provisioning any interface endpoints or DNS Firewall rules your compliance or monitoring tooling needs (ECR, SSM, `guardduty-data`, Resolver rule groups, etc.) directly in your own VPC.
---
## :material-shield-bug: Prompt Injection
Prompt injection is an application-level security concern: just as AWS secures the database engine but customers are responsible for preventing SQL injection, AWS secures the Bedrock infrastructure but **your application is responsible for preventing prompt injection**. For full details, see the [Amazon Bedrock prompt injection security documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-injection.html).
Key practices recommended by AWS:
- **Input validation** — validate and sanitize all user input before passing it to the model; remove or escape special characters and enforce expected formats.
- **Secure coding** — avoid string concatenation for building prompts; apply the principle of least privilege when granting access to resources.
- **System prompt scoping** — when using system prompts, clearly define what the model can and cannot do. Newer models differentiate between system and user prompts — check the model provider's documentation for model-specific guidance.
- **Security testing** — regularly test for prompt injection using penetration testing, static analysis, and dynamic application security testing (DAST).
- **Keep dependencies updated** — monitor AWS security bulletins and keep the Bedrock SDK and libraries up to date.
!!! tip "Use Amazon Bedrock Guardrails"
The most effective mitigation available within stdapi.ai is to configure an **Amazon Bedrock Guardrail**. Guardrails include a dedicated prompt attack detection layer and can be applied **per request** (via request headers) or **globally for the entire server** (via environment variables). See the [Bedrock Guardrails configuration section](operations_configuration.md#bedrock-guardrails) for setup instructions, and [Detect prompt attacks with Amazon Bedrock Guardrails](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-prompt-attack.html) for the upstream AWS documentation.
!!! warning "stdapi.ai does not include built-in prompt injection protection"
stdapi.ai passes user prompts to the model as-is and does not apply any custom filtering or sanitization against prompt injection. Protecting against this risk is the responsibility of your application and infrastructure.
---
## :material-shield-check: Security Best Practices
**Network Isolation:**
- :material-check: **Deploy in private subnets** — without direct internet access.
- :material-check: **Restrict security group inbound rules** — allow traffic to the stdapi.ai service only from the ALB or API Gateway security group on the application port.
- :material-check: **Apply strict egress rules** — limit outbound traffic to only the required AWS services (Bedrock, S3, etc.).
- :material-check: **Use VPC endpoints** — the Terraform module creates VPC endpoints for all required AWS services (`vpc_endpoints_allowed=true` by default), keeping all traffic to Bedrock, S3, SSM, CloudWatch Logs, and AI services off the public internet with no NAT Gateway required.
**WAF & Rate Limiting:**
- :material-check: **Enable AWS WAF** — the Terraform module supports WAF via `alb_waf_enabled=true`. When enabled, it activates the AWS Managed Core Rule Set (SQLi, XSS, common exploits), Linux Rule Set, and IP Reputation List (known malicious IPs).
- :material-check: **Configure rate limiting** — set `alb_waf_rate_limit` to limit the number of requests per IP per 5-minute window, protecting against abuse and quota exhaustion.
- :material-check: **Block anonymous IPs** — set `alb_waf_block_anonymous_ips=true` to block known VPNs, open proxies, and Tor exit nodes.
- :material-check: **Cap request body size** — add an AWS WAF `SizeConstraintStatement` on the request body (oversize handling set to block) to reject oversized bodies at the edge; see [Request Size & Resource Limits](#request-size-resource-limits).
**Secrets Management:**
- :material-check: **Use SSM Parameter Store or Secrets Manager** — always use encrypted secret storage for production API keys; never pass keys as plain environment variables.
- :material-check: **Rotate API keys** — Secrets Manager supports [automated secret rotation](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html) via Lambda; note that stdapi.ai reads the key once at startup, so a container restart is required after rotation.
- :material-check: **Share the Realtime signing key** — the [ephemeral client secrets](api_openai_realtime.md#ephemeral-client-secrets) minted for browser clients are signed, not stored, so every instance must sign with the same key. It is derived from the deployment's API key by default; a deployment running with no API key at all falls back to a per-process random key and must set [`REALTIME_CLIENT_SECRET_KEY`](operations_configuration.md#realtime-client-secret-key) instead. Treat it as a secret and store it the same way: anyone holding it can mint a client secret this deployment accepts.
!!! tip "Eliminate key rotation with AWS native auth"
When using API key authentication, rotation requires updating SSM/Secrets Manager and performing a rolling ECS task replacement. This is predictable but adds a deployment step.
To avoid key rotation entirely, use [**Amazon Cognito user pool tokens**](#amazon-cognito-user-pool-tokens), or **OIDC / Cognito / IAM Identity Center** via the ALB — there are no API keys to rotate, and access is granted and withdrawn in your identity provider without touching the deployment. With tokens validated by stdapi.ai, withdrawing access takes effect when the caller's current token expires: keep the app client's [access-token validity](#amazon-cognito-user-pool-tokens) short.
**IAM & Access Control:**
- :material-check: **Apply least-privilege IAM permissions** — ensure the ECS Task Role has only the [required permissions](operations_configuration.md#iam-permissions) needed for operation.
!!! info "Terraform Module"
The Terraform module enforces security defaults out of the box: **least-privilege IAM** roles scoped to the minimum permissions required by each ECS task, **KMS-encrypted** SSM parameters, log groups, and S3 buckets, and **network isolation** with ECS tasks in private subnets and Security Group rules that allow only ALB-to-service traffic.
---
## :material-shield-sync: Encryption in Transit
To ensure data protection during transmission, stdapi.ai supports several encryption-in-transit configurations.
| Mode | TLS scope | Config required | Typical use case |
|---|---|---|---|
| **Standard** | Entry point only | None | Default for most AWS deployments |
| **End-to-End** | Entry point → container | `GRANIAN_SSL_CERTIFICATE` + `GRANIAN_SSL_KEYFILE` | Compliance-mandated full-path encryption |
| **mTLS at Container** | Entry point → container, mutual | Above + `GRANIAN_SSL_CA` + `GRANIAN_SSL_CLIENT_VERIFY` | Mutual authentication between ALB and container |
| **mTLS at Entry Point** | Client → entry point, mutual | ALB or API Gateway configuration | Client certificate enforcement before traffic reaches the application |
### :material-shield-half-full: Standard
This is the default configuration when no additional encryption settings are applied. TLS is terminated at the entry point (ALB or API Gateway). Traffic between the entry point and the stdapi.ai service travels over HTTP within the private VPC, secured by mandatory Security Group rules. This is the most common configuration for AWS deployments.
!!! info "Terraform Module"
The Terraform module configures the ALB with the `ELBSecurityPolicy-TLS13-1-2-Res-PQ-2025-09` security policy by default (`alb_ssl_policy`), enforcing a minimum of TLS 1.2 and enabling TLS 1.3 with post-quantum hybrid key exchange. See [Data Sovereignty & Compliance](operations_compliance.md) for details on ALB TLS configuration and post-quantum key exchange support.
### :material-lock-check: End-to-End
If your compliance requirements mandate encryption all the way to the container, you can enable TLS between the load balancer and the container.
- **Native Support**: The provided container images run [Granian](https://github.com/emmett-framework/granian) natively. TLS can be enabled directly by setting the following Granian environment variables:
- `GRANIAN_SSL_CERTIFICATE`: Path to the SSL certificate file (e.g., `/etc/ssl/certs/server.crt`).
- `GRANIAN_SSL_KEYFILE`: Path to the SSL private key file (**PKCS#8 format only**, e.g., `/etc/ssl/private/server.key`).
- `GRANIAN_SSL_KEYFILE_PASSWORD`: (Optional) Password for the private key file.
- `GRANIAN_SSL_PROTOCOL_MIN`: (Optional) Minimum supported TLS version — `tls1.2` or `tls1.3` (defaults to `tls1.3`).
- **Mutual TLS — Container**: As an advanced form of end-to-end encryption, you can also require the upstream caller (e.g., the ALB) to present a certificate to the container.
- `GRANIAN_SSL_CA`: Path to the CA certificate bundle used to verify client certificates.
- `GRANIAN_SSL_CLIENT_VERIFY`: Set to `true` to enable client certificate verification.
- **ECS Certificate Mounting**: When running on AWS ECS, certificate and key files must be mounted into the container. This is typically achieved using **Amazon EFS** volumes or by baking the certificates into a custom image (not recommended for secrets).
- **Certificate Management**: Use ACM Private CA or a self-signed certificate for the internal service. Note that the ALB does not validate the backend certificate by default; it only ensures the connection is encrypted.
- **Sidecar Proxy**: Alternatively, you can run a sidecar container (like Nginx or Envoy) alongside stdapi.ai to handle TLS termination locally.
!!! info "Terraform Module Support"
The provided Terraform module **does not currently support** automatic configuration of end-to-end encryption. You will need to manually configure the ALB target group for HTTPS and manage certificate provisioning.
### :material-certificate-outline: Mutual TLS — Client to Entry Point
For environments requiring mutual authentication between the final client and the entry point:
- **Offloaded mTLS**: mTLS can be enabled at the **ALB** (using Application Load Balancer Mutual TLS) or **API Gateway** (using Mutual TLS authentication). In this setup, the entry point validates the client certificate before forwarding the request.
- **Service Mesh / Sidecar**: You can implement mTLS between internal services using a Service Mesh (like AWS App Mesh) or by using a sidecar proxy (like Envoy) that manages certificate rotation and verification.
---
## :material-arrow-right: Next Steps
- :material-cog: [**Configuration Reference**](operations_configuration.md) — Detailed environment variable list
- :material-shield-lock: [**Data Sovereignty & Compliance**](operations_compliance.md) — How we protect your data
- :material-server-network: [**Advanced Deployment**](operations_deploy_advanced.md) — Terraform infrastructure examples
---
## IAM Permissions
Source: https://stdapi.ai/md/operations_iam_permissions.md
# :material-shield-key: IAM Permissions
stdapi.ai requires specific AWS IAM permissions to access Amazon Bedrock models and other AWS services. The exact permissions needed depend on which features you enable.
!!! tip "Building Your Policy"
Combine the permission statements below based on the features you need. At minimum, you need the **Bedrock** permissions. Add statements for S3, TTS, STT, and other features as required by your deployment. Only include the statements you need — start with the Bedrock permissions and add others as required (least privilege).
!!! warning "One policy may not hold all of these"
A customer managed policy is capped at **6,144 characters**, a limit AWS does not raise through Service Quotas, and whitespace does not count toward it. Enabling most of the features on this page exceeds that in a single document — `CreatePolicy` then fails with `LimitExceeded: Cannot exceed quota for PolicySize: 6144`, naming no statement.
Attach several policies to the role instead of widening actions to save characters. Splitting by service keeps each one readable and reviewable; the Terraform module ships two for exactly this reason, one for Amazon Bedrock and one for the supporting services. A role takes up to 10 managed policies by default.
!!! info "Terraform Module"
The official [stdapi-ai Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) provisions the ECS task role with the required permissions automatically. This reference is for custom deployments and policy auditing.
!!! warning "Multi-Region Failover and Region-Scoped Policies"
By default, Amazon Bedrock **and** the other AWS AI services (Polly, Transcribe, Comprehend, Translate) are called in every region listed in [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions), failing over from one to the next on throttling, quota, or availability errors. See [Other AWS Services Failover](operations_resilience.md#other-aws-services-failover).
The statements on this page use region-agnostic resources, so they work as-is. However, any `aws:RequestedRegion` condition — in the policy itself, a permissions boundary, or a service control policy — must allow **all** configured regions, otherwise failover silently fails and requests error out once the first region is unavailable.
---
## :material-aws: Bedrock (Required) { #bedrock-iam }
**Environment Variables**: Always required
These permissions are mandatory for stdapi.ai to discover and invoke Amazon Bedrock models:
??? example "Bedrock IAM Policy Statements"
```json
{
"Sid": "BedrockModelInvoke",
"Effect": "Allow",
"Action": [
"bedrock:CountTokens",
"bedrock:GetAsyncInvoke",
"bedrock:InvokeGuardrailChecks",
"bedrock:InvokeModel",
"bedrock:InvokeModelWithBidirectionalStream",
"bedrock:InvokeModelWithResponseStream",
"bedrock:InvokeTool",
"bedrock:Rerank"
],
"Resource": "*"
},
{
"Sid": "BedrockAsyncInvokeTagging",
"Effect": "Allow",
"Action": [
"bedrock:TagResource"
],
"Resource": "arn:aws:bedrock:*:*:async-invoke/*"
},
{
"Sid": "BedrockModelDiscovery",
"Effect": "Allow",
"Action": [
"bedrock:ListFoundationModels",
"bedrock:GetFoundationModelAvailability",
"bedrock:ListProvisionedModelThroughputs",
"bedrock:ListInferenceProfiles"
],
"Resource": "*"
}
```
!!! note "Asynchronous Invocations"
`bedrock:GetAsyncInvoke` and `bedrock:TagResource` (on `arn:aws:bedrock:*:*:async-invoke/*`) serve Bedrock asynchronous invocations, used by video generation models and asynchronous embedding models such as TwelveLabs Marengo (`twelvelabs.marengo-embed-*`). They can be dropped if your deployment uses none of these models. `bedrock:ListAsyncInvokes` and `bedrock:ListTagsForResource` are **not** part of this core set — they are only needed for video job listing (see [Video Generation](#video-generation-optional)).
!!! note "Bidirectional Streaming"
`bedrock:InvokeModelWithBidirectionalStream` serves any model invoked over a persistent, two-way connection: live audio transcription with Amazon Nova Sonic (see [Speech-to-Text](#speech-to-text-optional)) and the [Realtime API](api_openai_realtime.md) (`POST /v1/realtime/client_secrets`, `WS /v1/realtime`) alike. No route-specific action exists for either — it is already part of the core Bedrock policy above.
---
## :material-storefront: Bedrock Marketplace Auto-Subscribe (Optional) { #bedrock-marketplace-auto-subscribe-iam }
**Environment Variables**: [`AWS_BEDROCK_MARKETPLACE_AUTO_SUBSCRIBE`](operations_configuration.md#bedrock-marketplace-auto-subscribe)
Required only if you want models sold as third-party AWS Marketplace listings to be usable without subscribing to each one by hand (`AWS_BEDROCK_MARKETPLACE_AUTO_SUBSCRIBE=true`, which is the default). The server never calls `Subscribe` itself: it keeps a listing with no agreement in the catalogue, and AWS creates the subscription under this role on the first invocation. It applies to whichever models AWS sells that way — see [Which Models Are Which](operations_cost_management.md#which-models-are-which); models billed as ordinary Amazon Bedrock usage need none of these permissions.
??? example "Bedrock Marketplace Auto-Subscribe IAM Policy Statement"
```json
{
"Sid": "BedrockMarketplaceAutoSubscribe",
"Effect": "Allow",
"Action": [
"aws-marketplace:Subscribe",
"aws-marketplace:ViewSubscriptions"
],
"Resource": "*"
}
```
!!! warning "Cost Consideration"
Automatic marketplace subscriptions may incur costs. Review AWS Marketplace pricing for individual models before enabling this feature, or set `AWS_BEDROCK_MARKETPLACE_AUTO_SUBSCRIBE=false` to require manual marketplace subscription.
---
## :material-storefront: AWS Marketplace Metering (AWS Marketplace Image Only) { #aws-marketplace-metering }
**Environment Variables**: none (always active on the AWS Marketplace image)
Required only for the **AWS Marketplace image** — not the community image. At startup it registers hourly usage with AWS Marketplace Metering on ECS, EKS, and Fargate; an `AccessDenied` error aborts startup.
??? example "AWS Marketplace Metering IAM Policy Statement"
```json
{
"Sid": "MarketplaceRegisterUsage",
"Effect": "Allow",
"Action": [
"aws-marketplace:RegisterUsage"
],
"Resource": "*"
}
```
---
## :material-directions-fork: Bedrock Inference Profiles, Prompt Routers and Prompt Management (Optional) { #bedrock-inference-profiles-and-prompt-routers-optional }
**Environment Variables**: [`AWS_BEDROCK_ALLOW_CROSS_REGION_INFERENCE_PROFILE_ARN`](operations_configuration.md#bedrock-allow-cross-region-profile-arn), [`AWS_BEDROCK_ALLOW_APPLICATION_INFERENCE_PROFILE_ARN`](operations_configuration.md#bedrock-allow-application-profile-arn), [`AWS_BEDROCK_ALLOW_PROMPT_ROUTER_ARN`](operations_configuration.md#bedrock-allow-prompt-router-arn), [`AWS_BEDROCK_ALLOW_PROMPT_ARN`](operations_configuration.md#bedrock-allow-prompt-arn), [`AWS_BEDROCK_MODEL_ARN_MAPPING`](operations_configuration.md#bedrock-model-arn-mapping)
Required only if you enable ARN-based routing features that allow users to pass inference profile, prompt router or Prompt Management prompt ARNs directly, or if you configure server-side ARN mappings.
??? example "Bedrock Inference Profiles, Prompt Routers and Prompt Management IAM Policy Statements"
```json
{
"Sid": "BedrockInferenceProfilesAndPromptRouters",
"Effect": "Allow",
"Action": [
"bedrock:GetInferenceProfile",
"bedrock:GetPromptRouter"
],
"Resource": "*"
},
{
"Sid": "BedrockPromptManagement",
"Effect": "Allow",
"Action": [
"bedrock:GetPrompt",
"bedrock:RenderPrompt"
],
"Resource": "arn:aws:bedrock:*:*:prompt/*"
}
```
`bedrock:GetPrompt` resolves the model bound to the prompt variant; `bedrock:RenderPrompt` is required because the prompt ARN is then sent to Bedrock as the invocation `modelId`.
!!! note "When to Include"
Add these permissions when:
- `AWS_BEDROCK_ALLOW_CROSS_REGION_INFERENCE_PROFILE_ARN=true`
- `AWS_BEDROCK_ALLOW_APPLICATION_INFERENCE_PROFILE_ARN=true`
- `AWS_BEDROCK_ALLOW_PROMPT_ROUTER_ARN=true`
- `AWS_BEDROCK_ALLOW_PROMPT_ARN=true` (`bedrock:GetPrompt` and `bedrock:RenderPrompt` only)
- `AWS_BEDROCK_MODEL_ARN_MAPPING` is configured with any mappings
---
## :material-shield-check: Bedrock Guardrails (Optional)
**Environment Variables**: [`AWS_BEDROCK_GUARDRAIL_IDENTIFIER`](operations_configuration.md#aws-bedrock-guardrail-identifier), [`AWS_BEDROCK_GUARDRAIL_VERSION`](operations_configuration.md#aws-bedrock-guardrail-version)
Required if you configure Bedrock Guardrails for content filtering, use the `moderation` request parameter, or select a guardrail on the [Moderations API](api_openai_moderations.md) (without a guardrail, that API falls back to [Comprehend toxicity moderation](#comprehend-moderation)). See the [Bedrock Guardrails](operations_configuration.md#bedrock-guardrails) configuration section.
??? example "Bedrock Guardrails IAM Policy Statement"
```json
{
"Sid": "BedrockGuardrails",
"Effect": "Allow",
"Action": [
"bedrock:ApplyGuardrail"
],
"Resource": "arn:aws:bedrock:*:*:guardrail/*"
}
```
---
## :material-database-lock: Bedrock Session Storage (Optional)
**Environment Variables**: none (enabled by the `store=true` request parameter; see [Bedrock Session Storage](operations_configuration.md#bedrock-session-storage-optional) configuration)
Required if clients use `store=true` on the [Responses](api_openai_responses.md#stored-responses) or [Chat Completions](api_openai_chat_completions.md#stored-chat-completions) APIs, or the [Conversations](api_openai_conversations.md) API, all of which persist state in Amazon Bedrock sessions.
??? example "Bedrock Session Storage IAM Policy Statement"
```json
{
"Sid": "BedrockSessionStorage",
"Effect": "Allow",
"Action": [
"bedrock:CreateSession",
"bedrock:GetSession",
"bedrock:UpdateSession",
"bedrock:CreateInvocation",
"bedrock:PutInvocationStep",
"bedrock:ListInvocations",
"bedrock:ListInvocationSteps",
"bedrock:GetInvocationStep",
"bedrock:EndSession",
"bedrock:DeleteSession",
"bedrock:TagResource",
"bedrock:ListTagsForResource"
],
"Resource": "arn:aws:bedrock:*:*:session/*"
},
{
"Sid": "BedrockSessionListing",
"Effect": "Allow",
"Action": "bedrock:ListSessions",
"Resource": "*"
}
```
`bedrock:ListSessions` serves the stored chat completions listing endpoint (`GET /v1/chat/completions`); the account-level `ListSessions` action does not support resource scoping. `bedrock:GetSession` is used on deletion and `bedrock:ListTagsForResource` on both deletion and listing, to check that a stored object belongs to the API it is requested from. `bedrock:UpdateSession` serves the conversation metadata update (`POST /v1/conversations/{conversation_id}`) only.
Add `kms:Decrypt` and `kms:GenerateDataKey` on the key when [`AWS_BEDROCK_SESSION_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-bedrock-session-encryption-key-arn) is configured.
---
## :material-layers-triple: Bedrock Mantle (Optional) { #bedrock-mantle-iam }
**Environment Variables**: [`AWS_BEDROCK_MANTLE_ENABLED`](operations_configuration.md#bedrock-mantle-enabled)
Required for [`AWS_BEDROCK_MANTLE_ENABLED`](operations_configuration.md#bedrock-mantle-enabled) (enabled by default), which exposes models served by the Amazon Bedrock Mantle endpoint (OpenAI GPT, xAI Grok, Google Gemma, and more). Without these permissions the server still starts normally: Mantle models are not listed and a warning is logged.
??? example "Bedrock Mantle IAM Policy Statements"
```json
{
"Sid": "BedrockMantleInference",
"Effect": "Allow",
"Action": [
"bedrock-mantle:CreateInference",
"bedrock-mantle:GetInference",
"bedrock-mantle:DeleteInference",
"bedrock-mantle:ListModels",
"bedrock-mantle:GetModel",
"bedrock-mantle:CancelInference",
"bedrock-mantle:CountTokens"
],
"Resource": "arn:aws:bedrock-mantle:*:*:project/*"
},
{
"Sid": "BedrockMantleBearerToken",
"Effect": "Allow",
"Action": "bedrock-mantle:CallWithBearerToken",
"Resource": "*"
}
```
`bedrock-mantle:CallWithBearerToken` authorizes the short-term bearer tokens the server derives from its AWS credential chain; it does not support resource scoping.
`bedrock-mantle:CountTokens` serves [`/anthropic/v1/messages/count_tokens`](api_anthropic_messages.md) for a Mantle-served model: Amazon Bedrock's own `CountTokens` accepts Anthropic models only, so the count is proxied to the Mantle endpoint. Without it that route answers `500`.
---
## :material-web: Web Search (Optional) { #web-search-iam }
**Environment Variables**: none (enabled by a `web_search` tool in a request)
Required for the built-in [web search tool](api_openai_responses.md#openai-gpt-web-search) on the OpenAI GPT-5.x family, whichever settings are in use. Each action is authorized only when the model actually attempts that call, and a denied call does not fail the request: AWS documents the model continuing with the information it already has and telling you it could not retrieve enough current information ([Identity and access management for Web Search](https://docs.aws.amazon.com/bedrock/latest/userguide/security-web-search.html)).
!!! warning "A missing web search permission produces no error and no server log entry"
The denial is handled inside the model call, so the request succeeds with a normal answer: the server sees nothing to report, and the response is indistinguishable from the model deciding it did not need to search. When answers never cite a source, check these permissions (and the Region the call was served in) before suspecting the model. AWS CloudTrail records the denied `bedrock-websearch` calls.
Add `bedrock-websearch:ExternalWebAccess` on top when a request can reach external web access — that is, when [`AWS_BEDROCK_EXTERNAL_WEB_ACCESS`](operations_configuration.md#bedrock-external-web-access) is enabled, or when [`AWS_BEDROCK_ALLOW_EXTERNAL_WEB_ACCESS_OVERRIDE`](operations_configuration.md#bedrock-allow-external-web-access-override) lets a client ask for it per request. Leaving it out is what keeps every search inside the AWS boundary.
??? example "Web Search IAM Policy Statement"
```json
{
"Sid": "BedrockWebSearch",
"Effect": "Allow",
"Action": [
"bedrock-websearch:InvokeSearch",
"bedrock-websearch:InvokeFetch"
],
"Resource": "*"
}
```
`InvokeSearch` and `InvokeFetch` are authorized on AWS-owned tool resources (`arn:aws:bedrock-websearch::aws:tool/`), which `"*"` covers; the AWS managed policies instead scope them to `arn:aws:bedrock-websearch:*:*:*`, and either form grants the same searches. `ExternalWebAccess` is a permission-only action with no resource of its own, so grant it with `"*"`. Web search runs in the Region that served the model call, in the [Regions where the tool is offered](api_openai_responses.md#openai-gpt-web-search); scope the statement with an `aws:RequestedRegion` condition to pin which Regions may run searches. See [Identity and access management for Web Search](https://docs.aws.amazon.com/bedrock/latest/userguide/security-web-search.html) and [Actions, resources, and condition keys](https://docs.aws.amazon.com/service-authorization/latest/reference/list_bedrock-websearch.html).
---
## :material-database: S3 File Storage (Optional)
**Environment Variables**: [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket), [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets)
Required for storing generated images, audio files, documents, and videos. See [Storage Configuration](operations_configuration.md#storage-configuration) for bucket setup details.
??? example "S3 File Storage IAM Policy Statements"
```json
{
"Sid": "S3FileStorage",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:PutObjectTagging",
"s3:GetObject",
"s3:DeleteObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": "arn:aws:s3:::AWS_S3_BUCKET_VALUE/*"
},
{
"Sid": "S3FileStorageList",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": "arn:aws:s3:::AWS_S3_BUCKET_VALUE"
}
```
!!! info "Replace Bucket Name"
Replace `AWS_S3_BUCKET_VALUE` with the value of your [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) environment variable. Repeat both statements for each [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets) bucket — they serve video generation, asynchronous embeddings (TwelveLabs Marengo, Amazon Nova), [large attachments](features.md#attachment-size) on any multimodal route, and [Speech-to-Text](#speech-to-text-optional) failover, and the Files API looks up objects across every configured bucket.
!!! note "Multipart Uploads"
Large files are uploaded and copied with the multipart API. Its `CreateMultipartUpload`, `UploadPart`, `UploadPartCopy`, and `CompleteMultipartUpload` operations have no dedicated IAM actions — they are authorized by `s3:PutObject` — which is why only the abort and listing actions appear above.
!!! note "Cross-Region Access"
Bucket ARNs are region-agnostic, so one statement per bucket covers every region. When a request fails over to another region, the server copies the object server-side through the destination region's S3 endpoint, reading the source bucket from there. Grant these actions on the source and destination buckets alike, and on the KMS key of each encrypted bucket.
**If your S3 bucket uses KMS encryption**, also add:
```json
{
"Sid": "KMSEncryptedBucket",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": "arn:aws:kms:REGION:ACCOUNT_ID:key/YOUR_KMS_KEY_ID",
"Condition": {
"StringEquals": {
"kms:ViaService": "s3.REGION.amazonaws.com"
}
}
}
```
!!! tip "KMS Security"
The `kms:ViaService` condition restricts KMS key usage to S3 service calls only, following AWS security best practices. Because the condition pins a single region, add one statement per region — with that region's key ARN and `s3.REGION.amazonaws.com` — when you use regional buckets with per-region keys.
---
## :material-magnify: Vector Stores (Optional) { #vector-stores-optional }
**Environment Variables**: [`AWS_S3_VECTORS_BUCKET`](operations_configuration.md#aws-s3-vectors-bucket), [`AWS_S3_VECTORS_REGION`](operations_configuration.md#aws-s3-vectors-region), [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket)
Required by the [Vector Stores API](api_openai_vector_stores.md). The indexed content lives in an [Amazon S3 vector bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors.html) you create; the stores' own records live in the general purpose bucket under [`AWS_S3_VECTOR_STORES_PREFIX`](operations_configuration.md#aws-s3-vector-stores-prefix) and are covered by the [S3 File Storage](#s3-file-storage-optional) statements above.
??? example "Vector Stores IAM Policy Statements"
```json
{
"Sid": "VectorStoreIndexes",
"Effect": "Allow",
"Action": [
"s3vectors:CreateIndex",
"s3vectors:DeleteIndex",
"s3vectors:PutVectors",
"s3vectors:GetVectors",
"s3vectors:QueryVectors",
"s3vectors:DeleteVectors"
],
"Resource": [
"arn:aws:s3vectors:REGION:ACCOUNT_ID:bucket/AWS_S3_VECTORS_BUCKET_VALUE",
"arn:aws:s3vectors:REGION:ACCOUNT_ID:bucket/AWS_S3_VECTORS_BUCKET_VALUE/index/*"
]
}
```
!!! info "Replace the Placeholders"
Replace `AWS_S3_VECTORS_BUCKET_VALUE` with your [`AWS_S3_VECTORS_BUCKET`](operations_configuration.md#aws-s3-vectors-bucket) value, `REGION` with [`AWS_S3_VECTORS_REGION`](operations_configuration.md#aws-s3-vectors-region), and `ACCOUNT_ID` with your account. The bucket ARN itself is needed for the index actions; the index ARN pattern covers the per-store indexes the gateway creates and deletes.
!!! note "The bucket is yours to create"
The gateway never creates or deletes the vector bucket, only the indexes inside it, so no bucket-level create or delete action is granted.
!!! note "Records live in the general purpose bucket"
Grant the [S3 File Storage](#s3-file-storage-optional) statements on [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) as well: the stores, their attached files and their batches are JSON objects there.
---
## :material-tray-arrow-down: Durable Vector Store Indexing (Optional) { #durable-vector-store-indexing }
**Environment Variables**: [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url)
Required to keep indexing a vector store file when the server that accepted it stops. The gateway both writes the work to the [Amazon SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) queue you create and reads it back, so it needs the producer and the consumer actions on that one queue.
??? example "Durable Vector Store Indexing IAM Policy Statements"
```json
{
"Sid": "VectorStoreIndexingQueue",
"Effect": "Allow",
"Action": [
"sqs:SendMessage",
"sqs:ReceiveMessage",
"sqs:DeleteMessage",
"sqs:ChangeMessageVisibility",
"sqs:GetQueueAttributes"
],
"Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:QUEUE_NAME"
}
```
!!! info "Replace the Placeholders"
`REGION`, `ACCOUNT_ID` and `QUEUE_NAME` are the three parts of your [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url). Grant this on the queue ARN itself — never on `*`.
!!! note "The queues are yours to create"
The gateway never creates, deletes or reconfigures a queue, so no `sqs:CreateQueue`, `sqs:DeleteQueue` or `sqs:SetQueueAttributes` is granted. `sqs:GetQueueAttributes` is read-only and is what lets the gateway honour your dead-letter queue's redrive policy.
!!! note "The dead-letter queue needs nothing"
Amazon SQS moves an exhausted message itself; the gateway never reads the dead-letter queue, so grant it nothing.
---
## :material-book-search: Knowledge Base Vector Stores (Optional) { #knowledge-base-vector-stores }
**Environment Variables**: [`AWS_BEDROCK_KNOWLEDGE_BASE_IDS`](operations_configuration.md#aws-bedrock-knowledge-base-ids)
Required to serve an [Amazon Bedrock knowledge base](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html) you created as a [vector store](api_openai_vector_stores.md#knowledge-base-stores). Grant one statement per allowlisted knowledge base, on its own ARN.
??? example "Knowledge Base Vector Stores IAM Policy Statements"
```json
{
"Sid": "BedrockKnowledgeBaseVectorStores",
"Effect": "Allow",
"Action": [
"bedrock:GetKnowledgeBase",
"bedrock:Retrieve",
"bedrock:ListDataSources",
"bedrock:IngestKnowledgeBaseDocuments",
"bedrock:ListKnowledgeBaseDocuments",
"bedrock:GetKnowledgeBaseDocuments",
"bedrock:DeleteKnowledgeBaseDocuments"
],
"Resource": "arn:aws:bedrock:REGION:ACCOUNT_ID:knowledge-base/KNOWLEDGE_BASE_ID"
}
```
!!! info "Replace the Placeholders"
Replace `REGION` with the first [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) entry, `ACCOUNT_ID` with your account, and `KNOWLEDGE_BASE_ID` with the knowledge base identifier — one ARN per entry of [`AWS_BEDROCK_KNOWLEDGE_BASE_IDS`](operations_configuration.md#aws-bedrock-knowledge-base-ids).
!!! note "No discovery action, deliberately"
`bedrock:ListKnowledgeBases` is **not** granted, and is not needed: the server never discovers knowledge bases it was not given. Only the identifiers in the allowlist are ever addressed, and any other one is answered as an unknown vector store.
!!! note "The knowledge base is yours to create"
The gateway never creates or deletes a knowledge base, so no create, update or delete action on the knowledge base itself is granted — only the documents of its data source.
!!! tip "Checked once at startup"
`bedrock:GetKnowledgeBase` — already needed to serve a store — is also called on every allowlisted entry at startup, to confirm each one is a kind the server can serve. No extra action is required for that check, and a role missing the action never stops the server: it starts with one warning per entry instead.
---
## :material-video: Video Generation (Optional) { #video-generation-optional }
**Environment Variables**: [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets)
Video generation itself runs on the core Bedrock asynchronous invocation permissions (`bedrock:InvokeModel`, `bedrock:GetAsyncInvoke`, `bedrock:TagResource`) plus [S3 File Storage](#s3-file-storage-optional) permissions on each regional bucket. The video job listing endpoint (`GET /v1/videos`) additionally requires:
??? example "Video Job Listing IAM Policy Statements"
```json
{
"Sid": "BedrockVideoJobListing",
"Effect": "Allow",
"Action": [
"bedrock:ListAsyncInvokes"
],
"Resource": "*"
},
{
"Sid": "BedrockVideoJobTags",
"Effect": "Allow",
"Action": [
"bedrock:ListTagsForResource"
],
"Resource": "arn:aws:bedrock:*:*:async-invoke/*"
}
```
The account-level `ListAsyncInvokes` action does not support resource scoping; `ListTagsForResource` reads the job metadata tags used to attribute listed jobs.
---
## :material-package-variant-closed: Batch Inference (Optional) { #batch-inference }
**Environment Variables**: [`AWS_BEDROCK_BATCH_ROLE_ARN`](operations_configuration.md#aws-bedrock-batch-role-arn), [`AWS_S3_BATCHES_PREFIX`](operations_configuration.md#aws-s3-batches-prefix), [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket), [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets)
The [Batch API](api_openai_batches.md) and the [Message Batches API](api_anthropic_batches.md) run on [Amazon Bedrock batch inference](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html), which needs **two** policies: the server's own, and a service role Amazon Bedrock assumes to read the requests and write the results.
### The Server's Policy
??? example "Batch Inference IAM Policy Statements"
```json
{
"Sid": "BedrockBatchJobs",
"Effect": "Allow",
"Action": [
"bedrock:CreateModelInvocationJob",
"bedrock:GetModelInvocationJob",
"bedrock:StopModelInvocationJob"
],
"Resource": "arn:aws:bedrock:*::model-invocation-job/*"
},
{
"Sid": "BedrockBatchPassRole",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam:::role/stdapi-ai-batch",
"Condition": {
"StringEquals": {"iam:PassedToService": "bedrock.amazonaws.com"}
}
}
```
Substitute `` with your AWS account ID, and the role ARN with the value of `AWS_BEDROCK_BATCH_ROLE_ARN`. The server also needs the [S3 File Storage](#s3-file-storage-optional) permissions on every bucket a batch may use.
### The Service Role
Create a role named by `AWS_BEDROCK_BATCH_ROLE_ARN` whose trust policy lets Amazon Bedrock assume it, scoped to your account and to batch jobs so it cannot be used by another account's jobs:
??? example "Batch Service Role Trust Policy"
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {"Service": "bedrock.amazonaws.com"},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {"aws:SourceAccount": ""},
"ArnEquals": {
"aws:SourceArn": "arn:aws:bedrock:*::model-invocation-job/*"
}
}
}
]
}
```
??? example "Batch Service Role Permissions Policy"
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BatchDataAccess",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": "arn:aws:s3:::/batches/*"
},
{
"Sid": "BatchDataListing",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::",
"Condition": {"StringLike": {"s3:prefix": "batches/*"}}
},
{
"Sid": "BatchModelInvocation",
"Effect": "Allow",
"Action": "bedrock:InvokeModel",
"Resource": [
"arn:aws:bedrock:*::foundation-model/*",
"arn:aws:bedrock:*::inference-profile/*"
]
}
]
}
```
Repeat the S3 statements for each `AWS_S3_REGIONAL_BUCKETS` bucket, and substitute `batches/` with your `AWS_S3_BATCHES_PREFIX`. A cross-region inference profile needs `bedrock:InvokeModel` on **both** the profile and the foundation models behind it — see [Inference Profiles](#bedrock-inference-profiles-and-prompt-routers-optional).
---
## :material-account-voice: Text-to-Speech (Optional) { #text-to-speech-optional }
**Environment Variables**: [`AWS_POLLY_REGION`](operations_configuration.md#aws-polly-region), [`DEFAULT_TTS_MODEL`](operations_configuration.md#default-tts-model), [`DEFAULT_TTS_LANGUAGE`](operations_configuration.md#default-tts-language), [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket), [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets)
Required for generating speech from text using Amazon Polly. See the [Audio and Text-to-Speech](operations_configuration.md#audio-and-text-to-speech) configuration section.
!!! tip "Optimize Performance"
Set [`DEFAULT_TTS_LANGUAGE`](operations_configuration.md#default-tts-language) to skip language detection and avoid Amazon Comprehend API calls, improving response times and reducing costs.
??? example "Polly Text-to-Speech IAM Policy Statements"
```json
{
"Sid": "PollyTextToSpeech",
"Effect": "Allow",
"Action": [
"polly:SynthesizeSpeech",
"polly:StartSpeechSynthesisStream",
"polly:DescribeVoices",
"polly:StartSpeechSynthesisTask",
"polly:GetSpeechSynthesisTask"
],
"Resource": "*"
},
{
"Sid": "PollyS3Storage",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::AWS_S3_BUCKET_VALUE/*"
}
```
!!! info "Only for Long Input"
The two task actions and the S3 statement serve [input above 3,000 characters](api_openai_audio_speech.md#long-input), which Amazon Polly synthesizes into a bucket co-located with the serving region.
Replace `AWS_S3_BUCKET_VALUE` with the value of your [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) environment variable, and repeat the statement for each [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets) bucket serving a Polly region. Amazon Polly writes the audio object with the identity that started the synthesis, which is why `s3:PutObject` belongs to this policy; the audio is then read back and deleted once the response has been sent.
!!! warning "Granting the permissions is not what enables long input"
The 3,000-character limit is decided by the configuration, not by this policy: text-to-speech stays capped at 3,000 characters per request — and requests above it are rejected with that limit — as long as no bucket is configured for the Polly regions ([`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket), [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets)).
With a bucket configured but these actions missing, long requests are accepted and then fail on a permission error instead. Grant the whole set, or leave the bucket unconfigured.
**If your S3 buckets use KMS encryption**, also add the KMS permissions for each bucket's key, with that region's `kms:ViaService` value.
---
## :material-microphone: Speech-to-Text (Optional) { #speech-to-text-optional }
**Environment Variables**: [`AWS_TRANSCRIBE_REGION`](operations_configuration.md#aws-transcribe-region), [`AWS_TRANSCRIBE_S3_BUCKET`](operations_configuration.md#aws-transcribe-s3-bucket), [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets), [`AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-transcribe-output-encryption-key-arn)
Required for transcribing audio files using Amazon Transcribe. Each transcription job stages its audio in a bucket co-located with the Transcribe endpoint, so the S3 statement must cover every bucket that serves a candidate region.
??? example "Transcribe Speech-to-Text IAM Policy Statements"
```json
{
"Sid": "TranscribeSpeechToText",
"Effect": "Allow",
"Action": [
"transcribe:StartTranscriptionJob",
"transcribe:GetTranscriptionJob",
"transcribe:DeleteTranscriptionJob",
"transcribe:StartStreamTranscription"
],
"Resource": "*"
},
{
"Sid": "TranscribeTagging",
"Effect": "Allow",
"Action": [
"transcribe:TagResource"
],
"Resource": "arn:aws:transcribe:*:*:transcription-job/*"
},
{
"Sid": "TranscribeS3Storage",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:PutObjectTagging",
"s3:GetObject",
"s3:DeleteObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": "arn:aws:s3:::AWS_TRANSCRIBE_S3_BUCKET_VALUE/*"
}
```
!!! info "Replace Bucket Name"
Replace `AWS_TRANSCRIBE_S3_BUCKET_VALUE` with the value of your [`AWS_TRANSCRIBE_S3_BUCKET`](operations_configuration.md#aws-transcribe-s3-bucket) environment variable (or [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) if using the same bucket).
!!! note "One Bucket per Candidate Region"
With the default multi-region behavior ([`AWS_TRANSCRIBE_REGION`](operations_configuration.md#aws-transcribe-region) unset), Transcribe fails over across the [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) that have a co-located bucket: the primary region uses the bucket above, the others their [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets) entry. Repeat the `TranscribeS3Storage` statement for each of those buckets.
On failover the audio is server-side copied from the previous candidate's bucket to the next one, which is why the copy and multipart actions are required. Set `AWS_TRANSCRIBE_REGION` to pin a single region and keep a single bucket.
!!! info "`StartStreamTranscription` needs no bucket"
`transcribe:StartStreamTranscription` serves [`stream=true`](api_openai_audio_transcriptions.md#streaming) requests, which send their audio to Transcribe directly instead of staging it. A deployment with no bucket at all still serves those, and only those — the `TranscribeS3Storage` statement above is what the other requests need.
**If your transcribe S3 buckets use KMS encryption**, also add the KMS permissions for each bucket's key, with that region's `kms:ViaService` value.
**Encrypting the transcription output with your own key** ([`AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-transcribe-output-encryption-key-arn)) additionally requires:
??? example "Transcribe Output Encryption IAM Policy Statement"
```json
{
"Sid": "TranscribeOutputEncryption",
"Effect": "Allow",
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt"
],
"Resource": "AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN_VALUE"
}
```
!!! info "Replace Key ARN"
Replace `AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN_VALUE` with the value of your [`AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-transcribe-output-encryption-key-arn) environment variable. The key policy must allow the same actions for this role; `kms:Decrypt` is what lets the finished transcript be read back.
---
## :material-earth: Language Detection (Optional)
**Environment Variables**: [`AWS_COMPREHEND_REGION`](operations_configuration.md#aws-comprehend-region)
Required for automatic language detection (used by TTS for voice selection).
??? example "Comprehend Language Detection IAM Policy Statement"
```json
{
"Sid": "ComprehendLanguageDetection",
"Effect": "Allow",
"Action": [
"comprehend:DetectDominantLanguage"
],
"Resource": "*"
}
```
---
## :material-shield-alert: Comprehend Moderation (Optional) { #comprehend-moderation }
**Environment Variables**: [`AWS_COMPREHEND_REGION`](operations_configuration.md#aws-comprehend-region)
Required for the [Moderations API](api_openai_moderations.md) toxicity backend — the default backend when no Bedrock guardrail is configured, and always available as the `amazon.comprehend-toxicity` model.
??? example "Comprehend Moderation IAM Policy Statement"
```json
{
"Sid": "ComprehendModeration",
"Effect": "Allow",
"Action": [
"comprehend:DetectToxicContent"
],
"Resource": "*"
}
```
---
## :material-translate: Text Translation (Optional)
**Environment Variables**: [`AWS_TRANSLATE_REGION`](operations_configuration.md#aws-translate-region)
Required for text translation features.
??? example "Translate Text Translation IAM Policy Statement"
```json
{
"Sid": "TranslateTextTranslation",
"Effect": "Allow",
"Action": [
"translate:TranslateText",
"translate:ListLanguages"
],
"Resource": "*"
}
```
!!! note "`translate:ListLanguages`"
Read once at startup, to refuse a language Amazon Translate does not support before the audio is transcribed. Without it translation still works: an unsupported language is then reported once the translation call itself fails.
---
## :material-cash-multiple: Cost Tracking (Optional) { #cost-tracking-iam }
**Environment Variables**: [`COST_TRACKING`](operations_configuration.md#cost-tracking)
Required for [`COST_TRACKING`](operations_configuration.md#cost-tracking) (disabled by default), which prices requests from the AWS Price List API. Without this permission the catalog stays empty and request logs carry no cost data.
??? example "Cost Tracking IAM Policy Statement"
```json
{
"Sid": "PricingCatalog",
"Effect": "Allow",
"Action": [
"pricing:GetProducts"
],
"Resource": "*"
}
```
---
## :material-account-cash: Per-User Cost Attribution (Optional) { #per-user-cost-attribution }
**Environment Variables**: [`AWS_BEDROCK_USER_ROLE_ARN`](operations_configuration.md#aws-bedrock-user-role-arn)
Required to run each end user's model calls under a role session of their own, so AWS reports [their spend separately](operations_cost_management.md#per-user-attribution). Three policies are involved: the server's own role must be allowed to open the sessions, the end user role must trust it to do so, and the end user role must be allowed to invoke models.
**1. On the server's role** — allow it to open sessions of the end user role, and of that role only:
??? example "Server Role IAM Policy Statement"
```json
{
"Sid": "EndUserRoleSessions",
"Effect": "Allow",
"Action": [
"sts:AssumeRole",
"sts:TagSession"
],
"Resource": "arn:aws:iam::ACCOUNT_ID:role/stdapi-ai-end-user"
}
```
**2. Trust policy of the end user role** — allow the server's role, and nothing else, to assume it *and* to tag the session. `sts:TagSession` is a separate action: without it, every session that carries the end user tag is denied.
??? example "End User Role Trust Policy"
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNT_ID:role/stdapi-ai-task-role"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
]
}
]
}
```
**3. Permission policy of the end user role** — everything AWS authorizes against the caller of a model invocation: the invocation actions on the models the deployment serves, and the guardrail the invocation carries:
??? example "End User Role IAM Policy Statements"
```json
{
"Sid": "EndUserModelInvoke",
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream"
],
"Resource": [
"arn:aws:bedrock:*:ACCOUNT_ID:inference-profile/*",
"arn:aws:bedrock:*:ACCOUNT_ID:application-inference-profile/*",
"arn:aws:bedrock:*:ACCOUNT_ID:default-prompt-router/*",
"arn:aws:bedrock:*::foundation-model/*"
]
},
{
"Sid": "EndUserApplyGuardrail",
"Effect": "Allow",
"Action": [
"bedrock:ApplyGuardrail"
],
"Resource": "arn:aws:bedrock:*:ACCOUNT_ID:guardrail/*"
}
```
Replace `ACCOUNT_ID` with your AWS account ID, and `stdapi-ai-task-role` with the role the server runs as.
!!! warning "An inference profile needs the foundation models behind it"
A cross-region inference profile routes to a foundation model in each of its Regions, and AWS authorizes **both** the profile ARN and every foundation model ARN it reaches. A policy naming only `inference-profile/...` fails with an access-denied error naming `foundation-model/...` in a Region you never configured. Keep `arn:aws:bedrock:*::foundation-model/...` alongside the profile, or the call is denied.
!!! warning "A configured guardrail is authorized against the end user"
A guardrail applied **during** an invocation — [`AWS_BEDROCK_GUARDRAIL_IDENTIFIER`](operations_configuration.md#aws-bedrock-guardrail-identifier), a model alias carrying one, or a request-level `moderation` parameter — is evaluated as part of that invocation, so AWS requires `bedrock:ApplyGuardrail` from the identity making the call. Without the `EndUserApplyGuardrail` statement, every model request fails with an access-denied error as soon as per-user attribution is enabled. See [Set up permissions to use Amazon Bedrock Guardrails](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-permissions.html). Narrow the resource to your guardrail ARN if you prefer.
!!! note "Name every model ARN form the deployment allows"
The `Resource` list must cover every ARN a request can resolve to. Add `arn:aws:bedrock:*:ACCOUNT_ID:prompt/*` when [`AWS_BEDROCK_ALLOW_PROMPT_ARN`](operations_configuration.md#bedrock-allow-prompt-arn) is enabled, and keep the application inference profile and prompt router entries above whenever [`AWS_BEDROCK_ALLOW_APPLICATION_INFERENCE_PROFILE_ARN`](operations_configuration.md#bedrock-allow-application-profile-arn), [`AWS_BEDROCK_ALLOW_PROMPT_ROUTER_ARN`](operations_configuration.md#bedrock-allow-prompt-router-arn) or [`AWS_BEDROCK_MODEL_ARN_MAPPING`](operations_configuration.md#bedrock-model-arn-mapping) can put one in front of a model. An ARN form the end user role does not name is denied under it while it still works on the server's role.
Add the [Web Search](#web-search-iam) actions to this role as well if you serve `web_search` requests: AWS evaluates them when the model actually runs a search, which happens inside the invocation the end user signed. A denied search does not fail the request, it degrades the answer.
!!! danger "A session tag is an access boundary only when the identity is verified"
The end user identity is taken from the authenticated caller only under [Amazon Cognito authentication](operations_authentication_security.md). With an API key, or with no authentication, it is whatever the client declared in the request body (`safety_identifier`, `user`, `metadata.user_id`) — so any caller holding the key can send another user's identifier and obtain that user's session tag.
Write policies conditioned on `aws:PrincipalTag/` only when [`AUTHENTICATION_MODE`](operations_configuration.md#authentication-mode) is `cognito`, which is the configuration where every request carries an identity the gateway verified. Anywhere else, treat the tag as cost metadata, never as an authorization input.
!!! tip "Restricting a role per end user"
Where the identity is verified, the session tag makes it testable in a policy: compare it to something on the resource side, so each session reaches only its own data — `"StringEquals": {"aws:ResourceTag/user": "${aws:PrincipalTag/user}"}`, an `s3:prefix` condition, or a `Resource` ARN embedding `${aws:PrincipalTag/user}`. A condition comparing the tag to itself always matches and restricts nothing. A `Deny` on any tag value the deployment does not expect is the other half of the same pattern. Set [`AWS_BEDROCK_USER_ROLE_TAG_KEY`](operations_configuration.md#aws-bedrock-user-role-tag-key) to the key the policy tests.
!!! note "Scope"
Only Bedrock model invocations run under the end user role, together with the guardrail applied during them. Standalone guardrail evaluations (the [Moderations API](api_openai_moderations.md)), reranking, video generation and its output files, speech, transcription and translation keep the server's own role, so the end user role needs none of their permissions — and the server's role still needs all of them.
---
## :material-key: API Key Authentication (Optional)
Required if you configure API authentication. See the [Authentication](operations_configuration.md#authentication) configuration section.
### SSM Parameter Store
**Environment Variables**: [`API_KEY_SSM_PARAMETER`](operations_configuration.md#api-key-ssm)
??? example "SSM Parameter Store IAM Policy Statements"
```json
{
"Sid": "SSMParameterAccess",
"Effect": "Allow",
"Action": [
"ssm:GetParameter"
],
"Resource": "arn:aws:ssm:REGION:ACCOUNT_ID:parameter/API_KEY_SSM_PARAMETER_VALUE"
}
```
!!! info "Replace Parameter Path"
Replace `API_KEY_SSM_PARAMETER_VALUE` with the value of your [`API_KEY_SSM_PARAMETER`](operations_configuration.md#api-key-ssm) environment variable (e.g., `/stdapi/prod/api-key`), and `REGION` with the server's own region (`AWS_REGION`) — parameters are read there, not in the Bedrock regions.
**If using encrypted SSM parameters**, also add:
```json
{
"Sid": "KMSDecryptionForSSM",
"Effect": "Allow",
"Action": [
"kms:Decrypt"
],
"Resource": "arn:aws:kms:REGION:ACCOUNT_ID:key/YOUR_KMS_KEY_ID",
"Condition": {
"StringEquals": {
"kms:ViaService": "ssm.REGION.amazonaws.com"
}
}
}
```
!!! tip "KMS Security"
The `kms:ViaService` condition restricts KMS key usage to SSM service calls only.
### Secrets Manager
**Environment Variables**: [`API_KEY_SECRETSMANAGER_SECRET`](operations_configuration.md#api-key-secretsmanager-secret)
??? example "Secrets Manager IAM Policy Statement"
```json
{
"Sid": "SecretsManagerAccess",
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue"
],
"Resource": "arn:aws:secretsmanager:REGION:ACCOUNT_ID:secret:API_KEY_SECRETSMANAGER_SECRET_VALUE"
}
```
!!! info "Replace Secret Name"
Replace `API_KEY_SECRETSMANAGER_SECRET_VALUE` with the value of your [`API_KEY_SECRETSMANAGER_SECRET`](operations_configuration.md#api-key-secretsmanager-secret) environment variable (e.g., `stdapi-api-key`), and `REGION` with the server's own region (`AWS_REGION`) — secrets are read there, not in the Bedrock regions.
---
## :material-file-document: Complete Policy Examples
??? example "Minimal Policy (Bedrock Only)"
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BedrockModelInvoke",
"Effect": "Allow",
"Action": [
"bedrock:CountTokens",
"bedrock:GetAsyncInvoke",
"bedrock:InvokeGuardrailChecks",
"bedrock:InvokeModel",
"bedrock:InvokeModelWithBidirectionalStream",
"bedrock:InvokeModelWithResponseStream",
"bedrock:InvokeTool",
"bedrock:Rerank"
],
"Resource": "*"
},
{
"Sid": "BedrockAsyncInvokeTagging",
"Effect": "Allow",
"Action": [
"bedrock:TagResource"
],
"Resource": "arn:aws:bedrock:*:*:async-invoke/*"
},
{
"Sid": "BedrockModelDiscovery",
"Effect": "Allow",
"Action": [
"bedrock:ListFoundationModels",
"bedrock:GetFoundationModelAvailability",
"bedrock:ListProvisionedModelThroughputs",
"bedrock:ListInferenceProfiles"
],
"Resource": "*"
},
{
"Sid": "BedrockMarketplaceAutoSubscribe",
"Effect": "Allow",
"Action": [
"aws-marketplace:Subscribe",
"aws-marketplace:ViewSubscriptions"
],
"Resource": "*"
},
{
"Sid": "MarketplaceRegisterUsage",
"Effect": "Allow",
"Action": [
"aws-marketplace:RegisterUsage"
],
"Resource": "*"
}
]
}
```
!!! note "Marketplace Auto-Subscribe (Default Enabled)"
The marketplace permissions are included because `AWS_BEDROCK_MARKETPLACE_AUTO_SUBSCRIBE` defaults to `true`. If you set it to `false`, you can remove the `BedrockMarketplaceAutoSubscribe` statement.
!!! note "Marketplace RegisterUsage (AWS Marketplace Image Only)"
`MarketplaceRegisterUsage` is only needed on the **AWS Marketplace image**; remove it when deploying the community image.
??? example "Production Policy (Bedrock + S3 + Authentication)"
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BedrockModelInvoke",
"Effect": "Allow",
"Action": [
"bedrock:CountTokens",
"bedrock:GetAsyncInvoke",
"bedrock:InvokeGuardrailChecks",
"bedrock:InvokeModel",
"bedrock:InvokeModelWithBidirectionalStream",
"bedrock:InvokeModelWithResponseStream",
"bedrock:InvokeTool",
"bedrock:Rerank"
],
"Resource": "*"
},
{
"Sid": "BedrockAsyncInvokeTagging",
"Effect": "Allow",
"Action": [
"bedrock:TagResource"
],
"Resource": "arn:aws:bedrock:*:*:async-invoke/*"
},
{
"Sid": "BedrockModelDiscovery",
"Effect": "Allow",
"Action": [
"bedrock:ListFoundationModels",
"bedrock:GetFoundationModelAvailability",
"bedrock:ListProvisionedModelThroughputs",
"bedrock:ListInferenceProfiles"
],
"Resource": "*"
},
{
"Sid": "BedrockMarketplaceAutoSubscribe",
"Effect": "Allow",
"Action": [
"aws-marketplace:Subscribe",
"aws-marketplace:ViewSubscriptions"
],
"Resource": "*"
},
{
"Sid": "MarketplaceRegisterUsage",
"Effect": "Allow",
"Action": [
"aws-marketplace:RegisterUsage"
],
"Resource": "*"
},
{
"Sid": "S3FileStorage",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:PutObjectTagging",
"s3:GetObject",
"s3:DeleteObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": "arn:aws:s3:::my-stdapi-bucket/*"
},
{
"Sid": "S3FileStorageList",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": "arn:aws:s3:::my-stdapi-bucket"
},
{
"Sid": "SSMParameterAccess",
"Effect": "Allow",
"Action": [
"ssm:GetParameter"
],
"Resource": "arn:aws:ssm:us-east-1:123456789012:parameter/stdapi/prod/api-key"
},
{
"Sid": "PricingCatalog",
"Effect": "Allow",
"Action": [
"pricing:GetProducts"
],
"Resource": "*"
}
]
}
```
!!! note "Marketplace Auto-Subscribe (Default Enabled)"
The marketplace permissions are included because `AWS_BEDROCK_MARKETPLACE_AUTO_SUBSCRIBE` defaults to `true`. If you set it to `false`, you can remove the `BedrockMarketplaceAutoSubscribe` statement to follow the principle of least privilege.
!!! note "Marketplace RegisterUsage (AWS Marketplace Image Only)"
`MarketplaceRegisterUsage` is only needed on the **AWS Marketplace image**; remove it when deploying the community image.
!!! note "Cost Tracking (Opt-In)"
`PricingCatalog` is only needed when [`COST_TRACKING`](operations_configuration.md#cost-tracking) is set to `true`; remove it otherwise.
---
## :material-table: Feature-Specific Permission Requirements
| Feature | Required Permissions | Configuration |
|-------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|
| **Bedrock Models (Invoke)** | `bedrock:CountTokens` `bedrock:InvokeGuardrailChecks` `bedrock:InvokeModel` `bedrock:InvokeModelWithBidirectionalStream` `bedrock:InvokeModelWithResponseStream` `bedrock:InvokeTool` (the Amazon Nova grounding server tool) `bedrock:Rerank` `bedrock:GetAsyncInvoke` and `bedrock:TagResource` (on `arn:aws:bedrock:*:*:async-invoke/*`) for async-invoke models (video, TwelveLabs Marengo embeddings) | Always required |
| **Realtime API** | `bedrock:InvokeModelWithBidirectionalStream` (already part of the core Bedrock policy above); no additional action | `POST /v1/realtime/client_secrets`, `WS /v1/realtime` |
| **Bedrock Models (Discovery)** | `bedrock:ListFoundationModels` `bedrock:GetFoundationModelAvailability` `bedrock:ListProvisionedModelThroughputs` `bedrock:ListInferenceProfiles` | Always required |
| **Bedrock Marketplace Auto-Subscribe** | `aws-marketplace:Subscribe` `aws-marketplace:ViewSubscriptions` | `AWS_BEDROCK_MARKETPLACE_AUTO_SUBSCRIBE=true` (default) |
| **AWS Marketplace Metering** | `aws-marketplace:RegisterUsage` | AWS Marketplace image only (always active); not required for the community image |
| **Bedrock Inference Profiles & Prompt Routers** | `bedrock:GetInferenceProfile` `bedrock:GetPromptRouter` `bedrock:GetPrompt` and `bedrock:RenderPrompt` (on `arn:aws:bedrock:*:*:prompt/*`) for Prompt Management prompts | `AWS_BEDROCK_ALLOW_*_ARN=true` or `AWS_BEDROCK_MODEL_ARN_MAPPING` configured |
| **Bedrock Guardrails & Moderations** | `bedrock:ApplyGuardrail` | `AWS_BEDROCK_GUARDRAIL_IDENTIFIER` |
| **Stored Responses & Chat Completions** | Bedrock session permissions (`bedrock:CreateSession`, `bedrock:GetSession`, `bedrock:*Invocation*`, `bedrock:ListSessions`, `bedrock:EndSession`, `bedrock:DeleteSession`, `bedrock:TagResource`, `bedrock:ListTagsForResource` on sessions) | `store=true` requests and stored-completion listings |
| **Bedrock Mantle** | `bedrock-mantle:CreateInference` `bedrock-mantle:GetInference` `bedrock-mantle:DeleteInference` `bedrock-mantle:ListModels` `bedrock-mantle:GetModel` `bedrock-mantle:CancelInference` (on `arn:aws:bedrock-mantle:*:*:project/*`) `bedrock-mantle:CallWithBearerToken` | `AWS_BEDROCK_MANTLE_ENABLED=true` |
| **Web Search** | `bedrock-websearch:InvokeSearch` `bedrock-websearch:InvokeFetch` `bedrock-websearch:ExternalWebAccess` only when external web access is enabled | `web_search` requests on the OpenAI GPT-5.x family |
| **File Storage** | `s3:PutObject` `s3:PutObjectTagging` `s3:GetObject` `s3:DeleteObject` `s3:AbortMultipartUpload` `s3:ListMultipartUploadParts` `s3:ListBucket` `s3:ListBucketMultipartUploads` on every bucket, including each `AWS_S3_REGIONAL_BUCKETS` entry | `AWS_S3_BUCKET` `AWS_S3_REGIONAL_BUCKETS` |
| **Video Generation** | Core Bedrock invoke permissions (incl. `bedrock:GetAsyncInvoke`, `bedrock:TagResource`) `bedrock:ListAsyncInvokes` and `bedrock:ListTagsForResource` (on `arn:aws:bedrock:*:*:async-invoke/*`) for job listing File Storage S3 permissions on each regional bucket | `AWS_S3_REGIONAL_BUCKETS` |
| **Batch Inference** | `bedrock:CreateModelInvocationJob` `bedrock:GetModelInvocationJob` `bedrock:StopModelInvocationJob` (on `arn:aws:bedrock:*:*:model-invocation-job/*`) `iam:PassRole` on the batch service role, scoped with `iam:PassedToService: bedrock.amazonaws.com` File Storage S3 permissions on each bucket a batch uses, plus the service role's own policy (see [Batch Inference](#batch-inference)) | `AWS_BEDROCK_BATCH_ROLE_ARN` |
| **Vector Stores** | `s3vectors:CreateIndex` `s3vectors:DeleteIndex` `s3vectors:PutVectors` `s3vectors:GetVectors` `s3vectors:QueryVectors` `s3vectors:DeleteVectors` (on the vector bucket and its indexes) File Storage S3 permissions on `AWS_S3_BUCKET` for the stores' records | `AWS_S3_VECTORS_BUCKET` `AWS_S3_VECTORS_REGION` |
| **Durable Vector Store Indexing** | `sqs:SendMessage` `sqs:ReceiveMessage` `sqs:DeleteMessage` `sqs:ChangeMessageVisibility` `sqs:GetQueueAttributes` (on the queue ARN only) | `AWS_SQS_VECTOR_STORE_QUEUE_URL` |
| **Knowledge Base Vector Stores** | `bedrock:GetKnowledgeBase` `bedrock:Retrieve` `bedrock:ListDataSources` `bedrock:IngestKnowledgeBaseDocuments` `bedrock:ListKnowledgeBaseDocuments` `bedrock:GetKnowledgeBaseDocuments` `bedrock:DeleteKnowledgeBaseDocuments` (on each allowlisted knowledge base ARN; no `bedrock:ListKnowledgeBases`) | `AWS_BEDROCK_KNOWLEDGE_BASE_IDS` |
| **KMS Encrypted S3 Buckets** | `kms:Decrypt` `kms:GenerateDataKey` with `kms:ViaService` condition | If S3 buckets use KMS encryption |
| **Text-to-Speech** | `polly:SynthesizeSpeech` `polly:DescribeVoices` `polly:StartSpeechSynthesisStream` for generative voices above 3,000 characters `polly:StartSpeechSynthesisTask`, `polly:GetSpeechSynthesisTask` and S3 `PutObject`/`GetObject`/`DeleteObject` on each bucket serving a Polly region, for the other voices above 3,000 characters | `AWS_POLLY_REGION` `AWS_S3_BUCKET` `AWS_S3_REGIONAL_BUCKETS` |
| **Speech-to-Text** | `transcribe:StartTranscriptionJob` `transcribe:GetTranscriptionJob` `transcribe:DeleteTranscriptionJob` `transcribe:StartStreamTranscription` `transcribe:TagResource` (on `arn:aws:transcribe:*:*:transcription-job/*`) File Storage S3 permissions on every bucket serving a candidate region `kms:GenerateDataKey`, `kms:Decrypt` on the output encryption key, when one is configured | `AWS_TRANSCRIBE_REGION` `AWS_TRANSCRIBE_S3_BUCKET` `AWS_S3_REGIONAL_BUCKETS` `AWS_TRANSCRIBE_STREAM_LANGUAGES` `AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN` |
| **Language Detection** | `comprehend:DetectDominantLanguage` | `AWS_COMPREHEND_REGION` |
| **Comprehend Moderations** | `comprehend:DetectToxicContent` | Moderations API without a configured guardrail |
| **Translation** | `translate:TranslateText` `translate:ListLanguages` (optional; validates the language pair before transcribing) | `AWS_TRANSLATE_REGION` |
| **Cost Tracking** | `pricing:GetProducts` | `COST_TRACKING=true` (opt-in; `false` by default) |
| **Per-User Cost Attribution** | `sts:AssumeRole` and `sts:TagSession` on the end user role, matched by that role's trust policy; on the end user role itself, `bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream` on every model ARN form the deployment allows, plus `bedrock:ApplyGuardrail` when a guardrail is configured (see [Per-User Cost Attribution](#per-user-cost-attribution)) | `AWS_BEDROCK_USER_ROLE_ARN` |
| **SSM Parameter Store** | `ssm:GetParameter` `kms:Decrypt` (if encrypted) | `API_KEY_SSM_PARAMETER` |
| **Secrets Manager** | `secretsmanager:GetSecretValue` | `API_KEY_SECRETSMANAGER_SECRET` |
---
## :material-account: IAM Role vs. IAM User
stdapi.ai supports both IAM roles and IAM users:
- **:material-aws: IAM Role (Recommended)**: Use when running on EC2, ECS, Lambda, or other AWS compute services. Attach the policy to the instance/task role.
- **:material-account: IAM User**: Use when running outside AWS or for development. Create an IAM user with the required permissions and configure AWS credentials via environment variables or AWS CLI configuration.
!!! success "Best Practice: Use IAM Roles"
When deploying on AWS infrastructure, always prefer IAM roles over IAM users with access keys. IAM roles provide automatic credential rotation and better security.
---
## :material-tag: AWS Tag Policies
If your AWS organization enforces a [tag policy](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_tag-policies.html), the following tag keys must be allowed on the relevant resource types.
| Tag key | Value | Applied to |
|---------|-------|------------|
| `stdapi-ai.expires` | `"true"` | S3 objects (Files API expiry) |
| `stdapi-ai.request_id` | request UUID | Bedrock async jobs, Transcribe jobs |
| `stdapi-ai.server_id` | server instance name | Bedrock async jobs, Transcribe jobs |
| `stdapi-ai.user_id` | user identifier | Bedrock async jobs, Transcribe jobs (when user identity is known) |
| `aws-apn-id` | `pc:` | All AWS resources created at runtime and by the Terraform module. This is a [standard AWS Marketplace attribution tag](https://docs.aws.amazon.com/PRM/latest/aws-prm-onboarding-guide/what-is-service.html) required by any AWS Marketplace product — allowing it benefits all such products deployed in your organization, not only stdapi.ai. |
---
## :material-arrow-right: Next Steps
---
## Licensing
Source: https://stdapi.ai/md/operations_licensing.md
# :material-scale-balance: Licensing
## :material-scale-balance: Choose the License That Fits Your Business
stdapi.ai is available under a dual-license model designed to support both open-source communities and commercial enterprises:
- :material-scale-balance:{ .lg .middle } __AGPL-3.0-or-later (Free & Open Source)__
---
Free forever for those who share their code
**Best for:**
- Open-source projects
- Research and education
- Development and testing
- Projects that can comply with AGPL
**Includes:** Community Docker image, full source code access
[:octicons-link-external-16: View License](https://spdx.org/licenses/AGPL-3.0-or-later.html)
- :material-briefcase:{ .lg .middle } __Commercial License (AWS Marketplace)__
---
Production-ready for proprietary applications — **14-day free trial included**
**Best for:**
- Internal company tools
- SaaS products
- Proprietary applications
- Commercial integrations
**Includes:** Hardened container, security updates, commercial support
[:octicons-arrow-right-24: Start 14-Day Free Trial on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-su2dajk5zawpo)
---
## :material-briefcase-check: Why Choose the Commercial License?
A commercial license through AWS Marketplace provides everything you need for production deployments:
### :fontawesome-solid-lock: **Keep Your Code Private**
Build proprietary applications without AGPL's source code disclosure requirements. Your intellectual property and modifications stay private.
### :fontawesome-solid-rocket: **Deploy with Confidence**
No copyleft obligations - integrate stdapi.ai into your products and internal tools without AGPL network use restrictions.
### :fontawesome-solid-shield-halved: **Production-Grade Security**
Access hardened and optimized container images with regular security updates, vulnerability patches, and AWS security best practices.
### :fontawesome-solid-arrows-turn-to-dots: **Modify & Distribute Freely**
Customize stdapi.ai to fit your needs and distribute it as part of your commercial offerings without restrictions.
### :fontawesome-solid-handshake: **Enterprise-Ready Terms**
Licensed under the [AWS Marketplace Standard Contract (SCMP)](https://aws.amazon.com/marketplace/management/terms) - straightforward, trusted terms used by thousands of enterprise products.
### :fontawesome-solid-credit-card: **Streamlined AWS Billing**
Purchase and pay through AWS Marketplace. Consolidated billing with your existing AWS infrastructure costs.
---
## :material-check-all: Commercial License Benefits
- :material-shield-check:{ .lg .middle } __Full Commercial Rights__
---
Use in closed-source applications, modify freely, and distribute as part of your products
- :material-file-code:{ .lg .middle } __No Source Disclosure__
---
AGPL network use requirements don't apply—keep your modifications private
- :material-domain:{ .lg .middle } __Internal Network Freedom__
---
Deploy on internal networks without triggering AGPL's source disclosure obligations
- :material-aws:{ .lg .middle } __Streamlined Procurement__
---
Purchase through AWS Marketplace with familiar terms and billing
- :material-docker:{ .lg .middle } __Hardened Container Images__
---
Optimized and security-hardened containers built for production workloads
- :material-security:{ .lg .middle } __Regular Security Updates__
---
Stay protected with timely security patches and vulnerability fixes
- :material-cloud-upload:{ .lg .middle } __Easy Deployment Options__
---
Quick deployment workflows optimized for AWS infrastructure
- :material-certificate:{ .lg .middle } __AWS Best Practices__
---
Deployment options follow AWS Well-Architected Framework guidelines
---
## :material-road-variant: Your Path to Production
A typical adoption path from evaluation to production:
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
eval["1. Evaluate Docker Community (free, AGPL)"] --> trial["2. Trial AWS Marketplace (14 days free)"]
trial --> prod["3. Production Terraform deploy (pay-per-use)"]
```
| Step | What | Cost | License |
|------|------|------|---------|
| **Evaluate** | Run the community Docker image locally | Free | AGPL-3.0 |
| **Trial** | Subscribe on AWS Marketplace and test in your AWS environment | Free for 14 days | Commercial |
| **Production** | Deploy with Terraform — same config, swap container image | AWS pay-per-use | Commercial |
No migration required between steps. Your Terraform configuration and application code remain the same.
### Quick Start (3 Steps)
1. **Subscribe** — Visit [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-su2dajk5zawpo) and subscribe to stdapi.ai (14-day free trial included)
2. **Accept Contract** — Accept the AWS Marketplace Standard Contract during checkout
3. **Deploy** — Use the [Terraform module](operations_getting_started.md) to deploy immediately with commercial rights activated
**Pricing:** The commercial license is billed at **$0.10/container-hour** through AWS Marketplace — no per-request markup and no markup on model usage. You pay Amazon Bedrock rates directly for inference. The bill therefore scales with **how many containers run, and for how long**: the Terraform module runs **one task per Availability Zone by default**, so the container count follows your AZ count unless you set it explicitly.
**Monthly cost examples** (license only, excluding AWS infrastructure and Bedrock model costs — 720 h/month per running container):
| Scenario | Containers | Container-hours/month | License cost |
|---|---|---|---|
| **Module default**, 3-AZ region, 24 × 7 | 3 | 2,160 h | ~$216/month |
| **Module default**, `us-east-1` (6 AZs), 24 × 7 | 6 | 4,320 h | ~$432/month |
| Cost-optimized: single container, 24 × 7 | 1 | 720 h | ~$72/month |
| Cost-optimized: single container, dev/staging with scheduled stop outside business hours | 1 | ~165 h | ~$17/month |
!!! note "Why the default is more than one container"
The Terraform module leaves `availability_zones_count` unset (it uses **all Availability Zones in the region**) and `autoscaling_min_capacity` unset (**one task per Availability Zone**), so a default deployment runs 3 containers in a typical 3-AZ region and 6 in `us-east-1`. The two cost-optimized rows above assume `autoscaling_min_capacity = 1`, which trades the multi-AZ redundancy of the default for a lower floor. See [Keeping It Low](operations_cost_management.md#keeping-it-low) for the settings that drive this number.
!!! note "Fargate Spot does not reduce the license cost"
The license fee is billed per container-hour regardless of the Fargate pricing model — a default 3-AZ deployment is 2,160 h × $0.10 = $216/month whether the underlying containers run on Spot or standard Fargate. Spot only discounts the **AWS infrastructure** cost (the ECS/Fargate compute itself), not the Marketplace license. See [Cost Management](operations_cost_management.md#gateway-cost) for the full infrastructure-vs-license breakdown.
Consolidated AWS billing — one bill, no separate invoice or procurement (AWS promotional credits do not apply to Marketplace charges).
### After Subscribing
Once subscribed, access the hardened container image from AWS Marketplace ECR and deploy using:
- **Terraform module** (recommended) — [See Quick Start guide](operations_getting_started.md#quick-start)
- **Manual ECS deployment** — [See manual setup](operations_deploy_advanced.md#manual-ecs-deployment)
Your commercial license activates automatically upon deployment.
---
## :material-help-circle-outline: Frequently Asked Questions
??? question "Can I try before I buy?"
Yes, two ways:
1. **Free AGPL version** - Use the community Docker image (AGPL-3.0) for evaluation, development, and testing. No cost, full features.
2. **14-day free trial** - AWS Marketplace commercial license includes a 14-day trial of the license. Test the hardened production container in your environment; AWS resources and model usage bill as usual from the first minute.
Both options give you full access to validate stdapi.ai before committing.
??? question "Do I need a commercial license for internal company tools?"
**Yes, in most cases.** The AGPL-3.0 license requires source code disclosure even for software accessed over a network (the "network use" clause).
**Examples requiring commercial license:**
- Internal API gateway for your team
- Company-wide AI chatbot
- Department-specific automation tools
- Private SaaS offering to customers
The commercial license removes these obligations and lets you keep your code and deployments private.
??? question "What if I'm okay with open-sourcing my application?"
If you're comfortable open-sourcing your entire application under AGPL-3.0 (including making source available to network users), the free community version works perfectly.
**AGPL-3.0 is fine for:**
- Open-source projects that share all code
- Research and academic projects
- Personal experiments and learning
The commercial license is for organizations that need proprietary code or can't meet AGPL's network disclosure requirements.
??? question "Can I switch from AGPL to commercial later?"
**Yes, absolutely.** This is a common path:
1. **Develop & test** with free AGPL community Docker image
2. **Subscribe** to AWS Marketplace when ready for production
3. **Deploy** with commercial container image - automatic license activation
No migration required. Just swap container images and deploy. Your Terraform configuration remains the same.
??? question "What's included in the AWS Marketplace Standard Contract?"
The [AWS Marketplace Standard Contract (SCMP)](https://aws.amazon.com/marketplace/management/terms) provides:
- Clear license grant terms for commercial use
- Standard warranty and liability provisions
- Straightforward payment and billing terms
- Professional indemnification terms
It's the same trusted contract used across thousands of enterprise software products on AWS Marketplace.
??? question "I'm building a product for multiple customers. How does licensing work?"
One AWS Marketplace subscription per running stdapi.ai deployment. The per-container-hour billing handles this automatically — each instance you run accumulates its own usage.
**Common patterns for ISVs and MSPs:**
- **One instance per customer** — each customer gets their own stdapi.ai deployment in their AWS account (or a dedicated account you manage). Each instance has its own Marketplace subscription billed to the owning account.
- **Shared multi-tenant instance** — one instance serves multiple customers. One subscription covers the shared deployment. Use the `safety_identifier` field (or the deprecated `user` field) in API requests to tag traffic per customer for audit trails and usage attribution.
For questions about volume deployments or enterprise licensing, contact [sales@stdapi.ai](mailto:sales@stdapi.ai).
??? question "Where are the third-party licences inside the container image?"
The image carries the licence and notice files of everything it redistributes, so a legal or compliance review can be done against the artifact itself:
- **Python dependencies** — each package keeps its `LICENSE`/`NOTICE`/`COPYING`/`AUTHORS` files (and its `METADATA` and SBOM) inside its own `.dist-info` directory under `/opt/app`.
- **FFmpeg** — `/usr/share/licenses/ffmpeg/`, alongside a package entry in the APK inventory so image scanners inventory it like any other installed package.
- **Swagger UI** (Apache-2.0) — `/usr/share/licenses/swagger-ui-dist/`, holding its `LICENSE`, its `NOTICE` and the bundled-dependency notice its script carries.
- **ReDoc** (MIT) — `/usr/share/licenses/redoc/`, holding its `LICENSE` and the bundled-dependency notice its script carries.
- **stdapi.ai** — `/usr/share/licenses/stdapi.ai/`, holding `LICENSE-AGPL` on the community image.
Swagger UI and ReDoc are the two documentation UIs served at `/docs` and `/redoc`. The image ships them itself, pinned to an exact release and verified against a recorded SHA-256 at build time, so those pages render with no outbound request at all.
??? question "How do I update stdapi.ai?"
Updates are container image redeployments — no data migration or downtime window required.
**Using the Terraform module** (recommended): the Terraform module is versioned in sync with the application. Bumping the module version in your Terraform config and running `terraform apply` updates the ECS task definition to the new container image and performs a rolling replacement of running tasks with zero downtime.
**Manual ECS deployment**: update the container image tag in your ECS task definition and trigger a new deployment.
Security patches are released as new container image versions. Subscribe to [GitHub releases](https://github.com/stdapi-ai/stdapi.ai/releases) to be notified.
---
## :material-email-outline: Support & Deployment Help
### :material-email: Email Support
**Email:** [support@stdapi.ai](mailto:support@stdapi.ai)
Response time: **1 business day** for commercial license subscribers.
For licensing questions, volume pricing, or custom deployment scenarios, contact [sales@stdapi.ai](mailto:sales@stdapi.ai).
### :material-tools: Managed Deployment Service
If you'd prefer not to manage the Terraform deployment yourself, a [managed deployment service](https://aws.amazon.com/marketplace/pp/prodview-xknxzjgl7zi5s) is available on AWS Marketplace. Choose between:
- **Guided setup** — step-by-step assistance while you retain full control
- **Fully managed** — the deployment is handled entirely on your behalf, within your AWS account
Response time is **1 business day** during the engagement. All work is performed inside your own AWS account — no data leaves your environment.
[:octicons-arrow-right-24: View Managed Deployment Service on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-xknxzjgl7zi5s)
---
## :material-arrow-right: Next Steps
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy the commercial container with Terraform
- :material-cash-multiple: [**Cost Management**](operations_cost_management.md) — Infrastructure, license, and AI usage cost breakdown
- :material-docker: [**Local Development**](operations_getting_started_local.md) — Evaluate with the free AGPL community image
- :material-email-outline: [**Contact**](contact.md) — Sales, support, and private offer requests
---
## Logging & Monitoring
Source: https://stdapi.ai/md/operations_logging_monitoring.md
# :material-chart-line: Logging & Monitoring
stdapi.ai provides production-grade observability out of the box. Track request performance, debug issues, monitor costs, and ensure compliance with structured JSON logging and OpenTelemetry integration.
**Why this matters:**
- **Troubleshoot issues fast** - Structured logs with request IDs, timings, and error details
- **Monitor costs** - Track model usage, request volume, and performance by endpoint
- **Ensure compliance** - Full audit trail with client IPs, user IDs, and request/response logging
- **Performance optimization** - Identify slow requests, high-latency endpoints, bottlenecks
- **AWS integration** - Native CloudWatch Logs, X-Ray, and service-level metrics
stdapi.ai emits structured JSON logs for every request, stream, and background task, and integrates with OpenTelemetry (OTel) for traces and metrics. This guide shows how to enable observability, read the logs, and correlate signals across systems.
- :material-clipboard-text-outline: __Structured JSON Logging__
JSON logs to STDOUT—perfect for AWS CloudWatch Logs. One event per line with all request context.
- :material-identifier: __Request Correlation__
All events for a request share the same `id` (returned as `x-request-id`). Track full request lifecycle across logs and traces.
- :material-aws: __AWS-Native Integration__
Works seamlessly with CloudWatch Logs, X-Ray, and service metrics. ECS auto-forwards STDOUT to CloudWatch.
- :material-graphql: __OpenTelemetry Tracing__
Enable `OTEL_ENABLED=true` to export spans to AWS X-Ray, Jaeger, Tempo, or any OTLP-compatible backend.
- :material-chart-line: __Performance Insights__
Track execution times, model usage, endpoint latency. Includes ready-to-use CloudWatch Logs Insights queries.
- :material-alert-decagram-outline: __Debug Mode__
Enable `LOG_REQUEST_PARAMS=true` for full request/response logging when troubleshooting issues.
---
## :material-rocket-launch: Quick Start
Set these environment variables, then restart the service (see the [Configuration Guide](operations_configuration.md) for details):
```bash
# Set minimum log level (optional, defaults to "info")
# Options: info, warning, error, critical, disabled
export LOG_LEVEL=warning
# Enable OpenTelemetry tracing
export OTEL_ENABLED=true
export OTEL_SERVICE_NAME=stdapi
# 0.0–1.0 (10% example)
export OTEL_SAMPLE_RATE=0.1
# Include request/response payloads in logs (for debugging ONLY)
export LOG_REQUEST_PARAMS=true
# Log client IP addresses (requires ENABLE_PROXY_HEADERS for real client IPs)
export LOG_CLIENT_IP=true
export ENABLE_PROXY_HEADERS=true # When behind ALB/CloudFront
```
!!! warning "Sensitive data and cost impact"
Enabling `LOG_REQUEST_PARAMS` may expose sensitive content in logs. Use only in development or during targeted troubleshooting. Redact secrets before sharing logs externally.
Additionally, logging full request/response payloads can dramatically increase log volume and costs, especially for large LLM prompts, tool calls, and generated outputs. In AWS CloudWatch Logs, ingestion and storage costs scale with log size. Prefer short retention, targeted sampling, and temporary enablement only when needed.
!!! info "Client IP Logging"
When `LOG_CLIENT_IP=true`:
- The `client_ip` field is added to request logs
- The client IP is added as `client.address` attribute to OpenTelemetry spans (when `OTEL_ENABLED=true`)
To log the real client IP address (instead of the proxy IP), also enable `ENABLE_PROXY_HEADERS=true` when running behind AWS ALB, CloudFront, or other reverse proxies. See the [Configuration Guide](operations_configuration.md#client-ip-logging) for details.
!!! tip "CloudWatch best practice"
JSON to STDOUT is optimal for CloudWatch Logs Insights. In AWS ECS, the task's log driver forwards container STDOUT to CloudWatch Logs automatically.
---
## :material-format-list-bulleted: Event Types
stdapi.ai emits five kinds of JSON events (one per line):
| Event | Description |
|:-----------------|:------------------------------------------------------------------------------------|
| `start` | Emitted once at server startup. Includes startup metadata and warnings. |
| `stop` | Emitted on graceful shutdown. Includes uptime. |
| `request` | One per HTTP request. Method, path, status, timings, and optional request/response. |
| `request_stream` | Streaming segments (SSE/audio). Indicates streaming activity and duration. |
| `background` | Background tasks correlated to the parent request. |
---
## :material-table-column: Common Fields
Each event shares core fields and may add type‑specific ones.
| Field | Applies to | Description |
|------------------------------------------:|:------------------------------------|:--------------------------------------------------------------------------------------------|
| `type` | all | One of `start`, `stop`, `request`, `request_stream`, `background` |
| `level` | all | `info`, `warning`, `error`, `critical` (controlled by `LOG_LEVEL`) |
| `date` | all | RFC3339, timezone‑aware timestamp |
| `server_id` | all | Instance identifier — on ECS: `task_id-container_name-uuid`; elsewhere: `hostname-pid-uuid` |
| `server_version` | all | Application version string |
| `error_detail` | all | Optional list of formatted exception strings |
| `id` | request, request_stream, background | Correlation ID (also returned as `x-request-id`) |
| `execution_time_ms` | request, request_stream, background | Duration of the handled block |
| `method` | request | HTTP method |
| `path` | request | Request path |
| `status_code` | request | Final HTTP status code |
| `client_ip` | request | Client IP address (if `LOG_CLIENT_IP=true`) |
| `client_user_agent` | request | When provided by client |
| `model_id` | request | Targeted model (if applicable) |
| `voice_id` | request | TTS voice (if applicable) |
| `model_regions` | request | AWS region(s) that handled the request; may contain multiple values when failover occurred |
| `request_user_id`, `request_org_id` | request | Propagated identifiers (if applicable) |
| `aws_role_session_name` | request | Session the request's model usage was billed under, with [per-user cost attribution](operations_cost_management.md#per-user-attribution) enabled; the same value AWS reports as the caller identity |
| `request_params` | request | Sanitized request payload (if `LOG_REQUEST_PARAMS=true`) |
| `request_response` | request | Sanitized response payload (if `LOG_REQUEST_PARAMS=true`) |
| `amzn_trace_id`, `apigw_request_id`, `cloudfront_request_id` | request | Edge correlation IDs copied from the `X-Amzn-Trace-Id`, `x-amz-apigw-id` and `X-Amz-Cf-Id` request headers (when present) |
| `aws_requests` | request, request_stream, background | AWS-side request IDs of downstream AWS API calls (see [AWS Service Correlation Metadata](#aws-service-correlation-metadata)) |
| `event` | background | Background operation name |
| `server_start_time_ms`, `server_warnings` | start | Startup metrics and warnings |
| `server_uptime_ms` | stop | Uptime at shutdown |
| `abandoned_background_tasks` | stop | Background tasks cancelled unfinished at shutdown, counted per kind; present only when some were, and the event is then `warning` (see [`SHUTDOWN_DRAIN_TIMEOUT`](operations_configuration.md#shutdown-drain-timeout)) |
!!! note "Understanding warnings and errors"
- For `request` events, default log levels are derived from the final HTTP status: 4xx → `warning`, 5xx → `error`. Unexpected server crashes (like HTTP 500) may appear as `critical`.
- Authentication/authorization: For security, client responses for `401` and `403` include only generic messages. Full diagnostic details are captured in server logs under `error_detail` and can be correlated via `id` (see `x-request-id`).
- `server_warnings` (on the `start` event) often highlights missing configuration and features that have been disabled as a result (for example, no S3 bucket configured disables certain image/audio features).
- `error_detail` (on any event) contains formatted exception traces and diagnostic hints, which frequently point to missing configuration, unavailable dependencies, or disabled features.
---
## :material-cash-multiple: Usage Metrics Fields
Usage is reported as a nested `usage` list on `request` / `request_stream` events. Each entry represents billed quantities for a specific **(service, model, operation, region, tier, routing, context)** combination. Zero/empty fields are omitted.
### Entry Structure
| Field | Type | Description |
|:----------------------------|:-----|:-------------------------------------------------------------------------------------------------------|
| `service` | str | AWS service/API: `bedrock-runtime`, `polly`, `transcribe`, `translate`, `comprehend` |
| `model` | str | Model identifier (e.g., `amazon.titan-embed-text-v2:0`, `amazon.transcribe`) |
| `operation` | str | Request route path (e.g., `/v1/chat/completions`, `/v1/embeddings`, `/v1/audio/speech`) |
| `region` | str | AWS region that served the request (part of the aggregation key) |
| `tier` | str | Service tier that actually served the call (AWS-reported when available, else as requested): `standard`, `flex`, `priority`, `batch` |
| `routing` | str | Serving profile, present as `"global"` (cross-region global routing) or `"latency"` (latency-optimized) |
| `context` | str | Present as `"long"` only when the call's prompt (input + cache read/write tokens) exceeded 200K tokens — billed at the long-context rate where AWS publishes one |
| `cost` | str | Exact cost as plain-decimal text (no exponent, no trailing zeros, e.g. `"0.000015"`) |
| `currency` | str | ISO currency code for `cost` |
| `costs` | dict| Per-currency exact cost text, replacing `cost`/`currency` when dimensions span multiple currencies |
| `input_tokens` | int | Real input token count (Converse, InvokeModel, embeddings) |
| `output_tokens` | int | Real output token count (Converse, InvokeModel) |
| `total_tokens` | int | Real total tokens (Converse provides this directly) |
| `cached_tokens` | int | Real cached read tokens from Converse prompt caching |
| `cache_write_tokens` | int | Real cache write tokens from Converse prompt caching |
| `cache_write_tokens_by_ttl` | dict| Cache write tokens by TTL, from Converse `cacheDetails` (e.g., `{"1h": 700, "5m": 100}`) |
| `output_images` | int | Real output image count from image generation models |
| `output_images_by_spec` | dict| Output image count keyed by `":"`, for models priced per resolution/quality |
| `output_seconds` | int | Billed output media duration in seconds (video generation, recorded at job submission) |
| `output_seconds_by_spec` | dict| Output seconds keyed by rate bucket (e.g., `{"hd": 5}`), for models priced per resolution |
| `input_seconds` | int | Real input media duration in seconds (Transcribe with a 15s minimum; audio/video embedding inputs) |
| `input_seconds_by_spec` | dict| Input seconds keyed by modality (e.g., `{"audio": 42}`), for models priced per media type |
| `input_images` | int | Input image count (multimodal embeddings, billed per image) |
| `input_images_by_spec` | dict| Input image count keyed by rate bucket (e.g., `{"document": 1}`) |
| `input_characters` | int | Real input character count (Polly `RequestCharacters`, Translate source text) |
| `comprehend_units` | int | Real Comprehend units (100-char units, 3-unit minimum per call) |
| `grounding_requests` | int | Built-in grounding tool invocations (e.g., Amazon Nova Grounding `web_search`, billed per request) |
| `search_units` | int | Rerank search units (one per rerank query) |
| `text_units` | int | Bedrock Guardrails text units (1,000 characters each), as reported by ApplyGuardrail, per applied policy |
!!! warning "Guardrail cost coverage"
Guardrail units are recorded only on routes that enforce the guardrail through the ApplyGuardrail API, which reports the units each policy consumed. Chat routes use Bedrock's native guardrail integration, whose responses carry no unit counts, so their guardrail cost is absent from these entries even though AWS charges it — see [route coverage](operations_configuration.md#route-coverage).
!!! note "Usage field placement"
- **Non-streaming requests**: Usage appears on the `request` event
- **Streaming requests**: Usage appears on the `request_stream` event (not `request`) — this includes streaming speech-to-text
- **Polly / Translate**: Always on `request` event
- Cost dashboards must aggregate **both** entry types by request `id`
!!! note "Zero values are omitted"
Fields are only populated when non-zero. Query for present fields rather than filtering by `> 0`. Missing fields = zero.
!!! note "Client disconnect behavior"
- **Chat (Converse) streams**: if the client disconnects before the trailing `metadata.usage` event arrives, the wrapper keeps draining the already-open Bedrock stream (bounded) looking for it, so the usage already billed by AWS is still recorded.
- **InvokeModel streams and image generation jobs**: if a stream/job is abandoned before its final `invocationMetrics` chunk (or job completion) arrives, no usage is produced and logged.
!!! note "No estimates"
All values are real AWS-billed quantities. No estimation is performed — if AWS doesn't return a count, the field is absent (zero).
---
## :material-chart-line: CloudWatch Metrics (EMF)
When `CLOUDWATCH_METRICS=true` is enabled, usage is emitted as CloudWatch Embedded Metric Format (EMF) lines to stdout. These are automatically extracted as CloudWatch Metrics on ECS with no extra IAM/API calls.
### Configuration
| Setting | Default | Description |
|:----------------------------------|:---------|:-------------------------------------------|
| `CLOUDWATCH_METRICS` | `false` | Enable/disable EMF emission |
| `CLOUDWATCH_METRICS_NAMESPACE` | `stdapi` | CloudWatch namespace for the metrics |
### EMF Line Structure
Each billed usage entry generates one EMF JSON line:
```json
{
"_aws": {
"Timestamp": 1748716800000,
"CloudWatchMetrics": [{
"Namespace": "stdapi",
"Dimensions": [["Model"]],
"Metrics": [
{"Name": "InputTokens", "Unit": "Count"},
{"Name": "OutputTokens", "Unit": "Count"}
]
}]
},
"Model": "anthropic.claude-sonnet-5",
"operation": "/v1/chat/completions",
"InputTokens": 1500,
"OutputTokens": 450
}
```
### Metric Details
- **Single dimension**: `Model` — provides low cardinality for cost control. A second `[Model, Currency]` dimension set is added for the `Cost` metric — see [Cost Management](operations_cost_management.md#cost-tracking-real-time-aws-pricing) for why cost needs its own dimension set.
- **`operation`**: Included as a queryable field (not a dimension)
- **Metric units**: `Count` for all token/image/character counts, `Seconds` for audio duration
!!! tip "Querying EMF metrics"
Use CloudWatch Logs Insights to search EMF lines:
```sql
fields @timestamp, Model, InputTokens, OutputTokens
| filter _aws.CloudWatchMetrics is not null
| sort @timestamp desc
| limit 20
```
!!! note "Performance"
EMF lines bypass log-level filtering and write directly to stdout. This ensures metrics are always available on ECS where CloudWatch Agent scrapes stdout.
---
## :material-currency-usd: Cost Tracking and Attribution { #cost-tracking-real-time-aws-pricing }
Real-time cost computation (`COST_TRACKING`), the price catalog and its accuracy caveats, the `Cost` EMF metric, and AWS-side cost attribution now live on their own page.
[:octicons-arrow-right-24: Cost Management](operations_cost_management.md)
---
## :material-link-variant: Correlating Logs and Traces
- Group events by `id` to reconstruct a full request lifecycle (request → stream(s) → background).
- The `x-request-id` response header exposes the same value so external systems can propagate correlation.
- With OTel enabled, a root span named like `POST /v1/...` is created and carries attributes: `http.method`, `http.url`, `http.user_agent`, `request.id`, `server.id`, `http.status_code`, and `duration_ms`.
!!! note "Anthropic-compatible routes use a different header name"
On `/anthropic/...` routes, the same correlation value is returned as `request-id` instead of `x-request-id`, following Anthropic's own convention.
!!! tip "Do and Don't for correlation"
- Do propagate `x-request-id` across client → service → downstreams when possible.
- Do use `request_stream` durations to account for total user‑perceived latency.
- Don't generate your own request IDs for the same hop; prefer the provided one.
---
## :material-magnify: Reading the Logs
- High latency: Inspect `execution_time_ms` on the `request` event. If the response was streamed, also sum `request_stream` durations. Combine with OTel spans to locate downstream delays (model provider, S3, etc.).
- Errors: Look for `level=critical` and `error_detail` (formatted exceptions). With OTel, the span is marked error with attributes `error=true` and `error.message`.
- Payload issues: Temporarily enable `LOG_REQUEST_PARAMS=true` to validate requests/responses, then disable.
- Client identification: `client_user_agent` and optional `request_user_id` / `request_org_id` help tie requests to users.
- Routing confirmation: `model_id` and `voice_id` confirm which provider/model/voice handled the request.
!!! warning "When to open a GitHub issue"
If you encounter `level=critical` events, capture representative JSON log lines (redacting sensitive data) and open an issue on [GitHub](https://github.com/stdapi-ai/stdapi.ai/issues). Include information about the failing request to help reproduce the issue.
---
## :material-filter: Controlling Log Verbosity
The `LOG_LEVEL` environment variable controls which log events are written to STDOUT. Set it to filter out lower-severity events. For detailed configuration options, see the [Logging Level](operations_configuration.md#logging-level) section in the Configuration Guide.
- **`info`** (default): All events are logged (info, warning, error, critical)
- **`warning`**: Only warnings and higher severity (warning, error, critical) - recommended for production
- **`error`**: Only errors and critical events
- **`critical`**: Only critical events
- **`disabled`**: No log output (not recommended)
```bash
# Production example: reduce log volume while maintaining visibility
export LOG_LEVEL=warning
```
!!! tip "Reducing CloudWatch Costs"
In high-traffic production environments, setting `LOG_LEVEL=warning` or `LOG_LEVEL=error` can significantly reduce CloudWatch Logs ingestion and storage costs by filtering out routine `info`-level events. This is especially effective when combined with appropriate retention policies.
Additionally, infrastructure routes are automatically excluded from logging to reduce noise: `/`, `/docs`, `/favicon.ico`, `/health`, `/openapi.json`, `/ping`, `/redoc`, `/robots.txt`, `/.well-known/api-catalog`, and `/.well-known/mcp/server-card.json`.
---
## :material-graphql: OpenTelemetry Integration
When `OTEL_ENABLED=true`:
- A span is created per request and for streaming/background blocks.
- Spans carry `request.id` and `server.id` for correlation.
- 4xx/5xx `status_code` marks the span with an error status.
- Sampling is controlled via `OTEL_SAMPLE_RATE`.
For exporters and advanced setup, rely on standard OTel environment variables supported by your exporter/backend.
!!! note "No X-Ray IAM permissions needed"
Export is push-based OTLP/HTTP to the endpoint you configure — the application makes no direct AWS X-Ray API calls and needs no extra IAM permissions for tracing. To forward traces to AWS X-Ray, point the OTLP endpoint at an [ADOT collector](https://aws-otel.github.io/docs/introduction), which handles the X-Ray upload using its own credentials.
---
## :material-code-json: Example Events
__Example — Request with payload logging enabled__
```json
{
"type": "request",
"level": "info",
"date": "2025-01-01T12:00:00Z",
"server_id": "stdapi-1",
"id": "a1b2c3d4",
"method": "POST",
"path": "/v1/chat/completions",
"status_code": 200,
"model_id": "anthropic.claude-sonnet-5",
"execution_time_ms": 842,
"request_params": {"messages": [{"role": "user", "content": "..."}]},
"request_response": {"id": "cmpl_...", "choices": [...], "usage": {...}}
}
```
__Example — Streaming segment (SSE/audio)__
```json
{
"type": "request_stream",
"level": "info",
"date": "2025-01-01T12:00:01Z",
"server_id": "stdapi-1",
"id": "a1b2c3d4",
"execution_time_ms": 1234
}
```
__Example — Background work correlated to a request__
```json
{
"type": "background",
"level": "info",
"date": "2025-01-01T12:00:02Z",
"server_id": "stdapi-1",
"id": "a1b2c3d4",
"event": "image-upload-s3",
"execution_time_ms": 97
}
```
__Example — Error with captured details__
```json
{
"type": "request",
"level": "critical",
"date": "2025-01-01T12:00:05Z",
"server_id": "stdapi-1",
"id": "e9f0a1b2",
"method": "POST",
"path": "/v1/images/edits",
"status_code": 500,
"error_detail": ["Traceback (most recent call last): ..."],
"execution_time_ms": 12
}
```
---
## :material-text-search: CloudWatch Logs Insights Queries
These examples assume JSON logs in CloudWatch Logs (default with ECS awslogs/awsfirelens). Adjust the log group and time range.
### 1) Follow a specific request across request/stream/background
```sql
fields @timestamp, type, level, path, event, status_code, execution_time_ms
| filter id = ""
| sort @timestamp asc
```
!!! tip
Copy the request ID from the `x-request-id` response header or any `request` log line. Expect one `request`, optional `request_stream` entries, and `background` entries.
### 2) Find recent errors with context
```sql
fields @timestamp, level, type, path, status_code, id, error_detail
| filter level in ["error", "critical"]
| sort @timestamp desc
| limit 100
```
### 3) High-latency endpoints (P95/P99)
```sql
fields path, execution_time_ms
| filter type = "request" and ispresent(execution_time_ms)
| stats pct(execution_time_ms, 95) as p95_ms, pct(execution_time_ms, 99) as p99_ms, avg(execution_time_ms) as avg_ms by path
| sort p95_ms desc
```
---
## :material-aws: AWS Service-Level Logs and Metrics
Beyond stdapi.ai logs and OTel traces,
use AWS-native signals from the underlying AI services to validate provider behavior,
monitor throttling/latency, and audit access.
Enable only what you need: some options can capture content and increase costs.
For details, refer to the official AWS documentation.
- CloudWatch Metrics: Throughput, latency, throttling, and error rates per service/region.
- CloudTrail: Control-plane auditing of API calls (who did what, when, from where).
- Content/Invocation logging: Optional features that may record inputs/outputs. Use with caution and encryption/retention controls.
- Correlation: Service logs won't include stdapi.ai's `x-request-id`. Correlate by time window, region, model/voice/job identifiers, and volume. Use stdapi.ai's `model_id`, `voice_id`, and `execution_time_ms` to narrow windows.
- Amazon Bedrock Invocation logging (optional): Export invocation metadata and, if enabled, content to CloudWatch Logs/S3/Firehose. Treat prompts/completions as sensitive; manage retention and KMS.
### AWS Service Correlation Metadata
stdapi.ai attaches correlation identifiers to outgoing service calls, allowing you to trace a stdapi.ai request back to the corresponding AWS service invocation.
| Key | Description |
|:-----------------------|:-----------------------------------------------------------------------------------------------------------------------------------|
| `stdapi-ai.request_id` | Matches the `id` field in stdapi.ai logs and the `x-request-id` response header |
| `stdapi-ai.server_id` | Matches the `server_id` field in stdapi.ai logs |
| `stdapi-ai.user_id` | Present only when the client supplies a user identifier — the `safety_identifier` field (or the deprecated `user` alias) in OpenAI requests, `metadata.user_id` in Anthropic requests |
Coverage and how to use it varies by service:
- **Bedrock — synchronous inference** (every chat, embedding, image and audio route served by `bedrock-runtime`): identifiers are embedded in the invocation request — both the `Converse` and the `InvokeModel` families, streaming included — and appear in Bedrock invocation log records when [model invocation logging](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html) is enabled. Filter by `requestMetadata.stdapi-ai\.request_id` in CloudWatch Logs Insights to join a Bedrock record with its stdapi.ai event.
- **Bedrock — asynchronous batch jobs**: identifiers are attached as resource tags on the job, visible in the AWS Console and searchable via the CLI/API.
- **Bedrock Mantle**: the Mantle endpoint accepts no request metadata. Attribute those requests with a [Project/Workspace](operations_configuration.md#bedrock-mantle-project) instead.
- **Transcribe — audio transcription**: identifiers are attached as job tags. stdapi.ai deletes completed transcription jobs automatically, so tags are only available while the job is still running.
Correlation also works in the other direction — from a stdapi.ai log event to the exact AWS API calls it made:
- **`aws_requests`** (on `request`, `request_stream` and `background` events): one entry per downstream AWS API call, with `service`, `operation`, `request_id` and — for failed calls — the AWS `error` code. The `request_id` value is the AWS-side request ID (`ResponseMetadata.RequestId` / `x-amzn-requestid` header): the identifier AWS Support asks for, and the `requestID` field of CloudTrail records. Each event keeps at most the 50 most recent calls.
- **Edge headers** (on the `request` event, when present): `amzn_trace_id` (`X-Amzn-Trace-Id`, injected by ALB and used by X-Ray), `apigw_request_id` (`x-amz-apigw-id`, API Gateway) and `cloudfront_request_id` (`X-Amz-Cf-Id`, CloudFront) tie the gateway's `id` to the edge service's own access logs. They are recorded as-is for correlation and never replace the gateway-generated request ID.
!!! note "Security"
Any `stdapi-ai.*` key supplied by the client is silently dropped before the service call. Only values injected by stdapi.ai itself are forwarded under that prefix.
---
## :material-cloud-check: Infrastructure Observability
!!! info "Terraform Module"
The Terraform module automatically configures production-ready observability for all infrastructure components — no manual setup required.
**Application Logs (ECS)**
Container STDOUT is forwarded to a dedicated CloudWatch Logs log group. Log retention defaults to **365 days** (`cloudwatch_logs_retention_in_days`). All log groups are **KMS-encrypted**.
**Container Insights**
ECS Container Insights is enabled by default (`container_insight = "enabled"`), providing CPU, memory, network, and storage metrics per task. Set `container_insight = "enhanced"` to enable enhanced observability with additional OS-level and application performance metrics.
**CloudWatch Alarms**
The module can trigger a CloudWatch alarm whenever an `error` or `critical` log event is detected. Enable with `alarms_enabled = true` and provide `sns_topic_arn` to receive notifications via SNS (email, Slack, PagerDuty, etc.). When enabled, a metric filter scans the ECS log group for any line containing `error` or `critical` and fires the alarm as soon as the count exceeds zero in a 5-minute window.
**ALB and WAF Logs**
ALB access logs and WAF logs are stored in dedicated S3 buckets, **KMS-encrypted**. These capture all HTTP requests at the infrastructure level — use them to audit traffic patterns and investigate security events before they reach the application.
---
## :material-wrench: Troubleshooting Checklist
- No logs visible: Ensure you are reading container STDOUT. On ECS/Kubernetes, verify the log driver and retention.
- Missing `request_params`: Confirm `LOG_REQUEST_PARAMS=true` and restart after changing environment variables.
- No traces: Verify `OTEL_ENABLED=true` and that exporters are configured and reachable.
- Correlation missed: Ensure clients read and propagate `x-request-id` for multi‑hop requests.
---
## :material-arrow-right: Next Steps
- :material-cash-multiple: [**Cost Management**](operations_cost_management.md) — Cost tracking, AWS Marketplace billing, and cost attribution
- :material-cog: [**Configuration Reference**](operations_configuration.md) — Complete list of environment variables including logging options
- :material-chart-bar: [**Data Sovereignty & Compliance**](operations_compliance.md) — Compliance-focused logging and audit requirements
- :material-server-network: [**Advanced Deployment**](operations_deploy_advanced.md) — Production deployment with observability stack
---
## Cost Management
Source: https://stdapi.ai/md/operations_cost_management.md
# :material-cash-multiple: Cost Management
Running stdapi.ai bills into three independent buckets:
| Cost | Billed by | Scales with | Covered by AWS credits |
|:------------------------------|:------------------------------------------------------------------------------|:----------------------|:-----------------------------------|
| **AI service usage** | AWS (Amazon Bedrock, Polly, Transcribe, …) **or** AWS Marketplace, per model | Requests and tokens | Only the AWS-billed models |
| **Gateway infrastructure** | AWS (ECS Fargate, ALB, S3, CloudWatch, …) | Running containers | Yes |
| **Gateway license** | AWS Marketplace (Commercial License only) | Container-hours | No — Marketplace is never credited |
**AI service usage dominates any normal deployment.** The gateway runs on a small ARM64 container, so its infrastructure and license form a low fixed floor that model spend passes quickly. Read the AI-cost sections first; [Gateway Cost](#gateway-cost) matters mainly at low traffic or under a hard cost constraint.
stdapi.ai **adds no markup**: model usage is billed to you by AWS at the same rate as calling Bedrock directly.
---
## :material-tag-search: Knowing the Price Before You Call
The [Model Pricing API](api_model_pricing.md) (`GET /model_pricing`) exposes the AWS unit prices stdapi.ai resolved for each model, from the same published-price catalog it uses to estimate costs. Use it for cost-aware model selection — comparing candidates before routing traffic, rather than discovering the rate afterwards in the logs. Without a deployment, the [Models](models.md) page publishes the same rates per region and tier.
---
## :material-store: Marketplace-Billed vs AWS-Billed Models
A separate **AWS Marketplace** line on your bill for Bedrock usage is expected, not an error. AWS states it plainly:
!!! quote "[Amazon Bedrock FAQ](https://aws.amazon.com/bedrock/faqs/) — *Why do I see a billing entry for AWS Marketplace for my usage of Amazon Bedrock?*"
"Customers will see an AWS Marketplace bill for certain Bedrock serverless models and Bedrock Marketplace models. This is because these models are sold by third party providers as *Third-Party Content*, as described in the AWS service terms section 50.12."
### Why It Matters: AWS Credits
This is the practical consequence, and it is easy to get wrong when planning a budget:
!!! warning "Promotional credits do not apply to AWS Marketplace charges"
The [AWS Promotional Credit Terms](https://aws.amazon.com/awscredits/) exclude AWS Marketplace from eligible services: *"Promotional Credit will not be applied to any fees or charges for Amazon Mechanical Turk, AWS Managed Services, Ineligible AWS Support, **AWS Marketplace**, … (collectively, 'Ineligible Services')."*
A model billed through AWS Marketplace therefore **consumes real money even when your account holds AWS credits**, while the same workload on an AWS-billed model would draw them down. If you are running on credits, promotional funding, or an activate/startup programme, model choice changes your actual outlay — not just your reported spend.
The same applies to committed-spend agreements: check whether your EDP or Private Pricing Agreement counts Marketplace spend before assuming a discount applies.
### Which Models Are Which
Some models on Amazon Bedrock are sold as third-party AWS Marketplace listings; the rest bill as ordinary Amazon Bedrock usage. Open-weight models are generally in the second group. Which side a model sits on is the provider's and AWS's decision, it changes without notice, and a model can move after launch — so check your own account rather than trusting any list, this page included.
!!! tip "How to check, and what each check can prove"
- **Your invoice is the authority.** Group Cost Explorer by service, or read the AWS Marketplace line on the bill. Nothing else settles it.
- A model that needs `aws-marketplace:Subscribe` on first use **is** a Marketplace listing. AWS documents the [permission error](https://repost.aws/knowledge-center/bedrock-resolve-marketplace-permission) this produces.
- In the AWS Price List, an *"(Amazon Bedrock Edition)"* product name marks a Marketplace listing — the signal stdapi.ai's price catalog parses. It confirms one when present, but **cannot rule one out**: models do bill through Marketplace while carrying no such product name, no `MP:` usage type, and no price-list row in commercial regions at all.
### What stdapi.ai Does About It
- **Both paths are priced.** [Cost tracking](#cost-tracking-real-time-aws-pricing) ingests Marketplace listings and native Bedrock rows alike, so a request log entry carries a cost regardless of how AWS bills it.
- **Subscriptions are handled automatically.** AWS creates the Marketplace subscription on first invocation; stdapi.ai keeps a not-yet-subscribed listing in the catalogue so that call can happen, when [`AWS_BEDROCK_MARKETPLACE_AUTO_SUBSCRIBE`](operations_configuration.md#bedrock-marketplace-auto-subscribe) is enabled (the default), and hides it otherwise. It requires `aws-marketplace:Subscribe` and `aws-marketplace:ViewSubscriptions` — see [IAM Permissions](operations_configuration.md#bedrock-iam). Without them, the first call to a third-party model fails with `AccessDeniedException`.
- **Cost tracking does not separate the two lines.** Request logs report what a call cost, not which AWS invoice section it lands on. Use Cost Explorer, grouped by service, to see the Marketplace split.
---
## :material-currency-usd: Cost Tracking (Estimated from Published AWS Prices) { #cost-tracking-real-time-aws-pricing }
Cost tracking is **opt-in and off by default**. When `COST_TRACKING=true` is enabled, stdapi.ai estimates the cost of every request from AWS's published prices — an estimate computed at request time, not your billed amount read back from AWS. Costs are attributed to the actual region where each request was served.
### How It Works
1. **Price Catalog**: At startup, stdapi.ai fetches the AWS Price List API for all configured regions and services in a background task, then caches in memory. Server readiness never waits on it: requests served before the load completes simply record usage without a cost. Failed loads are retried with exponential backoff (1–15 min), and each attempt's outcome is logged as a `background` event named `price_catalog_load`
2. **On-Demand Refresh**: The catalog is refreshed whenever a newly available Bedrock model is discovered with no catalog entry yet — not on a proactive schedule. If that refresh's AWS call fails (Price List throttling, missing permissions), the failure never propagates: the errors are captured per region and service rather than raised, the triggering request completes normally, and the new model stays unpriced until a later refresh succeeds. The refresh itself emits no per-request diagnostic — the miss surfaces on first use through the unpriced-model `error_detail` described in step 5
3. **Per-Request Computation**: For each request, costs are computed by multiplying billed quantities by the resolved unit price
4. **Built-in Defaults**: A few models are not in the Price List API at all — the Stability AI Image Services, priced on the [AWS pricing page](https://aws.amazon.com/bedrock/pricing/), and OpenAI's hosted GPT models, priced on their Bedrock model cards. Those published rates ship built in, used only when AWS publishes no row for the model; `COST_PRICE_OVERRIDES` still takes precedence
5. **Fallback on a Missing Price**: Once the catalog has been fetched, if a specific model/dimension has no resolvable price in it, the cost field is omitted for that entry rather than blocking the request, and the request log carries a `warning`-level `error_detail` naming the model and unpriced dimensions (a hint to supply the missing price via `COST_PRICE_OVERRIDES`)
!!! warning "Pricing is an estimate, not a bill"
stdapi.ai resolves prices from AWS's own Price List API and does its best to match every request to the right unit price — including tier, cache TTL, cross-region routing, region fallback, and image resolution/quality where applicable. This is still a **best-effort approximation**, not a guarantee: AWS's Price List API doesn't reliably map a Bedrock model ID to its own pricing rows, some pricing dimensions aren't modeled at all (see [Known Limitations](#known-limitations)), and fallbacks (regional, tier) substitute a nearby price when the exact one isn't published. For billing-critical use, always reconcile against AWS Cost Explorer or your actual invoice.
### Configuration
| Setting | Default | Description |
|:--------|:--------|:------------|
| `COST_TRACKING` | `false` | Enable/disable per-request cost estimation (needs `pricing:GetProducts`) |
| `COST_PRICE_OVERRIDES` | `{}` | JSON map for operator-supplied prices for models not in AWS catalog |
### Request Log Format
Each usage entry includes cost and currency when resolved:
```json
{
"service": "bedrock-runtime",
"model": "anthropic.claude-sonnet-5",
"operation": "/v1/chat/completions",
"region": "us-east-1",
"input_tokens": 1500,
"output_tokens": 450,
"cost": "0.004575",
"currency": "USD"
}
```
Costs are plain-decimal strings rather than floats, so no precision is lost and no exponent or trailing zeros appear. The request-level total is also logged:
```json
{
"cost": {
"USD": "0.012345"
}
}
```
### EMF Cost Metric
When CloudWatch metrics are enabled, a `Cost` metric (unit: None) is emitted under the `["Model", "Currency"]` dimension set — a **separate directive** from the quantity metrics, which stay under `["Model"]`. A single directive spanning both sets would also publish `Cost` bare-by-`Model`, silently summing across currencies. Query `Cost` with **both** `Model` and `Currency` dimensions:
```json
{
"_aws": {
"CloudWatchMetrics": [
{
"Namespace": "stdapi",
"Dimensions": [["Model"]],
"Metrics": [{"Name": "InputTokens", "Unit": "Count"}]
},
{
"Namespace": "stdapi",
"Dimensions": [["Model", "Currency"]],
"Metrics": [{"Name": "Cost", "Unit": "None"}]
}
]
},
"Model": "anthropic.claude-sonnet-5",
"Currency": "USD",
"Cost": 0.004575,
"InputTokens": 1500
}
```
### Regional Price Fallback
Some models — mostly older/deprecated ones — aren't published in every region's Price List (e.g. priced in `us-east-1` but not any EU region). A region with no price for a given model/dimension/tier always borrows one from a nearby region instead of omitting the cost:
1. Prefers another region in the same geography (`eu-west-3` tries other `eu-*` regions first)
2. Falls back to `us-east-1`, `eu-west-1`, or `us-west-2` — the regions always fetched regardless of your configured Bedrock regions
3. If neither is available, the cost is omitted as usual
This is a substitute price, not the actual published price for that region.
### Multi-Currency Support
stdapi.ai detects currency from the AWS partition:
- Standard AWS: USD
- AWS European Sovereign Cloud (EUSC): EUR
- AWS US GovCloud: USD
- AWS China: CNY
Costs are **never summed across currencies** — this safety behavior is always on, regardless of settings. It matters when a single request's billed dimensions resolve to different currencies, which can happen with [regional fallback](#regional-price-fallback) crossing a partition boundary (e.g. a EUSC deployment falling back to a standard-AWS-priced region). A usage entry that spans more than one currency reports a `costs` map (instead of `cost`/`currency`) with every currency's own amount:
```json
{
"service": "bedrock-runtime",
"model": "anthropic.claude-sonnet-5",
"input_tokens": 1500,
"output_tokens": 450,
"costs": {
"EUR": "0.0021",
"USD": "0.0028"
}
}
```
When only one currency is involved, the entry's `cost`/`currency` reports that currency directly instead. The request-level `cost` total still aggregates cleanly per currency either way.
### Routing-Tier Pricing
AWS prices some models differently per serving profile: the cross-region "global" routing profile is *lower* than the plain/regional rate (confirmed live: Claude Sonnet 4.5 input tokens at $3.30/M regional vs $3.00/M global; GPT-5.6 Luna, Sol and Terra are each about 9% cheaper on it — the ratio is regional = global × 1.1, so global is ~9% cheaper and regional ~10% dearer, which are the same figure read in opposite directions), while latency-optimized serving (requested via the `X-Amzn-Bedrock-PerformanceConfig-Latency: optimized` header) is *higher*. stdapi.ai tracks the profile that served each request (`"routing": "global"` or `"latency"`) and prices it at the matching rate. A geographic (`us.`/`eu.`/`apac.`) profile is billed at the regional rate, which is what AWS charges for it.
Falling back is normal, not a gap: where AWS publishes no distinct global rate for a model, the request is priced at that model's regional rate rather than left uncosted. This covers models whose rates come from the AWS pricing page or a model card instead of the Price List API, too. Bedrock Mantle offers no cross-region inference at all — a Mantle request is always served in the Region it was sent to, and always billed at the In-Region rate.
### Service Tiers as a Cost Lever
AWS prices each [service tier](operations_configuration.md#default-model-service-tiers-section) differently: `flex` trades latency for a lower rate, `priority` does the opposite. Two ways to apply one without changing client code: `DEFAULT_MODEL_SERVICE_TIERS` pins a tier per model, and a [model alias](operations_configuration.md#model-aliases-configuration) pins one per alias — publishing, say, a `flex` name for batch workloads and a `priority` name for interactive ones over the same model. Set [`AWS_BEDROCK_ALLOW_SERVICE_TIER_OVERRIDE`](operations_configuration.md#aws-bedrock-allow-service-tier-override) to `false` to stop clients selecting another tier, and the cost profile you configured holds. Both settings and the override gate cover models served through the Bedrock Converse and InvokeModel APIs; a Bedrock Mantle-served model runs on the tier its own request names.
### Batch Inference { #batch-inference }
Requests sent through the [Batch API](api_openai_batches.md) or the [Message Batches API](api_anthropic_batches.md) are billed by AWS at the model's published **batch** rate, roughly half its on-demand rate. stdapi.ai prices them at that rate, reported with `"tier": "batch"` in the usage entry, and falls back to half the standard rate when AWS publishes no batch rate for a model.
Usage is recorded **once per batch**, when the batch reaches a terminal state and its totals are read — not per request. The entry therefore appears on whichever read observed the end of the batch, which is a poll rather than the call that created it. A batch that is cancelled before its requests run records nothing.
### Vector Stores { #vector-stores }
Indexing a file into a [vector store](api_openai_vector_stores.md) costs one embedding call per passage, and a search costs one per query. Those calls are recorded and priced like any other embedding usage, against the model in [`VECTOR_STORE_EMBEDDING_MODEL`](operations_configuration.md#vector-store-embedding-model) — a large file is many passages, so the cost of an attach scales with the file, not with the request.
Indexing runs after the response is sent, so its usage is reported on a `background` log event of its own rather than on the request that started it. Correlate the two through the `id` field they share.
The **storage and request charges of the vector storage itself are not recorded**: like the bytes the [Files API](api_openai_files.md) stores, they appear on your AWS bill and not in the usage log. Read them from AWS Cost Explorer, filtered on Amazon S3 Vectors — the service holding [`AWS_S3_VECTORS_BUCKET`](operations_configuration.md#aws-s3-vectors-bucket).
A deployment that sets [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url) adds Amazon SQS requests to that bill — a handful per attached file, plus one long-poll receive per idle server every 20 seconds. It adds **no embedding cost**: a job replayed after a server was replaced skips whatever already completed, so a file is never embedded twice.
A [knowledge base store](api_openai_vector_stores.md#knowledge-base-stores) is billed differently, because the retrieval happens inside Amazon Bedrock and no embedding call of this server's is involved:
- A **fully managed** knowledge base charges a flat rate per retrieval call, with the document parsing, the embedding and the reranking included in it. Every query of a search is one retrieval, recorded as one `search_units` unit under the model `amazon.bedrock-knowledge-base` and priced from the rate on the [Amazon Bedrock pricing page](https://aws.amazon.com/bedrock/pricing/) — the Price List API publishes no row for it, so that rate is built in and can be replaced through `COST_PRICE_OVERRIDES`.
- A knowledge base whose **vector storage you provisioned yourself** has no published per-retrieval rate: its search costs whatever its embedding model and its vector database charge, both billed outside this server's view. Nothing is recorded for it rather than a figure that would be invented.
- **Attaching a file records nothing, on either kind.** AWS publishes no per-document ingestion charge — a fully managed knowledge base includes parsing and embedding in its rates and bills its index storage monthly per GB of raw data, and the other kind bills the embedding model and the parser it is configured with.
Read all of these from AWS Cost Explorer, filtered on Amazon Bedrock.
### Long-Context Pricing
Some 1M-context-capable models are billed by AWS at a higher rate — roughly double for input-side tokens — once a call's prompt (input + cache read/write tokens) passes the boundary that model publishes. **That boundary differs per model**: Claude Sonnet 4 (via the `context-1m` `anthropic-beta` flag) switches at 200K tokens, while the OpenAI GPT-5.6 models bill their short-context rate all the way to 272K. stdapi.ai applies each model's own boundary, prices the whole call at the published long-context rate, and reports it with `"context": "long"` in the usage entry. When AWS publishes no long-context rate for a model, the standard rate is used as the best available estimate.
### Built-in Tool Pricing
Amazon Nova's built-in grounding tool (`web_search` → `nova_grounding`, supported by Nova 2 and Nova Premier) is billed by AWS per grounding request, on top of token usage. stdapi.ai counts grounding invocations in the model response and reports them as `grounding_requests`, priced from the model's published per-request rate.
The Amazon Bedrock web search tool the OpenAI GPT-5.x family uses on [`/v1/responses`](api_openai_responses.md#openai-gpt-web-search) is billed per query at one flat rate — the same for every model — in every Region where that tool is available. Because AWS publishes it without a model, it is reported on its own usage entry under the model `amazon.bedrock-web-search`, also as `grounding_requests`, beside the invocation's own token entry. A single turn may run several queries, and one tool call may issue more than one, so the count comes from the queries the response reports rather than from the number of tool calls; page reads have no published per-query rate and are not counted.
### Image Pricing Granularity
Some image-generation models (currently: Amazon Titan Image Generator G1/V2 and Nova Canvas) are priced by AWS per resolution/quality combination, not a flat per-image rate. stdapi.ai automatically prices these per-image, based on the actual requested size and quality, falling back to a flat per-image rate for models where this isn't wired up yet (e.g. Stability). No configuration needed — this is purely additive precision with no accuracy trade-off.
### Media Input Pricing
Multimodal embedding inputs are billed by AWS per media unit on top of (or instead of) tokens: per input image (with a distinct "document image" rate where offered) and per second of audio/video. stdapi.ai records image counts directly and audio/video durations from the AWS-reported segment timings of the asynchronous (segmented) processing path, reported as `input_images`/`input_seconds` with their `*_by_spec` breakdowns. Rerank queries are recorded as `search_units` (one per query).
### Known Limitations
- **Speech-modality tokens** (speech-to-speech models, including [Realtime API](api_openai_realtime.md#cost) sessions): speech and text tokens are recorded in separate buckets and priced at their own published rates, so no modality is priced as the other. Where a model publishes no speech rate at all, its speech tokens fall back to the text rate, which underestimates speech-heavy calls. A Realtime session bills continuously in both directions for as long as it is open, not per request; usage is recorded per answer, matching each `response.done` event in the request log.
- **Asynchronous (segmented) embeddings**: AWS reports no token usage for this processing path (used automatically for large inputs), so segmented text embeddings report no token cost; audio/video durations are recovered from the AWS-reported segment timings and billed.
- **Synchronous audio/video embedding inputs**: media duration is only available from AWS on the segmented (asynchronous) processing path — small audio/video inputs processed synchronously report no duration and no per-second cost. No estimate is substituted (this app only reports AWS-confirmed real usage).
- **Client disconnect during streaming**: streamed chat responses still record their final usage after a disconnect. For streamed responses on other routes and for image generation jobs, AWS bills the input tokens and everything generated up to the cancellation, but no usage is recorded for that call. No estimate is substituted.
- **Rerank queries with more than 100 documents**: AWS bills one search unit per 100 documents; the document count isn't visible at recording time, so one unit per query is recorded.
- **Reserved capacity pricing**: if a request explicitly asks for AWS's Reserved Capacity service tier, its cost is computed at the standard on-demand rate instead — Reserved Capacity uses a separate monthly-commitment pricing model this app doesn't ingest. Avoid relying on this app's cost figures for Reserved Capacity workloads.
- Some very new or region-specific models may have no published price anywhere yet — AWS publishes pricing after model availability, sometimes with a delay.
- **AWS GovCloud**: the Price List API has no GovCloud endpoint, so catalog prices cannot be fetched there — usage is still recorded, a startup warning is emitted, and only `COST_PRICE_OVERRIDES` entries produce costs.
### Override Map for Missing Models
Some models — recently released ones, or region-specific listings — may not appear yet in the AWS Price List API. Use `COST_PRICE_OVERRIDES` to fill gaps (Bedrock models only — Polly/Transcribe/Translate/Comprehend prices always come from the catalog):
```bash
export COST_PRICE_OVERRIDES='{"anthropic.claude-sonnet-5":{"input_tokens":0.000003,"output_tokens":0.000015}}'
```
Prices are per **one unit** (token, character, second) in your partition's currency.
### IAM Permissions
Ensure your IAM role includes pricing read access (the Price List API serves identical data from its three commercial endpoints, and sovereign partitions such as EUSC host their own; the nearest one is selected automatically from your configured Bedrock regions):
```json
{
"Effect": "Allow",
"Action": ["pricing:GetProducts"],
"Resource": "*"
}
```
---
## :material-tag-multiple: AWS Cost Attribution
Cost tracking prices **each request** as it happens. AWS-side attribution answers a different question — whose spend lands on the **AWS bill**, in [Cost Explorer](https://docs.aws.amazon.com/cost-management/latest/userguide/ce-what-is.html) and [CUR 2.0](https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html). The two are complementary: request logs for granular analysis, AWS attribution for invoicing and chargeback.
| Dimension | Mechanism | Reported in | Setup |
|:------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------|:------------------------------------------------------------------|
| **Service / gateway** | [IAM principal attribution](https://docs.aws.amazon.com/bedrock/latest/userguide/cost-mgmt-iam-principal-tracking.html) — AWS captures the caller identity | Cost Explorer, CUR 2.0 | Automatic; tag the execution role for finer breakdowns |
| **Application / workload** | [Bedrock Project/Workspace](operations_configuration.md#bedrock-mantle-project) (Bedrock Mantle models) | Cost Explorer, CUR 2.0 | Set `AWS_BEDROCK_MANTLE_PROJECT` |
| **End user** | `stdapi-ai.user_id` request metadata and job tags | stdapi.ai logs, Bedrock invocation logs | Clients send `safety_identifier` — `user` is a deprecated alias — (OpenAI) or `metadata.user_id` (Anthropic) |
| **End user, on the AWS bill** | [Per-user role sessions](#per-user-attribution) — each user's model calls run under a session of their own | Cost Explorer, CUR 2.0 | Set `AWS_BEDROCK_USER_ROLE_ARN` to a role you create |
| **Team / tenant** | Request metadata attached by the [model alias](operations_configuration.md#model-aliases-configuration) the client names | Bedrock model invocation logs only | Give each team its own alias with a `metadata` entry, then [enable and deliver model invocation logging](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html) |
!!! warning "Alias metadata is not a cost allocation tag"
A model alias' `metadata` travels as Amazon Bedrock **request metadata**: it appears in [model invocation logs](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html), which you enable and deliver to S3 or CloudWatch Logs yourself, and which it can be filtered on. It never appears in the stdapi.ai request logs, in Cost Explorer or in CUR 2.0 — splitting the bill by team still needs one of the mechanisms above.
### IAM Principal Attribution
Amazon Bedrock records the IAM identity behind every `bedrock-runtime` inference call and forwards it to Cost Explorer and CUR 2.0. Nothing to configure in stdapi.ai, and no change to client requests.
By default stdapi.ai calls AWS with a **single execution role** — the ECS task role — so all Bedrock spend is attributed to that one identity, the standard LLM-gateway pattern AWS documents. To break that total down, tag the execution role (for example `team` or `cost-center`), then activate those keys in the AWS Billing console under **Cost allocation tags**, filtering by type **IAM principal**. To split it per end user instead, see [Per-User Attribution](#per-user-attribution) below.
!!! note "Attribution grain"
IAM principal attribution aggregates per usage type per day — it never yields a per-request cost. Use the [request logs](#cost-tracking-real-time-aws-pricing) for that.
### Per-User Attribution
Every end user is attributed at the request level out of the box: the client-supplied identifier is recorded as `stdapi-ai.user_id` in the request log — alongside that request's computed cost — and forwarded to Bedrock as request metadata, so per-user spend can be aggregated from the logs.
To split the **AWS bill itself** per end user, give each one an identity of their own. Set `AWS_BEDROCK_USER_ROLE_ARN` to a role you create, and stdapi.ai opens a short-lived session of that role per end user and runs their model calls under it. AWS then reports each user separately in Cost Explorer and CUR 2.0, from the invoice rather than from the logs.
=== ":material-cog: Configure"
| Setting | Purpose |
|:--------|:--------|
| [`AWS_BEDROCK_USER_ROLE_ARN`](operations_configuration.md#aws-bedrock-user-role-arn) | The role each end user's calls run under. Enables the feature. |
| [`AWS_BEDROCK_USER_ROLE_TAG_KEY`](operations_configuration.md#aws-bedrock-user-role-tag-key) | Session tag key carrying the user identity (`user` by default). |
| [`AWS_BEDROCK_USER_ROLE_SESSION_DURATION`](operations_configuration.md#aws-bedrock-user-role-session-duration) | Session lifetime, 900–3600 seconds. Sessions are cached and reused. |
| [`AWS_BEDROCK_USER_ROLE_REQUIRE_IDENTITY`](operations_configuration.md#aws-bedrock-user-role-require-identity) | Reject requests that identify no end user, instead of billing them to the gateway. |
The role and the two IAM policies it needs are in [IAM Permissions](operations_iam_permissions.md#per-user-cost-attribution).
=== ":material-account-check: Identify the user"
The identity is taken, in order:
1. the **authenticated caller**, when [Amazon Cognito authentication](operations_authentication_security.md) is enabled — the identity the gateway itself verified;
2. the identifier the request declares: `safety_identifier` (or the deprecated `user`) on the OpenAI-compatible APIs, `metadata.user_id` on the Anthropic Messages API.
A request carrying neither runs under the gateway's own identity, unless `AWS_BEDROCK_USER_ROLE_REQUIRE_IDENTITY` is enabled, in which case it is rejected with a `400`. That rejection covers the requests that would run under the end user role, and only those — the services listed in the coverage warning below stay on the gateway's identity either way, with the one exception noted there. Some APIs — audio transcription among them — have no end user field, so there the identity can only come from an authenticated caller.
!!! danger "A declared identifier is chosen by the caller"
The identity is verified only when it comes from an authenticated caller. With an API key, or with no authentication, any client can send any other user's `safety_identifier`, and the resulting session — its name, its tag, its line in the bill — is that other user's. This is cost metadata, not an authorization boundary: see [Restricting a role per end user](operations_iam_permissions.md#per-user-cost-attribution) before writing an IAM policy on the session tag.
=== ":material-magnify: Read the bill"
Each user appears as a distinct caller identity in [CUR 2.0](https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html)'s `line_item_iam_principal` column, which holds the full ARN — `arn:aws:sts:::assumed-role//`. Two operator steps in the AWS console:
- In **Data Exports**, enable *Include caller identity (IAM principal) allocation data* under **Additional export content** — on a new CUR 2.0 standard data export, or on an existing one through **Edit** ([editing export details](https://docs.aws.amazon.com/cur/latest/userguide/dataexports-edit-export-details.html): the report name and Billing view are fixed, the export content is not).
- To group by user in Cost Explorer, activate the session tag key under **Billing → Cost allocation tags**, filtering by type **IAM principal**; it is then offered under **Group by → Tag** as `iamPrincipal/`. The key appears there only after that identity has made at least one call, and AWS takes up to 24 hours to list it plus up to 24 hours to activate it.
The request log's `aws_role_session_name` field records the session each request was billed under, which is what correlates a log line with a CUR row.
!!! warning "What is covered, and what is not"
Per-user sessions apply to **model invocations**, and to the guardrail applied during them. Everything else the gateway calls on your behalf — standalone guardrail evaluations, reranking, video generation jobs and their output files, speech, transcription and translation — stays on the gateway's own identity. The one exception is a real-time speech-to-speech session, which is refused rather than billed to the gateway once [`AWS_BEDROCK_USER_ROLE_REQUIRE_IDENTITY`](operations_configuration.md#aws-bedrock-user-role-require-identity) is enabled. Bedrock Mantle requests are attributed with [Projects](operations_configuration.md#bedrock-mantle-project) instead.
!!! note "Cardinality"
AWS multiplies CUR rows by the number of calling identities, and aggregates them per usage type per day. A deployment with a very large or unbounded user population gets a proportionally larger export, and still no per-request cost — the [request logs](#cost-tracking-real-time-aws-pricing) remain the per-request source.
---
## :material-server: Gateway Cost
Secondary for most deployments — read this section if you run under a hard cost constraint or at low traffic, where the gateway's fixed floor is a visible share of the bill.
### Infrastructure
The gateway is a single small container. The Terraform module defaults to **ARM64, 0.25 vCPU and 512 MiB** — the smallest Fargate size — which keeps this bucket cheap, though it is still a floor you pay whether or not any request arrives.
The [Terraform module](operations_getting_started.md) provisions:
| Component | Notes |
|:----------------------------------|:-------------------------------------------------------------------------|
| **ECS Fargate** service | The dominant infrastructure cost; 0.25 vCPU / 512 MiB ARM64 by default, times `autoscaling_min_capacity` |
| **Application Load Balancer** | Hourly rate plus LCU; skipped when you attach your own |
| **S3 buckets** (regional) | Input files, generated media; see `AWS_S3_VIDEOS_EXPIRES_AFTER` |
| **S3 vector bucket** (optional) | Created with `aws_s3_vectors_bucket_create`; billed on stored vectors and on the bytes each search reads — see [Vector Stores](#vector-stores) |
| **SQS queue** + dead-letter queue (optional) | Created with `aws_sqs_vector_store_queue_create`; a handful of requests per attached file, plus one long poll per task every 20 seconds |
| **CloudWatch** logs, metrics, alarms | Grows with log verbosity and EMF metric volume |
| **KMS**, IAM, networking | Keys and roles; NAT/VPC endpoints only when the module creates the network |
| **WAF** (optional) | Per-rule and per-request charges when `alb_waf_enabled = true` |
#### Cost Tiers at a Glance
Both infrastructure and license cost scale with **how many containers run and for how long** — there's no single "deployment cost," just a wide range between a minimal setup and a full production one. Three representative configurations:
| Tier | Configuration | You pay for | Main cost driver |
|:-----|:--------------|:-------------|:-------------------|
| **Minimal** | 1 ARM64 task, [Fargate Spot](operations_deploy_advanced.md#cost-optimized-deployment) (~70% cheaper than on-demand), [scheduled service hours](#keeping-it-low), no ALB — reached via [Service Discovery](operations_deploy_advanced.md#integration-with-existing-infrastructure) | Spot compute only for the scheduled hours; license for those same hours; no ALB/WAF charges at all | Hours the schedule keeps the service running — see [License](#license) for a worked example (~$17/month at ~165 h/month) |
| **Standard** | 1 on-demand task, single AZ (`subnet_ids` limited to one subnet or `autoscaling_min_capacity = 1`), ALB enabled, running 24/7 | On-demand compute for one container around the clock, plus the ALB's hourly rate and LCU usage | The ALB becomes a fixed cost on top of continuous compute; no Spot discount |
| **Full production** | One task per Availability Zone (module default), ALB + WAF, running 24/7 | Compute × number of AZs, ALB + WAF, and the per-container-hour license × total task-hours across all AZs | AZ count is the main multiplier — it scales infrastructure and license cost together |
None of these include AI service usage (Bedrock, Polly, Transcribe, …), which is billed separately by AWS at cost — see the buckets table at the top of this page.
#### Keeping It Low
- **Stop the service outside business hours.** `autoscaling_schedule_stop` and `autoscaling_schedule_start` take cron expressions, so the cluster can run weekdays only and cost nothing at night — the single largest saving for an internal workload.
```hcl
autoscaling_schedule_stop = "cron(0 19 ? * MON-FRI *)"
autoscaling_schedule_start = "cron(0 8 ? * MON-FRI *)"
```
- **Reuse existing network infrastructure.** Passing `subnet_ids` and `security_group_id` deploys into your VPC and creates **no additional NAT gateways or load balancers** — see [Integration with Existing Infrastructure](operations_deploy_advanced.md). On a small deployment these are often larger than the gateway itself.
- **Scale to your real floor.** `autoscaling_min_capacity` sets how many containers run at idle. Because the Commercial License is billed per container-hour, this number drives both infrastructure *and* license cost.
- **Use Fargate Spot** and trim log retention, as the [Cost-Optimized Deployment](operations_deploy_advanced.md#cost-optimized-deployment) example does.
- **Trim observability volume.** Raising `LOG_LEVEL` and disabling request/response payload logging cut CloudWatch ingestion — see [Controlling Log Verbosity](operations_logging_monitoring.md#controlling-log-verbosity). Note that **EMF metric lines are not affected by `LOG_LEVEL`**: they are written to stdout on every request whenever `CLOUDWATCH_METRICS` is on, so disabling that setting is the only way to remove them.
!!! note "Estimating before you deploy"
stdapi.ai publishes no infrastructure dollar estimate — the tiers above describe direction and drivers, not quotes: the total depends on your region, replica count, schedule and whether the module creates networking. Price the component list above with the [AWS Pricing Calculator](https://calculator.aws/) for your own configuration.
---
### License
stdapi.ai is [dual-licensed](operations_licensing.md): the free AGPL-3.0-or-later license costs nothing but requires sharing your modifications, while the Commercial License — billed per container-hour via AWS Marketplace, no per-request or per-token component — is what production deployments typically run. See [Licensing](operations_licensing.md) for the full pricing table and trial/private-offer terms.
!!! info "The license is itself a Marketplace charge"
Being an AWS Marketplace subscription, the Commercial License appears on the Marketplace section of your bill and — like [Marketplace-billed models](#why-it-matters-aws-credits) — **cannot be paid with AWS promotional credits**.
---
## :material-arrow-right: Next Steps
- :material-chart-line: [**Logging & Monitoring**](operations_logging_monitoring.md) — Usage metrics fields, CloudWatch EMF, and log verbosity
- :material-tag-search: [**Model Pricing API**](api_model_pricing.md) — Query AWS unit prices per model for cost-aware selection
- :material-cog: [**Configuration Reference**](operations_configuration.md) — `COST_TRACKING`, `COST_PRICE_OVERRIDES` and IAM permissions
- :material-scale-balance: [**Licensing**](operations_licensing.md) — AGPL vs commercial license and AWS Marketplace subscription
- :material-email-outline: [**Contact**](contact.md) — Private offers, committed usage, and billing questions
---
## Resilience & Failover
Source: https://stdapi.ai/md/operations_resilience.md
# :material-shield-check: Resilience & Failover
stdapi.ai on AWS is designed for high availability at every layer — from intelligent multi-region request routing to the underlying infrastructure running the service. This page covers both the application-level region routing for Amazon Bedrock and the infrastructure resilience built into the Terraform module.
---
## :material-directions-fork: Region Routing
stdapi.ai can automatically distribute Bedrock requests across your configured AWS regions. When a region becomes temporarily unavailable or hits quota limits, eligible failures are retried in another enabled region — no client changes needed.
!!! tip "Each Region Contributes Its Own Quota"
Bedrock quotas are per-region: every region you enable adds its own independent tokens-per-minute and requests-per-minute limits, so a multi-region deployment draws on multiple independent quotas rather than one. How much of that headroom a given workload reaches depends on the quotas granted per model in each region and on the routing strategy in use.
!!! info "What failover covers"
Retrying in another region is conditional, and the carve-outs matter more than the happy path:
- **Synchronous** requests retry across regions within the same request, each candidate region tried at most once.
- **Streaming** requests can only fail over **before** the stream opens; once bytes are flowing the region is locked.
- **Asynchronous** jobs select a region at job start and do not move.
- **Requests carrying S3 inputs** are pinned to a single region and do not fail over at all.
See [Failover Scope](#failover-scope) and [S3-Aware Region Selection](#s3-aware-region-selection) for the full behaviour.
### :material-information-outline: Overview
Region routing activates when you have **two or more regions** in `AWS_BEDROCK_REGIONS`. The server tracks the health of each region per model and steers traffic away from regions that are returning errors.
- :material-alert-circle-outline: __Quota & Throttling__
Triggers on `ThrottlingException`, `TooManyRequestsException`, `ServiceQuotaExceededException`
- :material-server-off: __Regional Unavailability__
Triggers on `ServiceUnavailableException`, `InternalServerException`, `ModelNotReadyException`
- :material-timer-sand: __Exponential Backoff__
Quota errors: delay doubles per consecutive error, capped at 1 hour
- :material-timer-outline: __Fixed Backoff__
Unavailability errors: fixed configurable delay, default 30 s
- :material-rotate-right: __Configurable Retry Count__
Set `AWS_BEDROCK_MAX_RETRIES` to cap the attempts per request; each region is tried at most once
---
### :material-swap-horizontal: Routing Strategies
Set the strategy with `AWS_BEDROCK_REGION_ROUTING`:
| Strategy | Description | Prompt Caching | Default |
|---|---|---|---|
| `ordered` | Try regions in the order listed in `AWS_BEDROCK_REGIONS`, skipping any that are currently blocked | :material-check: Compatible | :material-check: Yes |
| `lowest_latency` | Prefer the region with the lowest measured round-trip latency | :material-check: Compatible | |
| `round_robin` | Distribute requests evenly across available regions | :material-close: Not compatible | |
| `disabled` | No routing; each model uses its primary region only | :material-check: Compatible | |
#### Ordered (default)
Regions are tried in the order they appear in `AWS_BEDROCK_REGIONS`. The first healthy region wins. This is the best choice when you want predictable routing and prompt caching, since requests for a given model tend to land on the same region as long as it is healthy.
#### Lowest Latency
At startup the server measures round-trip latency to each region and prefers the fastest one. If that region becomes blocked, the next-fastest is used. Good for latency-sensitive workloads where you want the server to pick the closest region automatically.
#### Round Robin
Requests rotate evenly across healthy regions. This maximizes aggregate throughput when you need to spread load, but is **incompatible with prompt caching** because consecutive requests for the same model may land on different regions.
---
### :material-cog-outline: Configuration
```bash
# Required: at least two regions
export AWS_BEDROCK_REGIONS=us-east-1,us-west-2,eu-west-1
# Strategy (default: ordered)
export AWS_BEDROCK_REGION_ROUTING=ordered
# Cap on retries across regions per request (default: 9, i.e. 10 attempts)
# Each region is tried at most once, so with 3 regions a request makes at most 3 attempts: r1, r2, r3
export AWS_BEDROCK_MAX_RETRIES=9
# Enable adaptive retry mode — dynamically throttles back retries under congestion (default: false)
export AWS_ADAPTIVE_RETRY=false
# How long to avoid a region after a quota/throttling error (seconds, default: 60)
# This is the base value — the actual delay doubles on each consecutive quota error,
# up to a hard ceiling of 1 hour.
export AWS_BEDROCK_REGION_ROUTING_QUOTA_BACKOFF_SECONDS=60
# Hard ceiling on quota backoff per region (seconds, default: 3600 = 1 hour)
export AWS_BEDROCK_REGION_ROUTING_MAX_QUOTA_BACKOFF_SECONDS=3600
# Factor × max quota backoff after which the consecutive-error counter resets (default: 2)
export AWS_BEDROCK_REGION_ROUTING_QUOTA_STALE_FACTOR=2
# How long to avoid a region after an unavailability error (seconds, default: 30)
export AWS_BEDROCK_REGION_ROUTING_UNAVAILABLE_BACKOFF_SECONDS=30
```
!!! tip "Single-Region Deployments"
With only one region configured, routing is automatically disabled regardless of the strategy setting.
---
### :material-transit-connection-variant: How It Works
1. **Model discovery** — At startup, stdapi.ai discovers which models are available in each configured region.
2. **Region selection** — When a request arrives, the router picks the best region for that model based on the active strategy and current region health.
3. **Automatic failover** — For synchronous and streaming requests without S3 inputs, the retry loop walks the regions in priority order and stops once every candidate has been tried, or once `AWS_BEDROCK_MAX_RETRIES` retries are spent — whichever comes first. A region is never attempted twice within the same request: it is still blocked by the backoff its failure just recorded, and a second error there would only deepen that backoff. All retryable errors escalate to the next region immediately, except a read timeout ([`AI_RESPONSE_TIMEOUT`](operations_configuration.md#ai-response-timeout)): the model has already been invoked and is billed by AWS whatever the client does, so the request fails with a `503` rather than paying a second region for the same generation. When S3 inputs are present, the region is pinned and botocore's adaptive retries handle resilience within that region (see [S3-Aware Region Selection](#s3-aware-region-selection)).
4. **Backoff tracking** — Regions that produce errors are temporarily deprioritized. Quota errors use exponential backoff (base interval doubles per consecutive error, capped at 1 hour); unavailability errors use a fixed backoff. Once the backoff expires, regions rejoin the rotation.
5. **Client-side backoff hint** — If every attempt is exhausted, the resulting `429` response carries a `retry-after` header set to the shortest quota backoff applied during the request, i.e. the delay after which the first blocked region rejoins the rotation. OpenAI, Anthropic and Cohere SDKs honour it natively, so clients wait exactly as long as needed instead of applying a blind exponential backoff — note that all three cap a server-supplied delay at 60 s and fall back to their own backoff beyond that, so an escalated quota backoff is only partly respected. The header is omitted when no quota backoff was recorded (for example on a single-region deployment, where no routing state exists).
```mermaid
%%{init: {'flowchart': {'htmlLabels': true, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart LR
client["Your App"] -->|API request| stdapi
subgraph stdapi [" stdapi.ai"]
router["Region Router\n(strategy + health)"]
end
router -->|"region selected"| r1
subgraph aws [" Amazon Bedrock"]
r1["us-east-1"]
r2["us-west-2"]
r3["eu-west-1"]
end
r1 -->|"ThrottlingException"| router
router -->|"retry → next region"| r2
r2 -->|"✓ success"| stdapi
stdapi -->|response| client
```
#### Failover Scope
| API Style | Failover Behavior |
|---|---|
| Synchronous (Converse, InvokeModel) | Automatic failover across regions within the same request, each candidate tried once; S3-pinned requests stay on the pinned region with botocore adaptive retries |
| Streaming (ConverseStream, InvokeModelWithResponseStream) | Failover across regions **before** the stream opens, each candidate tried once; once streaming begins the region is locked. S3-pinned requests stay on the pinned region with botocore adaptive retries. |
| Asynchronous (StartAsyncInvoke) | Region is selected once at job start; no mid-job failover |
---
### :material-text-box-search-outline: Logging
Every request log includes a `model_regions` field (a set) showing which AWS region(s) handled the request. A single request may touch more than one region when failover occurs mid-request.
```json
{
"type": "request",
"model_id": "anthropic.claude-sonnet-5",
"model_regions": ["us-east-1"],
...
}
```
!!! info "Elevated Log Level on Failover"
When a region is skipped due to a quota or unavailability error, the request log level is elevated to warning so these events are visible even when filtering for warnings only.
---
### :material-bucket-outline: S3 Data Handling
Many Bedrock operations accept S3 URIs as input (e.g. images, PDFs) or produce S3 output (e.g. async invocations). stdapi.ai includes several features to handle S3 data seamlessly across regions.
#### S3-Aware Region Selection
When a request references S3 data, the router takes the data location into account:
- **S3 inputs present** — All S3-sourced input files for the request are tracked and their regions are ranked by **descending total data volume**. Only the **single best region** is used — the retry loop is pinned to it. This is required because S3 content blocks are resolved once for a specific region and cannot be re-resolved for a different one; retrying on another region would send a cross-region S3 reference that Bedrock cannot access. If none of the S3 input regions are regions where the model is available, the router falls back to the first model region that has a configured S3 bucket (the object will be copied there before invocation). If no such bucket region exists either, the request is rejected with an error.
- **S3 required, no S3 inputs** — Operations that need an S3 bucket (e.g. async invocations) restrict candidates to model regions that have a configured S3 bucket. If no region has a bucket, the request is rejected with an error.
- **No S3 constraint** — All regions where the model is available are considered.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart TD
req["Incoming Request"] --> check{"S3 inputs\npresent?"}
check -->|"Yes"| rank["Rank S3 regions\nby data volume"]
rank --> overlap{"Model available\nin any S3 region?"}
overlap -->|"Yes"| pin["Pin to single\nbest region"]
overlap -->|"No"| bucketed{"Model region\nwith S3 bucket?"}
bucketed -->|"Yes"| pin_bucket["Pin to first bucketed\nmodel region\n(object will be copied)"]
bucketed -->|"No"| err["❌ Error: no viable region\nfor model + S3 inputs"]
check -->|"No"| s3req{"S3 bucket\nrequired?"}
s3req -->|"Yes"| s3cap["Model regions\nwith S3 bucket only"]
s3cap --> empty{"Any found?"}
empty -->|"No"| err2["❌ Error: no region\nhas a configured bucket"]
empty -->|"Yes"| multi["Multi-region\ncandidates"]
s3req -->|"No"| multi
pin --> invoke["Invoke Bedrock"]
pin_bucket --> invoke
multi --> invoke
```
!!! warning "No Cross-Region Failover with S3 Inputs"
When S3 input files are present, the region is locked before the request is made. If that region is throttled or unavailable, the request fails rather than retrying on another region with a stale S3 reference.
#### S3 HTTP URL to S3 URI Conversion
If a user passes an S3 HTTP URL (including presigned URLs) as input, stdapi.ai automatically converts it to an `s3://` URI when the bucket is recognized. This avoids unnecessary HTTP round-trips and allows Bedrock to access the object directly.
Recognized buckets include:
- The application's own buckets (`AWS_S3_BUCKET` and `AWS_S3_REGIONAL_BUCKETS`)
- Any bucket listed in `AWS_S3_ACCEPTED_BUCKETS`
Both virtual-hosted style (`https://bucket.s3.region.amazonaws.com/key`) and path-style (`https://s3.region.amazonaws.com/bucket/key`) URLs are supported.
#### Cross-Region S3 Copy
When the selected Bedrock region differs from the region where the input S3 object resides, stdapi.ai copies the object to a bucket in the target region before invoking the model. The copy uses server-side copy for objects up to 5 GiB and multipart copy for larger objects.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
input[" s3://bucket-us-east-1/file"]
copy["Server-side copy\n≤5 GiB: single copy\n>5 GiB: multipart"]
dest[" s3://bucket-us-west-2/file"]
bedrock[" Bedrock us-west-2"]
input -->|"selected region ≠ object region"| copy
copy --> dest
dest --> bedrock
```
#### Accepted S3 Buckets
You can declare external S3 buckets that the application has read access to. These buckets are then recognized for S3 HTTP URL conversion and region-aware routing:
```bash
export AWS_S3_ACCEPTED_BUCKETS='{"my-data-bucket": "us-east-1", "my-eu-bucket": "eu-west-1"}'
```
Keys are bucket names, values are the AWS region where each bucket resides.
#### Regional S3 Buckets
Asynchronous invocations require an S3 bucket in the same region as the Bedrock endpoint. When routing is enabled, configure regional buckets so the router can place async jobs in any eligible region:
```bash
export AWS_S3_REGIONAL_BUCKETS='{"us-east-1": "my-bucket-use1", "us-west-2": "my-bucket-usw2"}'
```
!!! info "Terraform Module"
When using the Terraform module, regional S3 buckets are configured automatically. Manual `AWS_S3_REGIONAL_BUCKETS` configuration is only needed for direct deployments.
!!! note
If a region has no configured bucket, it is excluded from async invocation routing but remains available for synchronous and streaming requests.
---
### :material-map-marker-radius-outline: Model Region Restrict
You can restrict specific models to a fixed set of regions. This is useful when a model offers important features only in certain regions (e.g. Nova grounding is only available in `us-east-1`):
```bash
export AWS_BEDROCK_MODEL_REGION_RESTRICT='{"amazon.nova-pro-v1:0": ["us-east-1"]}'
```
Keys are Bedrock model IDs (or prefixes). Values are ordered lists of allowed regions. The model is **only** made available in those regions—no fallback to other regions occurs. The order of the list determines the routing priority when multiple regions are listed (with the default `ordered` routing strategy).
---
### :material-swap-horizontal: Deprecated Model Fallback
When a client sends a request using a model ID that has been retired or superseded, stdapi.ai can transparently reroute it to the recommended replacement — no client changes needed.
This is controlled by [`AWS_BEDROCK_DEPRECATED_MODEL_FALLBACK`](operations_configuration.md#bedrock-deprecated-model-fallback) (default: `true`).
#### How It Works
1. On a cache miss, the deprecation registry is consulted for a replacement.
2. If the replacement is itself deprecated, the chain is followed until a live model is found or the chain ends.
3. If a live replacement is found, the request proceeds with it. A **warning** is recorded in the request log and the log level is elevated to `warning` so the event is visible in monitoring.
4. If no live model is found at the end of the chain, a `404` is returned naming both the original deprecated ID and the last replacement tried.
!!! warning "`AWS_BEDROCK_LEGACY` — Use with caution"
Setting `AWS_BEDROCK_LEGACY=true` forces stdapi.ai to keep serving legacy (end-of-life) models. AWS may deny requests to such models with an access error if you have not been actively using the model recently, causing failover to break silently. Only set this option if using a legacy model is absolutely required.
#### Legacy Model Warnings
Using a **legacy** model (one AWS has scheduled for end-of-life) also emits a `warning`-level log entry, including the EOL date when known:
```text
Model 'anthropic.claude-haiku-4-5-20251001-v1:0' is legacy and will reach end-of-life on 2027-06-19. Please migrate to a supported model.
```
Models whose EOL date falls within the current cache window are **proactively excluded** at cache refresh time, so they are never served to clients even if AWS has not yet removed them from the available models list. See [which models are marked legacy](models.md).
#### Strict Mode
Set `AWS_BEDROCK_DEPRECATED_MODEL_FALLBACK=false` to disable the fallback. Requests using a deprecated model ID will fail with a `404` that includes the recommended replacement, forcing clients to update their code explicitly:
```text
The model `amazon.titan-text-lite-v1` does not exist or you do not have access to it. This model is deprecated or pending deprecation, please use 'amazon.nova-lite-v1:0' instead. Call the models endpoint to list the models this server provides.
```
#### Extending the Registry
The built-in deprecation registry covers all models listed in the [Amazon Bedrock model lifecycle](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle.html). Use [`AWS_BEDROCK_DEPRECATED_MODELS`](operations_configuration.md#bedrock-deprecated-models) to add custom mappings or override existing ones.
---
### :material-account-voice: Other AWS Services Failover
Amazon Polly, Transcribe, Translate, and Comprehend follow the same regional-failover pattern as Bedrock. Each service has its own region setting — [`AWS_POLLY_REGION`](operations_configuration.md#aws-polly-region), [`AWS_TRANSCRIBE_REGION`](operations_configuration.md#aws-transcribe-region), [`AWS_TRANSLATE_REGION`](operations_configuration.md#aws-translate-region), [`AWS_COMPREHEND_REGION`](operations_configuration.md#aws-comprehend-region) — and when left unset, every region in `AWS_BEDROCK_REGIONS` becomes a candidate, tried in order with automatic failover on region-level errors:
- **Polly** — voice availability is discovered per engine (Standard, Neural, Long-form, Generative) across all candidate regions at startup; each synthesis call routes to a region offering the requested engine and voice.
- **Transcribe** — candidate regions are restricted to those with a co-located S3 bucket ([`AWS_TRANSCRIBE_S3_BUCKET`](operations_configuration.md#aws-transcribe-s3-bucket) or a regional bucket in [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets)); on a region-level error the audio is copied to the next candidate's bucket and the job restarts there.
- **Translate** and **Comprehend** — calls try each candidate region in order and fail over on throttling, service unavailability, or network errors.
When several regions are candidates, per-region SDK retries are capped by [`AWS_FAILOVER_MAX_RETRIES`](operations_configuration.md#failover-max-retries) so that failover across regions replaces deep in-region retrying.
Setting an explicit region for any of these services pins it to that single region, disabling failover.
### :material-rocket-launch-outline: Fault-Tolerant Startup
A Bedrock region that cannot be reached at startup (invalid region for the account, network issue, throttling) does not block the server from starting: it is skipped with an `unreachable_bedrock_regions` warning, its models are served from the remaining regions, and the region is retried automatically on the next model list refresh ([`MODEL_CACHE_SECONDS`](operations_configuration.md#model-cache-seconds)). Startup only fails when **every** configured region is unreachable, or when **every** per-model availability check errors — see [Unreachable Region Tolerance](operations_configuration.md#aws-bedrock-regions).
---
## :material-server-network: Infrastructure Resilience
The Terraform module deploys stdapi.ai following AWS best practices for high availability and fault tolerance. Every component is designed to handle failures transparently — no additional configuration required.
- :material-view-module: __Multi-AZ Fargate Tasks__
ECS tasks spread across all Availability Zones; a single AZ failure does not interrupt service
- :material-autorenew: __Stateless Service Design__
stdapi.ai keeps no state on disk — a failed task is replaced without loss of stored data, since all persistent data lives in S3
- :material-heart-pulse: __ALB Health Checks__
Unhealthy tasks drained and replaced within seconds; traffic rerouted to healthy AZs automatically
- :material-earth: __Bedrock Cross-Region Inference__
Bedrock-native routing across AWS regions provides an extra failover layer on top of stdapi.ai's own [region routing](#region-routing)
- :material-database-check: __S3 Eleven-Nines Durability__
99.999999999% object durability; regional buckets co-located with each Bedrock endpoint
- :material-rocket-launch-outline: __Fast Task Startup__
New tasks become healthy in under 30 seconds, minimizing the recovery window after any failure
- :material-update: __Zero-Downtime Updates__
Rolling deployments and ALB connection draining let in-flight requests finish on the outgoing task before it is deregistered
```mermaid
%%{init: {'flowchart': {'htmlLabels': true, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart TB
client["Your App"]
subgraph deployment["AWS Region (ECS deployment)"]
alb[" ALB + WAF"]
b_local[" Amazon Bedrock"]
s3r[" S3"]
subgraph az_a["Availability Zone A"]
ecs_a[" stdapi.ai ECS Fargate"]
end
subgraph az_b["Availability Zone B"]
ecs_b[" stdapi.ai ECS Fargate"]
end
end
subgraph br2["Bedrock Region 2"]
b2[" Amazon Bedrock"]
bs2[" Regional S3"]
end
subgraph brn["Bedrock Region N"]
bn[" Amazon Bedrock"]
bsn[" Regional S3"]
end
client -->|"HTTPS"| alb
alb --> ecs_a & ecs_b
ecs_a & ecs_b --> s3r
ecs_a & ecs_b --> b_local
ecs_a & ecs_b -.->|"region routing"| b2
ecs_a & ecs_b -.->|"region routing"| bn
b_local -.-|"cross-region inference"| b2
b2 -.-|"cross-region inference"| bn
```
### :material-view-dashboard-variant: Multi-AZ & ECS Service Resilience
**Stateless by design.** stdapi.ai stores no state on disk — all persistent data lives in S3. Each ECS Fargate task is fully replaceable: ECS can terminate and relaunch a failed task without any loss of stored data, and a request interrupted by a task replacement is one the client can retry.
**Multi-AZ spread.** The Terraform module places ECS tasks across all available Availability Zones in the region. If an AZ experiences a partial or full failure, tasks in the remaining AZs continue to process requests without interruption. The default configuration maintains at least one task per Availability Zone, so capacity remains in the other AZs during a task replacement event.
**Auto-scaling.** Task count scales automatically based on CPU utilization, memory utilization, and ALB request count — whichever metric signals pressure first. Fargate Spot is optionally available for cost-sensitive deployments — see [Cost-Optimized Deployment](operations_deploy_advanced.md#cost-optimized-deployment) for the trade-offs.
!!! info "Terraform Module"
Minimum capacity defaults to the number of deployed Availability Zones (one task per AZ, via `autoscaling_min_capacity`). Maximum capacity is configurable (`autoscaling_max_capacity`, default: `null` — uses the AWS Application Auto Scaling default). Auto-scaling targets CPU and memory utilization as well as ALB request count per target, so the service scales out under any of these pressure signals.
**Fast startup.** The stdapi.ai container image is optimized for minimal startup time — a new task typically becomes healthy in under 30 seconds. Fast startup is critical for recovery: when ECS detects a failed task it launches a replacement immediately, keeping the degraded window short and ensuring the service restores full capacity without manual intervention.
**Zero-downtime updates.** ECS rolling deployments start the new container version and wait for it to pass health checks before draining the old task. The ALB connection draining period lets in-flight requests complete on the outgoing task before it is deregistered, so an application update does not cut off calls that are already under way — provided they finish within the draining window.
#### :material-tray-arrow-down: Work That Outlives Its Request { #vector-store-indexing }
Vector store [indexing](api_openai_vector_stores.md#indexing-is-asynchronous) runs in the background rather than on the request path, so a task replaced while it is in flight interrupts it. A task asked to stop finishes what it can first — under [`SHUTDOWN_DRAIN_TIMEOUT`](operations_configuration.md#shutdown-drain-timeout) — but that grace period is short by design, since ECS sends `SIGKILL` 30 seconds after `SIGTERM` by default, so it is a courtesy rather than a guarantee. Nothing is stranded when it runs out:
- A file left `in_progress` with nothing indexing it any more is settled as `failed`, with `last_error` saying the indexing was interrupted, the next time the file, the store, or its file list is read. Attach the file again to index it — no store is left reporting `in_progress` for good, and no client polls forever.
- Deleting a file from a vector store removes its passages from the index **before** the record that names them, so a task lost mid-delete leaves the deletion to be finished by the next read rather than leaving content searchable. Either way the file stops being searchable and stops being listed the moment the API answers.
**Hand the work to a queue instead.** Set [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url) and indexing stops depending on the task that accepted it: another task finishes the job.
- Attaching a file records the work on your Amazon SQS queue **before the response is sent**, once every record it names is already durable in S3. There is no window in which the client has been told the file is attached and the work has not been handed over.
- Every task reads that queue, so the fleet you already run is the pool of consumers — no extra service, no extra container-hour, and consumer redundancy across Availability Zones for free.
- A task killed mid-job never confirms the message, so Amazon SQS hands it to another task, which finishes it. Work already completed is not redone, so a recovery costs no extra embeddings.
- A file the gateway genuinely cannot index is retried a bounded number of times, then reported as `failed` exactly as an unqueued deployment reports it, and its message is kept in your dead-letter queue.
- A task busy serving requests does not take jobs off the queue: indexing yields to the clients that are waiting.
The queue needs a **standard** queue with a dead-letter queue behind it, and the [durable indexing permissions](operations_iam_permissions.md#durable-vector-store-indexing) on it. The Terraform module provisions both when `aws_sqs_vector_store_queue_create` is set.
### :material-connection: ALB Resilience
The Application Load Balancer is a fully managed, natively multi-AZ AWS service:
- **Cross-AZ load balancing** — traffic is distributed evenly across tasks in all healthy AZs.
- **Health check integration** — the ALB polls `/health` on each task; a task is removed from rotation after consecutive failed health checks and readded as soon as it recovers.
- **WAF protection** — when enabled, WAF sits in front of the ALB and mitigates DDoS and rate-limit abuse before requests reach the application.
!!! info "Terraform Module"
- **Load balancing algorithm** — uses `weighted_random` with **anomaly mitigation enabled**, automatically reducing traffic sent to tasks exhibiting elevated error rates before they are fully drained.
- **Idle timeout** — set to **3600 s (1 hour)** (`alb_idle_timeout`) to accommodate long-running streaming LLM responses. Without a sufficiently large timeout, the ALB may terminate connections mid-stream for slow or large generations.
!!! tip "ALB is not a single point of failure"
AWS manages ALB node redundancy across AZs automatically. An AZ failure reduces capacity but does not take the load balancer offline.
### :material-earth: Bedrock Cross-Region Inference
Amazon Bedrock supports [cross-region inference profiles](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html), which allow Bedrock to automatically route a model invocation to another AWS region when the primary region is throttled or temporarily unavailable. stdapi.ai enables cross-region inference by default (`AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=true`).
This creates two complementary failover layers:
| Layer | Where it operates | When it triggers |
|---|---|---|
| **stdapi.ai region routing** | Application level — across your configured regions | Quota exceeded, throttling, regional unavailability |
| **Bedrock cross-region inference** | Bedrock service level — transparent within AWS | Bedrock-internal capacity events |
Together, they maximize model availability without any client-side changes.
!!! note "Compliance-aware cross-region inference"
Set `AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false` to restrict Bedrock to region-local inference, ensuring data stays within a specific geography (e.g. EU-only for GDPR compliance). See [Data Sovereignty & Compliance](operations_compliance.md) and the [GDPR deployment example](operations_deploy_advanced.md#production-deployment-fully-featured).
### :material-bucket-outline: S3 Resilience
S3 stores multimodal inputs and outputs (images, PDFs, audio) used by Bedrock operations:
- **99.999999999% (11 nines) object durability** — data is stored redundantly across multiple devices and AZs within a region.
- **99.99% availability SLA** — designed for continuous availability with no planned downtime.
- **Regional buckets** — for multi-region deployments, each Bedrock region has a dedicated S3 bucket co-located in the same region. This eliminates cross-region data transfer for async and multimodal operations and satisfies data residency requirements.
---
### :material-shield-star: Ultimate Multi-Region Deployment
For the highest possible resilience, deploy two independent stdapi.ai stacks in separate AWS regions and connect them with **AWS Global Accelerator**. Global Accelerator routes each client to the **nearest healthy region** using geographic proximity — both regions are active simultaneously. If one region's ALB fails health checks, GA automatically reroutes its traffic to the other region within seconds.
Additional Bedrock regions (without ECS) can be added to `AWS_BEDROCK_REGIONS` in each stack to expand model availability and quota without deploying more ECS infrastructure.
!!! tip "No dedicated sample yet"
There is no ready-to-use Terraform example for this two-stack + Global Accelerator topology. Start from the [Production Deployment](operations_deploy_advanced.md#production-deployment-fully-featured) module configuration and deploy it twice — once per region — then add Global Accelerator in front of both.
**What this adds on top of a single-region deployment:**
| Component | Single region | Multi-region + GA |
|---|---|---|
| ECS Fargate | Multi-AZ in one region | Multi-AZ in **two** regions |
| ALB | One ALB | One ALB per region |
| Entry point | ALB DNS name | **Single Anycast IP via Global Accelerator** |
| Traffic routing | — | Geographic proximity (nearest region wins) |
| Regional failover | None | Automatic, within seconds |
| Bedrock quota | One region's quota | Multiple independent quotas |
```mermaid
%%{init: {'flowchart': {'htmlLabels': true, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart LR
client["Your App"]
ga[" Global Accelerator"]
subgraph region_a["AWS Region A"]
direction TB
alb_a[" ALB + WAF"]
ecs_a[" stdapi.ai ECS Fargate"]
b_a[" Amazon Bedrock"]
s3_a[" S3"]
end
subgraph region_b["AWS Region B"]
direction TB
alb_b[" ALB + WAF"]
ecs_b[" stdapi.ai ECS Fargate"]
b_b[" Amazon Bedrock"]
s3_b[" S3"]
end
subgraph region_c["AWS Region C (Bedrock only)"]
direction TB
b_c[" Amazon Bedrock"]
s3_c[" Regional S3"]
end
client -->|"HTTPS"| ga
ga -->|"geo-routing"| alb_a
ga -->|"geo-routing"| alb_b
alb_a --> ecs_a
alb_b --> ecs_b
ecs_a --> b_a & s3_a
ecs_b --> b_b & s3_b
ecs_a & ecs_b -.->|"region routing"| b_c
```
**How Global Accelerator integrates:**
- **Geographic proximity routing** — GA resolves each client to the nearest AWS region over the public internet, then carries the traffic over the AWS backbone to the ALB in that region. Both regions serve live traffic simultaneously.
- **Health-based failover** — GA continuously health-checks each ALB endpoint. If a region's ALB stops responding, GA automatically reroutes its traffic to the other region within seconds — with no DNS TTL delay.
- **Single Anycast entry point** — clients always connect to the same two static IPs regardless of which region handles the request. No client reconfiguration is needed during a regional failure.
!!! note "API key synchronization"
Both ECS stacks must share the same API key so clients can reach either region transparently. Use `api_key_secretsmanager_secret` pointing to a cross-region replicated Secrets Manager secret, or set the same key via `api_key` in both modules.
---
## :material-lightbulb-outline: Best Practices
**Infrastructure:**
- :material-terraform: **Use the Terraform module** — The [stdapi-ai Terraform module](operations_getting_started.md#quick-start) provisions all resilience features out of the box: multi-AZ ECS, ALB health checks, auto-scaling, WAF, and CloudWatch alarms. Deploying manually risks missing critical settings.
- :material-earth: **Run at least two Bedrock regions** — Configure `aws_bedrock_regions` with two or more regions so the deployment draws on more than one independent Bedrock quota and eligible failures can retry elsewhere. A single region is a single point of failure for quota limits. Each additional region is also a cost driver — see [Cost Management](operations_cost_management.md).
**Region routing:**
- :material-check: **Start with `ordered`** — It provides failover without sacrificing prompt caching.
- :material-speedometer: **Use `lowest_latency`** only if your server's network position varies or you want the fastest region chosen automatically.
- :material-rotate-right: **Use `round_robin`** for high-throughput batch workloads where prompt caching is not needed.
- :material-timer-check-outline: **Keep backoff values moderate** — The defaults (60 s for quota, 30 s for unavailability) work well for most workloads. Very short backoffs may cause premature retries against a region that is still overloaded.
- :material-counter: **Tune `AWS_BEDROCK_MAX_RETRIES`** — The default of 9 exceeds any realistic region count, so a routed request already tries every candidate region once. Lower it (e.g. `2`) to give up after fewer regions; raising it only deepens in-region retrying for single-region and S3-pinned requests.
- :material-pulse: **Consider `AWS_ADAPTIVE_RETRY`** — Enable this when many concurrent clients share the same endpoint and sustained congestion is likely. It paces retries based on real-time error signals, reducing the risk of retry storms — at the cost of potentially higher per-request latency under load. Avoid it for latency-sensitive or low-traffic workloads.
- :material-magnify: **Monitor `model_regions` in logs** — If one region consistently appears in error logs, consider adjusting its quota or removing it from the region list.
- :material-bucket-outline: **Declare accepted buckets** — If your users provide S3 URLs from buckets outside the application's own buckets, add them to `AWS_S3_ACCEPTED_BUCKETS` so the router can resolve their region and convert HTTP URLs to S3 URIs.
- :material-pin-outline: **Pin models when needed** — Use `AWS_BEDROCK_MODEL_REGION_RESTRICT` for models that have region-specific features (e.g. grounding) so requests for that model are only served where the feature exists. The model will be restricted exclusively to the listed regions.
- :material-swap-horizontal: **Plan for model deprecations** — Keep `AWS_BEDROCK_DEPRECATED_MODEL_FALLBACK=true` (the default) so clients survive AWS model retirements without downtime. Switch to `false` in environments where you want to enforce explicit client migrations.
---
## :material-arrow-right: Next Steps
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy to AWS with two Terraform commands
- :material-server-network: [**Advanced Deployment**](operations_deploy_advanced.md) — Multi-region Terraform examples with resilience configured
- :material-cog: [**Configuration Reference**](operations_configuration.md) — All routing and failover environment variables
- :material-shield-lock: [**Data Sovereignty & Compliance**](operations_compliance.md) — GDPR-compliant region configuration
- :material-cash-multiple: [**Cost Management**](operations_cost_management.md) — What each additional region and task costs
- :material-email-outline: [**Contact**](contact.md) — Discuss a multi-region or high-availability deployment
---
## Data Sovereignty & Compliance
Source: https://stdapi.ai/md/operations_compliance.md
# :material-shield-lock: Data Sovereignty & Compliance
stdapi.ai is deployed entirely within your AWS account. Model inference, data storage, and service calls run on the AWS services and regions you explicitly configure — no third party sits between your users and your models, and the gateway contacts no vendor endpoint of its own.
!!! success "What this means for your organization"
- **No vendor endpoint in the request path** — no third party sits between your users and your models; inference runs on the AWS services and regions you enable
- **Amazon Bedrock does not train on your data** — prompts and completions are not used for model training, and model providers have no access unless you explicitly enable `provider_data_share`
- **AWS services carry enterprise compliance certifications** — GDPR, ISO 27001/27017/27018, SOC 1/2/3, HIPAA, FedRAMP (Moderate and High), PCI-DSS, and more via Amazon Bedrock
- **Those certifications are not inherited** — AWS compliance certifications apply to the AWS services and regions you choose; they are not inherited by stdapi.ai or by your application
- **All data encrypted in transit and at rest** — TLS 1.2+ on all AWS service calls; the Terraform module additionally configures the ALB with TLS 1.2+ with TLS 1.3 and post-quantum key exchange enabled, and Customer Managed KMS keys for all stored data
- :material-map-marker-check: __Region-Locked Processing__
Every AWS service call (Bedrock, S3, Polly, Transcribe, Comprehend, Translate) is restricted to your configured regions
- :material-shield-key: __No Third-Party Egress__
The application initiates no third-party calls of its own — only AWS services. The exceptions are all under your control: remote URLs your own clients supply (SSRF-guarded), OTLP trace export when you enable telemetry, a Mantle endpoint you override, and web searches Bedrock runs when a client asks for a grounding tool
- :material-lock: __Data in Transit Encrypted__
All AWS service calls use TLS 1.2+. The Terraform module configures the ALB with TLS 1.3 and post-quantum hybrid key exchange.
- :material-database-off: __No Persistent State on Compute__
ECS containers hold no user data — all persistent storage lives in your S3 buckets, encrypted at rest
!!! success ":material-check-decagram: AWS Compliance Certifications"
All AWS services used by stdapi.ai (Bedrock, Polly, Transcribe, Comprehend, Translate) are in scope for **ISO 27001/27017/27018** and the full AWS ISO certification suite. Amazon Bedrock additionally covers **SOC 1/2/3**, **HIPAA**, **GDPR**, **FedRAMP** (Moderate and High), **PCI-DSS**, and **CSA STAR Level 2**. Amazon Comprehend and Polly are also **HIPAA**-eligible.
Third-party audit reports (SOC, PCI, ISO, etc.) can be downloaded directly from **AWS Artifact** — no need to request them manually. See [Compliance validation for Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/compliance-validation.html) for the current list of in-scope programs, [AWS Compliance Programs](https://aws.amazon.com/compliance/programs/) for the full catalogue, and [AWS Services in Scope](https://aws.amazon.com/compliance/services-in-scope/) to verify certifications per service.
---
## :material-application-outline: Application Data Flow
The diagram below shows exactly where data flows and what is retained at each step:
```mermaid
%%{init: {'flowchart': {'htmlLabels': false}} }%%
flowchart TD
client["Client request"] -->|"HTTPS (TLS 1.2+, TLS 1.3 and PQ key exchange enabled)"| alb["ALB (your AWS account)"]
alb -->|"HTTP (private VPC)"| ecs["ECS container\n(in-memory only, stateless)"]
ecs -->|"HTTPS — inference"| bedrock["Amazon Bedrock\n(no prompt retention)"]
ecs -->|"HTTPS — temp files"| s3["Amazon S3\n(your bucket, TTL = request duration)"]
ecs -->|"HTTPS — metadata only"| cw["Amazon CloudWatch\n(no prompt content by default)"]
ecs -.->|"HTTPS — when used"| ai["Polly / Transcribe\nComprehend / Translate"]
ecs -.->|"when a client passes a public\nfile URL (SSRF-validated)"| web["Public web"]
bedrock -.->|"when a client requests\na web search tool"| web
```
**What this means:**
- **ECS container** — holds request data in memory only; stateless between requests; no disk writes
- **Amazon Bedrock** — processes the inference and returns the result; does not retain prompts ([AWS source](https://docs.aws.amazon.com/bedrock/latest/userguide/data-protection.html))
- **Amazon S3** — temporary storage for multimodal inputs/outputs (images, audio, PDFs); files are deleted immediately after the request completes — see [S3 Data Storage](#s3-data-storage) for the lifecycle-policy failsafe
- **Amazon CloudWatch** — receives structured request metadata (method, path, status, model, latency); prompt and response content are **never logged by default** (requires `LOG_REQUEST_PARAMS=true` to enable)
- **Amazon Polly / Transcribe / Comprehend / Translate** — used only when audio or translation features are invoked; see [AI service opt-out](#aws-ai-service-improvement-opt-out) for data retention controls
- **Public web** — reached two ways, both driven by the client: the gateway fetches a file the client referenced by `http(s)` URL (destination chosen by the caller, every hop validated against the SSRF guard, private and loopback ranges refused by default), or Bedrock performs a web search on the model's behalf when the client asks for that tool. Neither happens for requests that send inline data or S3 references and no web-search tool; see the outbound paths below
stdapi.ai communicates with the seven AWS services above, all within the regions you configure, plus a handful of conditional AWS services enabled only by specific features: AWS SSM and Secrets Manager (API key storage, if configured), AWS STS (account ID lookup when the ECS task metadata endpoint is unavailable — credentials themselves come from the standard AWS credential chain), the AWS Price List API (only when `COST_TRACKING=true`), AWS Marketplace Metering (AWS Marketplace image only), `bedrock-agent-runtime` (Amazon Bedrock sessions backing `store=true` on the Responses and Chat Completions APIs, and the Rerank API), `bedrock-agent` (Bedrock Prompt Management, only when [`AWS_BEDROCK_ALLOW_PROMPT_ARN`](operations_configuration.md#bedrock-allow-prompt-arn) is enabled), and `bedrock-mantle` (the Amazon Bedrock Mantle endpoint serving OpenAI GPT, xAI Grok, Google Gemma and similar models, enabled by default via [`AWS_BEDROCK_MANTLE_ENABLED`](operations_configuration.md#bedrock-mantle-enabled)).
The server initiates no third-party calls of its own: it contacts no external API, analytics service, or vendor endpoint on its own behalf. Four outbound paths can leave AWS, each driven by your own configuration or your own clients:
- **Remote URLs supplied by a client** — when a request references an input file by `http(s)` URL, the gateway downloads that URL as instructed. The destination is chosen by the caller, never by the server, and every connection (including redirect hops) is validated against [`SSRF_PROTECTION_BLOCK_PRIVATE_NETWORKS`](operations_configuration.md#ssrf-protection-block-private-networks), enabled by default. Clients that send inline data or S3 references never trigger any outbound fetch.
- **OpenTelemetry trace export** — when [`OTEL_ENABLED`](operations_configuration.md#otel-enabled) is set to `true`, traces are exported in OTLP format to [`OTEL_EXPORTER_ENDPOINT`](operations_configuration.md#otel-exporter-endpoint), which defaults to a collector on localhost. Tracing is disabled by default, and the endpoint is yours to choose.
- **An operator-overridden Mantle endpoint** — [`AWS_BEDROCK_MANTLE_ENDPOINT_URL`](operations_configuration.md#bedrock-mantle-endpoint-url) replaces the default `https://bedrock-mantle.{region}.api.aws` address. The override must use `https`, and a `{region}` placeholder is substituted when present, but its host is whatever you set; left unset, Mantle traffic stays on the AWS endpoint in your configured regions.
- **Model-side web search** — when a client includes a web search tool in a request (Amazon Nova's `nova_grounding` system tool, or the native `web_search` tool on Claude models), Amazon Bedrock performs the search on the model's behalf and queries the public web with content derived from the prompt. The gateway issues no such request itself, and no call triggers it unless the client asks for the tool. The built-in [web search](api_openai_responses.md#openai-gpt-web-search) on the OpenAI GPT-5.x family is the one case that stays inside AWS by default: it is answered from the Amazon Bedrock web index and cache, in the Region that served the call, unless you set [`AWS_BEDROCK_EXTERNAL_WEB_ACCESS`](operations_configuration.md#bedrock-external-web-access) and grant the matching [IAM permission](operations_iam_permissions.md#web-search-iam). Amazon Nova's code interpreter (`nova_code_interpreter`) likewise executes inside Bedrock, not in your container. See [Built-in Tool Pricing](operations_cost_management.md#built-in-tool-pricing) for how these invocations are metered, and [`AWS_BEDROCK_MODEL_REGION_RESTRICT`](operations_configuration.md#bedrock-model-region-restrict) to pin the models that offer them.
### Data in Transit
| Connection | Protocol | Notes |
|---------------------|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Client → ALB | HTTPS (TLS 1.2 / TLS 1.3) | ALB HTTPS listener; supports TLS 1.3 and post-quantum hybrid key exchange ([ALB security policies](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/describe-ssl-policies.html)) |
| ALB → ECS container | HTTP | Private VPC traffic, isolated within AWS network infrastructure |
| ECS → AWS services | HTTPS (TLS 1.2+) | AWS confirms: *"Within AWS, all inter-network data in transit supports TLS 1.2 encryption"* ([source](https://docs.aws.amazon.com/bedrock/latest/userguide/data-protection.html)) |
ALB supports **TLS 1.3** and **post-quantum hybrid key exchange** (ML-KEM / Kyber combined with a classical algorithm), so the session key is secure even against a future quantum adversary. See [ALB security policies](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/describe-ssl-policies.html).
### Stateless Design
ECS containers hold no user data between requests. All persistent data (multimodal inputs, async job outputs) is stored exclusively in your S3 buckets. When a container is stopped or replaced, no user data is lost.
---
## :material-brain: Amazon Bedrock
### Data Privacy
AWS gives you explicit control over whether your prompts and outputs are retained from inference requests via a **data retention mode**. The mode can be set at the account or project level and applies consistently across all inference calls. For full details, see the [Amazon Bedrock data retention documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/data-retention.html).
#### Data Retention Modes
| Mode | Behavior |
|-----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `default` | AWS may retain data for safety and abuse-prevention purposes. The model provider does **not** receive it. Actual retention depends on the model — consult the model's terms for specifics. |
| `provider_data_share` | AWS retains and shares your inference data with the model provider per their requirements. Required for access to certain models (see below). |
| `none` | **Zero data retention (ZDR).** No request or response data is written to durable storage by AWS or shared with the model provider. |
!!! info "Your retention policy takes precedence over model access"
If your account or project is configured for zero data retention (`data_retention_mode: none`) and you invoke a model that requires retention, Amazon Bedrock **blocks the request and returns an error** rather than retaining the data anyway.
#### Zero Data Retention (ZDR) for High-Compliance Accounts
Some models require data retention for safety and abuse-prevention purposes. If your organization requires zero data retention for compliance reasons and needs access to these models, contact your **AWS account manager** to discuss eligibility. ZDR access is evaluated on a per-account, per-model basis in coordination with the model provider.
You can also enforce a zero-retention policy organization-wide via an AWS Service Control Policy (SCP) — contact your AWS account manager or cloud team to set this up.
#### `provider_data_share` Mode and Model Availability
Certain models — for example, models that require provider-side safety review — are only accessible if your account is configured to share inference data with the model provider. This is an explicit opt-in: most models do not require it, and AWS blocks the request if your retention policy does not permit it.
!!! warning "Understand the implications before enabling `provider_data_share`"
When this mode is active, AWS retains and shares your inference data with the relevant model provider per their requirements. Prefer enabling it at the project level rather than account-wide, and verify which models require it before doing so. See the [Amazon Bedrock data retention documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/data-retention.html) for configuration steps.
#### Model Deployment Account Architecture
The default isolation guarantee is enforced by the **Model Deployment Account** architecture: for each model provider, AWS maintains isolated accounts where model inference runs. AWS confirms:
> *"Model providers don't have any access to those accounts. [...] Because the model providers don't have access to those accounts, they don't have access to Amazon Bedrock logs or to customer prompts and completions."*
This means that regardless of the geographic origin of a model, inference runs on AWS-owned infrastructure and — unless `provider_data_share` mode is explicitly configured — your prompts never reach the model provider. See [which models run in which geography](models.md).
### Abuse Detection
AWS operates automated abuse detection mechanisms on Amazon Bedrock to identify activity that violates AWS or model provider terms of service. Full details are in the [Amazon Bedrock abuse detection documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/abuse-detection.html).
Key points:
- **Zero operator access (ZOA):** No AWS operator can access model inputs or outputs.
- **No storage of inputs or outputs by default:** AWS does not store model inputs or outputs unless a specific model requires it for safety and abuse-prevention purposes (see [Data Privacy](#data-privacy)); full zero data retention is the `none` retention mode.
- **Model-specific retention for abuse detection:** A small number of models require short-term retention of flagged or all traffic for automated offline abuse detection. For example, classifier-flagged traffic for certain OpenAI models may be retained for up to 30 days, and some Anthropic models require opting in to share retained traffic with the provider for abuse review. Eligible customers can request full ZDR for these models through their AWS account team.
- **CSAM detection:** AWS uses automated mechanisms (hash matching, classifiers) to detect child sexual abuse material in image inputs. Detected content is blocked (`400 ValidationException`), may be stored for review, and may be reported to NCMEC or relevant authorities.
- **Policy violations:** If abuse is detected, AWS may contact the email address on your AWS account and may suspend access to affected models. Keep your AWS account contact information current and monitored.
### Encryption at Rest and in Transit
From the [Amazon Bedrock FAQs](https://aws.amazon.com/bedrock/faqs/):
> *"Your data in Amazon Bedrock is always encrypted in transit and at rest, and you can optionally encrypt the data using your own keys."*
See [KMS Encryption](#kms-encryption) below for how stdapi.ai handles encryption at the infrastructure level.
### Cross-Region Inference Profiles and Data Geography
When cross-region inference is enabled, Bedrock may route a request to another region within the inference profile's scope. AWS defines two types of profiles ([source](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference-support.html)):
| Profile type | Example ID prefix | Geography |
|-------------------------------------|-----------------------------|-------------------------------------------------------------------------------------------|
| **Geography-pinned** (US, EU, APAC) | `us.`, `eu.`, `apac.` | Fixed destination list — **never changes**, guaranteed to stay within the named geography |
| **Global** | `global.` | May route to any AWS commercial region worldwide |
A model ID carrying **no prefix** is not an inference profile at all: it is a plain in-region invocation, served entirely by the region the request is sent to.
AWS explicitly states:
> *"if an inference profile is tied to a geography (such as US, EU, or APAC), its destination Region list will never change."*
Set `AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false` to prevent Bedrock from using global profiles. Geography-pinned profiles (`us.*`, `eu.*`, `apac.*`) remain available and provide resilience within their geography. See [Region-Specific Configuration](#region-specific-configuration) for examples.
### Model Providers and Data Access
All third-party models available through Bedrock are subject to the same Model Deployment Account architecture: the provider's software runs in AWS-owned, AWS-operated accounts that the provider cannot access. Your prompts and completions are never shared with any model provider, regardless of where that provider is headquartered.
This applies to providers from every geography, for example:
- **Alibaba Cloud** (China 🇨🇳) — Qwen models
- **Amazon** (United States 🇺🇸) — Nova and Titan models
- **Anthropic** (United States 🇺🇸) — Claude models
- **Cohere** (Canada 🇨🇦) — Command and Embed models
- **DeepSeek** (China 🇨🇳) — DeepSeek models
- **Meta** (United States 🇺🇸) — Llama models
- **MiniMax** (China 🇨🇳) — MiniMax models
- **Mistral AI** (France 🇫🇷) — Mistral models
- **Moonshot AI** (China 🇨🇳) — Kimi models
- **Stability AI** (United Kingdom 🇬🇧) — Stable Diffusion models
- **Writer** (United States 🇺🇸) — Palmyra models
---
## :material-microphone: AWS AI Services
Amazon Polly, Transcribe, Comprehend, and Translate each run in an independently configurable region. By default all four treat every `AWS_BEDROCK_REGIONS` entry as a candidate and fail over between them, so pointing `AWS_BEDROCK_REGIONS` to your target geography is usually sufficient.
### AWS AI Service Improvement Opt-Out
!!! warning "Action required before processing sensitive data with Polly, Transcribe, Comprehend, or Translate"
Unlike Amazon Bedrock, AWS may use content processed by these four AI services to improve service quality **by default**. This means audio recordings, transcription text, translated content, and language detection inputs could be used for model training unless you opt out.
Opt out by configuring an [AI services opt-out policy](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_ai-opt-out.html) at the AWS Organizations level. This is a **one-time action** in the AWS Console. It applies to your entire account immediately and AWS also deletes previously stored content:
> *"When you opt out of content use by an AWS AI service, that service deletes all of the associated historical content that was shared with AWS before you set the option."*
You do not need to opt out for Amazon Bedrock — Bedrock does not use prompts or completions to improve models, and does not share them with model providers unless you enable `provider_data_share`. Retention itself is governed separately by your [data retention mode](#data-retention-modes), and by the model-specific [abuse-detection](#abuse-detection) rules that apply to a small number of models.
---
## :material-shield-check: Content Safety
Content filtering is a control you configure, not one the gateway supplies of its own. It enforces an [Amazon Bedrock guardrail](operations_configuration.md#bedrock-guardrails) that you create and own, evaluated by Bedrock in your account and in your configured regions. With no guardrail configured, nothing is filtered or refused on inference — the [Moderations API](api_openai_moderations.md) still classifies content on demand, but classification is a report, not a block.
There is no separate stdapi.ai content policy, model evaluation, or review stage layered on top. Every behavior below is the enforcement of the policy **you** define on the guardrail, plus the classification backends the Moderations API exposes.
### Guardrail Coverage
A guardrail set through [`AWS_BEDROCK_GUARDRAIL_IDENTIFIER` and `AWS_BEDROCK_GUARDRAIL_VERSION`](operations_configuration.md#aws-bedrock-guardrail-identifier) — or carried by a [model alias](operations_configuration.md#model-aliases-configuration), which overrides the global one for the requests naming that alias — applies to **every route**:
| Routes | Direction checked |
|--------------------------------------------------------------|--------------------------------------------------------------------------------------|
| Chat Completions, Responses, Completions, Anthropic Messages | **Input and output**, natively inside the Bedrock invocation |
| Embeddings, Rerank, Images Generations / Edits, Videos, Audio Speech | **Input** — the client-supplied text, checked before it reaches the backend |
| Audio Transcriptions, Audio Translations | **Output** — the transcript or translated text, checked before it is returned |
| Realtime | **Input and output**, per turn — written items before the model sees them, transcribed caller speech, and each completed answer |
Routes whose AWS backend has no native guardrail mechanism call the ApplyGuardrail API explicitly; see [Route Coverage](operations_configuration.md#route-coverage) for the mechanism used per route, and [Bedrock Guardrails](operations_iam_permissions.md#bedrock-guardrails-optional) for the IAM permission it requires.
Clients cannot weaken the policy: [`AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE`](operations_configuration.md#aws-bedrock-allow-guardrail-override) is `false` by default, so guardrail request headers and request-body guardrail configuration are ignored while a guardrail is configured. It is auto-enabled at startup only when no guardrail is configured anywhere — there is then no policy to bypass.
!!! warning "Two places a configured guardrail does not reach"
- **Bedrock Mantle-served models** — guardrails are a `bedrock-runtime` feature and do not apply to requests served by the Mantle endpoint. Startup emits a warning counting the affected models; set [`AWS_BEDROCK_MANTLE_ENABLED=false`](operations_configuration.md#bedrock-mantle-enabled) to close the gap. A **model alias** that carries a guardrail while pointing at a Mantle-served model is fatal at startup rather than silently unguarded.
- **Batch inference** — the Bedrock batch API cannot carry a guardrail. A batched request that a configured guardrail would apply to is **refused**, not run unguarded; send those requests without batching.
Both are fail-closed by design: the failure mode is a refused request or a refused startup, never content served past a policy the operator believes is active.
### Intervention Behavior
What a client sees when the guardrail intervenes depends on the mechanism the route uses:
| Route family | Result of a blocking intervention |
|--------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
| ApplyGuardrail-enforced routes (embeddings, rerank, images, videos, speech, transcription, translation, moderations) | HTTP **400** with error code `content_filter`, carrying the guardrail's own configured blocked messaging |
| Chat Completions, Responses, Completions | The response is returned and reports `content_filter` as its finish (or incomplete) reason |
| Anthropic Messages | The response is returned with stop reason `refusal` |
| Realtime | A terminal `error` event and close code `3000` — the session ends |
A **masking-only** intervention (the guardrail's sensitive-information policy anonymizing rather than blocking) substitutes the masked text instead of failing: masked input reaches the backend model already masked, and a masked transcript or translation is returned on the plain `json` and `text` formats. Response formats that cannot represent masked text — `srt`, `vtt`, `verbose_json`, `diarized_json` — fail with the same `content_filter` error rather than return the unmasked content.
On the Realtime API the check cannot precede delivery in one direction: the model's speech is streamed while it is generated and its transcript is only complete once the answer is over, so a blocked answer may already have been partly heard when the session ends.
!!! info "Guardrail evaluation is a billed AWS operation"
AWS charges for the guardrail on every route it applies to. Only the ApplyGuardrail-enforced routes report the units consumed, which appear per guardrail policy in [usage logs and cost tracking](operations_logging_monitoring.md). Native chat routes are billed by AWS but return no unit counts, so tracked cost on them is lower than the AWS bill by the guardrail's share.
### Content Classification
The [Moderations API](api_openai_moderations.md) (`POST /v1/moderations`) classifies text and images against three backends, all inside your AWS account:
| Backend | Requires | Inputs |
|--------------------------------------------------------------------------|-------------------------------------------------------|------------------|
| Amazon Bedrock Guardrails | A configured guardrail resource | Text and images |
| Inline guardrail checks (`InvokeGuardrailChecks`) | **No guardrail resource** — only a Bedrock region offering the operation | Text only |
| Amazon Comprehend toxicity detection | Nothing — always available | Text only |
Because the last two need no setup, `/v1/moderations` works on any deployment: with no guardrail configured it resolves to inline guardrail checks where a configured region offers them, then falls back to Comprehend. See [Model Support](api_openai_moderations.md#model-support) for the selection rules and category mapping, and [Comprehend Moderation](operations_iam_permissions.md#comprehend-moderation) for its permission.
The same classification is available inline on generation: the `moderation` request parameter of the Chat Completions and Responses APIs reports how the guardrail assessed the input and the output of that request. It requires a guardrail resource, and is rejected on Mantle-served models.
### Personal Data in Content
Personal data is handled in exactly two places, both of which you switch on deliberately:
- **The guardrail's sensitive-information policy** — the PII entity types and regular expressions you configure on the guardrail are masked or blocked wherever that guardrail is checked, in both directions, following the intervention behavior above.
- **Amazon Transcribe PII redaction** — a client may request `ContentRedaction` on a transcription; only the single-output `redacted` mode is accepted, so no unredacted copy is produced. See [Audio Transcriptions](api_openai_audio_transcriptions.md).
Just as importantly, there is no personal-data handling anywhere else, and none should be assumed:
- The gateway performs **no detection, classification or redaction of its own** — everything above is AWS-side policy evaluation.
- Only **text** is submitted for checking on inference routes. Images, audio and documents passed to a model are not scanned for personal data; the Moderations API is the one surface that submits images to a guardrail.
- Objects held temporarily in your S3 buckets are **not scanned or redacted** — see [S3 Data Storage](#s3-data-storage) for their lifecycle.
- Application logs are not filtered for personal data; prompt and response content simply never reaches them unless you enable it — see [Logging](#logging).
---
## :material-bucket-outline: S3 Data Storage
S3 is used as temporary storage for multimodal content (images, PDFs, audio files) passed to or returned from Bedrock and the AI services. Data is stored only in buckets you own and configure.
- **Primary bucket** (`AWS_S3_BUCKET`) — must reside in the same AWS region as the first entry in `AWS_BEDROCK_REGIONS`.
- **Regional buckets** (`AWS_S3_REGIONAL_BUCKETS`) — for multi-region deployments, one bucket per Bedrock region ensures each region reads and writes data locally. When using the Terraform module, these are created automatically.
- **Lifecycle policies** — the Terraform module applies a 1-day lifecycle policy to the temporary prefix as a failsafe. The application itself removes temporary files as soon as the operation completes, so files rarely remain beyond the duration of a single request.
S3 stores data within the AWS region where each bucket is created. Data does not leave that region unless you explicitly configure replication.
---
## :material-text-box-check-outline: Logging
### Application Logging
By default, stdapi.ai logs only request metadata — HTTP method, path, status code, execution time, and model identifier. **Prompt and response content are never written to logs** unless explicitly enabled.
Setting `LOG_REQUEST_PARAMS=true` enables full request/response payload logging. This is **disabled by default** and should remain disabled in production environments handling sensitive data. See [Logging & Monitoring](operations_logging_monitoring.md) for details.
### Amazon Bedrock Invocation Logging
Bedrock optionally supports invocation logging — recording model inputs and outputs to S3 or CloudWatch Logs. This feature is **disabled by default** on AWS. When enabled:
- **S3 destination**: objects are encrypted using SSE-KMS (CMK supported via key policy).
- **CloudWatch destination**: log group can be encrypted with a KMS CMK.
- Logging scope is configurable: metadata only, or including full prompt and completion content.
See [Amazon Bedrock invocation logging](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html) for configuration details.
---
## :material-key-outline: KMS Encryption
All data stored by stdapi.ai is encrypted at rest. The Terraform module automatically creates a **Customer Managed Key (CMK)** with a restrictive key policy and automatic annual rotation enabled.
For higher compliance needs, AWS KMS supports additional controls: custom key policies, crypto-shredding, multi-region keys, and [CloudHSM-backed custom key stores](https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) for FIPS 140-3 Level 3 hardware-validated key storage.
!!! tip "Bring your own CMK"
To use an existing CMK, create the S3 bucket and CloudWatch log groups outside the Terraform module and pass them via `aws_s3_bucket` and related parameters. See [Advanced Deployment](operations_deploy_advanced.md#integration-with-existing-infrastructure).
Two stores are encrypted under a key you name rather than the bucket's own, so their key policy can be scoped to this workload instead of to everything the bucket holds:
| Setting | Encrypts | Default when unset |
|-------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------|---------------------------------------------|
| [`AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-transcribe-output-encryption-key-arn) | Each transcription job's output, with the job's request identifiers as the KMS encryption context | The bucket's own default encryption |
| [`AWS_BEDROCK_SESSION_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-bedrock-session-encryption-key-arn) | The Amazon Bedrock session storage behind stored responses, chat completions and conversations | The AWS-managed key |
---
## :material-shield-star: AWS Security Hub, GuardDuty & DNS Firewall Integration
Security Hub Foundational Security Best Practices control mapping, GuardDuty Runtime Monitoring, and Route 53 Resolver DNS Firewall are infrastructure-level security controls, covered on the [Authentication & Security](operations_authentication_security.md) page — see [AWS Security Hub, GuardDuty & DNS Firewall Integration](operations_authentication_security.md#aws-security-hub-guardduty-dns-firewall-integration) for the full module compliance mapping and configuration details.
---
## :material-cog-outline: Compliance Configuration Reference
| Variable | Purpose | Compliance relevance |
|---------------------------------------------|---------------------------------------------------------------|-------------------------------------------------------|
| `AWS_BEDROCK_REGIONS` | Ordered list of Bedrock regions | Restrict model inference to a specific geography |
| `AWS_BEDROCK_CROSS_REGION_INFERENCE` | Enable Bedrock cross-region routing within configured regions | Set `false` to restrict inference to a single region |
| `AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL` | Allow Bedrock to route globally outside configured regions | Set `false` to enforce geographic boundaries |
| `AWS_S3_BUCKET` | Primary S3 bucket | Must be in your target region |
| `AWS_S3_REGIONAL_BUCKETS` | Per-region S3 buckets for multi-region setups | Prevent cross-region data transfer |
| `AWS_POLLY_REGION` | Polly service region | Pin to your target geography |
| `AWS_TRANSCRIBE_REGION` | Transcribe service region | Pin to your target geography |
| `AWS_TRANSCRIBE_S3_BUCKET` | S3 bucket for Transcribe audio files | Must be in the same region as `AWS_TRANSCRIBE_REGION` |
| `AWS_COMPREHEND_REGION` | Comprehend service region | Pin to your target geography |
| `AWS_TRANSLATE_REGION` | Translate service region | Pin to your target geography |
| `AWS_BEDROCK_GUARDRAIL_IDENTIFIER` | Guardrail applied to every route | Enforce your content and sensitive-information policy |
| `AWS_BEDROCK_GUARDRAIL_VERSION` | Version of that guardrail | Pin the exact policy version in force |
| `AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE` | Allow clients to override the configured guardrail | Keep `false` so no client can bypass the policy |
!!! warning "Service regions are unpinned by default"
Left unset, `AWS_POLLY_REGION`, `AWS_TRANSCRIBE_REGION`, `AWS_COMPREHEND_REGION` and `AWS_TRANSLATE_REGION` make every [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) entry a candidate, with automatic failover between them. Restrict `AWS_BEDROCK_REGIONS` to compliant regions, or pin each service explicitly, so no request can be served outside your target geography.
---
## :material-map-marker-multiple: Region-Specific Configuration
=== ":fontawesome-solid-earth-europe: EU / GDPR"
To ensure all data processing stays within the European Union:
```bash
# Restrict Bedrock to EU regions
export AWS_BEDROCK_REGIONS=eu-west-1,eu-west-3,eu-central-1,eu-north-1
# Prevent Bedrock from using global profiles that could route outside the EU.
# Geography-pinned eu.* profiles are still used for failover within the EU.
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false
# S3 bucket in an EU region (must match the first entry in AWS_BEDROCK_REGIONS)
export AWS_S3_BUCKET=my-stdapi-eu-bucket
```
!!! tip "Ready-to-use Terraform example"
:fontawesome-solid-earth-europe: **Multi-region GDPR (EU)** — [getting_started_production_gdpr](https://github.com/stdapi-ai/samples/tree/main/getting_started_production_gdpr)
=== ":fontawesome-solid-earth-americas: United States"
To restrict all data processing to US AWS regions:
```bash
# Restrict Bedrock to US regions
export AWS_BEDROCK_REGIONS=us-east-1,us-west-2,us-east-2
# Prevent Bedrock from using global profiles that could route outside the US.
# Geography-pinned us.* profiles are still used for failover within the US.
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false
# S3 bucket in a US region
export AWS_S3_BUCKET=my-stdapi-us-bucket
```
!!! tip "Ready-to-use Terraform example"
:fontawesome-solid-earth-americas: **Multi-region US** — [getting_started_production_us](https://github.com/stdapi-ai/samples/tree/main/getting_started_production_us)
=== ":fontawesome-solid-earth-asia: Asia Pacific"
To restrict all data processing to Asia Pacific AWS regions:
```bash
# Restrict Bedrock to APAC regions
export AWS_BEDROCK_REGIONS=ap-northeast-1,ap-southeast-1,ap-southeast-2
# Prevent Bedrock from using global profiles that could route outside APAC.
# Geography-pinned apac.* profiles are still used for failover within APAC.
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false
# S3 bucket in an APAC region
export AWS_S3_BUCKET=my-stdapi-apac-bucket
```
---
## :material-gavel: US Law and Cloud Provider Obligations
AWS is incorporated and headquartered in the United States. Regardless of where your data is stored, AWS as a legal entity is subject to US law — including statutes with explicit extraterritorial reach. This section describes the statutes and the AWS commitments and technical controls that exist around them; it is not legal advice, and how these statutes apply to your data is a question for your own counsel.
### CLOUD Act
The **Clarifying Lawful Overseas Use of Data (CLOUD) Act, 2018** requires US-based cloud providers to produce customer data in response to a valid US law enforcement warrant or court order, **even when the data is stored outside the United States**. Content data requires a court-issued warrant establishing probable cause — the highest legal standard under US criminal procedure.
Key limits and AWS commitments ([AWS CLOUD Act page](https://aws.amazon.com/compliance/cloud-act/)):
- **Challenge mechanism**: AWS can file a motion to quash or modify requests that would require violating another country's laws, and preserves this right contractually.
- **Customer notification**: AWS's policy is to notify customers of government data requests unless prohibited by a court order (gag order).
- **Zero cross-border content disclosures**: AWS publicly reports that it has not disclosed enterprise or government content stored outside the US to the US government since it began reporting this metric in 2020.
- **Administrative access controls**: The [AWS Nitro System](https://aws.amazon.com/ec2/nitro/) restricts administrative access to customer workloads — including by AWS employees — a design validated by independent security audit (NCC Group).
### FISA Section 702
Section 702 of the **Foreign Intelligence Surveillance Act** authorizes US intelligence agencies to compel US-based electronic communication service providers to deliver communications of non-US persons located outside the United States — without a per-target warrant. This authority was reauthorized in April 2024 and has no provider challenge mechanism equivalent to the CLOUD Act.
The encryption controls described on this page apply here as they do everywhere else: data stored by stdapi.ai is encrypted with a **customer-managed KMS key (CMK)** whose policy and grants you control, and whose usage is visible to you in CloudTrail (see [KMS Encryption](#kms-encryption)). Contractual protections, likewise, cannot override applicable US law. Whether a given combination of controls satisfies your obligations is a legal assessment to make with your own counsel.
### Transfer Frameworks
**EU — Data Privacy Framework and SCCs.** The European Commission adopted the EU-US Data Privacy Framework (DPF) in July 2023, requiring proportionality constraints on US intelligence collection and establishing a Data Protection Review Court (DPRC) as a redress mechanism for EU individuals. AWS is certified under the DPF ([AWS DPF page](https://aws.amazon.com/compliance/eu-us-data-privacy-framework/)). A legal challenge by noyb is pending before the CJEU but has not resulted in invalidation as of this writing.
AWS also incorporates **Standard Contractual Clauses (SCCs)** into its Data Processing Addendum by default — providing a contractual transfer mechanism independent of the DPF. SCCs cannot override applicable US law, but they contractually bind AWS to notify customers of legal demands and challenge requests where legally permissible. Maintaining SCCs alongside the DPF is the recommended posture.
**APAC and other regions.** There is no unified bilateral transfer framework equivalent to the EU-US DPF for APAC jurisdictions. Enterprises typically rely on AWS contractual commitments, customer-managed encryption, and jurisdiction-specific legal analysis. AWS maintains local certifications (IRAP for Australia, ISMAP for Japan, K-ISMS for South Korea, MTCS for Singapore) relevant to respective regulatory environments.
!!! info "Recommended posture"
Deploy in AWS regions within your target geography, encrypt stored data with customer-managed KMS keys whose policy you control, and maintain SCCs as a transfer mechanism independent of DPF validity. This combines a residency control, an encryption control and a contractual transfer mechanism; assess with your own counsel whether it meets your obligations.
---
## :material-lightbulb-outline: Best Practices for High-Compliance Deployments
- :material-check: **Restrict all region settings to compliant regions** — `AWS_BEDROCK_REGIONS` is the primary control; all services default to it, and any optional per-service override must stay within your target geography.
- :material-check: **Set `AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false`** — stdapi.ai will then use geography-pinned inference profiles (`us.*`, `eu.*`, `apac.*`), whose destination region list AWS commits to keeping within the named geography.
- :material-check: **Opt out of AWS AI service improvement** via an AWS Organizations policy — one-time console action covering Polly, Transcribe, Comprehend, and Translate.
- :material-check: **Configure an Amazon Bedrock guardrail and leave `AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE` at `false`** — the policy then applies to every route in both directions and no client can weaken it. If you rely on it, also set `AWS_BEDROCK_MANTLE_ENABLED=false`, since guardrails do not apply to Mantle-served models. See [Content Safety](#content-safety).
- :material-check: **Use a CMK with a restrictive key policy** — the Terraform module creates one by default, so the key policy, its grants and its CloudTrail usage records are yours to control. For stricter control: bring your own key, limit decrypt to the ECS task role, enable automatic rotation, crypto-shredding for right-to-erasure, or a CloudHSM-backed store for FIPS 140-3 Level 3.
- :material-check: **Confirm `LOG_REQUEST_PARAMS` is disabled** (the default) in production — prompt and response content are then kept out of application logs entirely. If Bedrock invocation logging is enabled for audit purposes, configure a KMS CMK for the S3 or CloudWatch destination.
- :material-check: **Use AWS PrivateLink** for Bedrock and S3 to keep service calls off the public internet, and **enable TLS 1.3 on the ALB** to activate post-quantum hybrid key exchange.
- :material-check: **Enable AWS CloudTrail** in all configured regions to monitor API activity across Bedrock, S3, KMS, and AI services.
- :material-check: **Enable AWS Security Hub and GuardDuty** on the account — deploy via the Terraform module and set `compliance_vpc_endpoints_enabled=true`, `guardduty_vpc_endpoint_enabled=true`, and `dns_firewall_enabled=true` (dedicated VPC only) to close the remaining FSBP gaps, support GuardDuty Runtime Monitoring, and block DNS resolution of known-malicious domains. See [AWS Security Hub, GuardDuty & DNS Firewall Integration](operations_authentication_security.md#aws-security-hub-guardduty-dns-firewall-integration).
- :material-check: **EU users: maintain SCCs alongside the DPF** — SCCs (included by default in the AWS Data Processing Addendum) provide a transfer mechanism independent of DPF validity and contractually bind AWS to challenge requests and notify you.
---
## :material-domain: Regulated Industries
### :material-hospital-box: Healthcare (HIPAA)
**Amazon Bedrock is a HIPAA-eligible service** and is covered under the [AWS Business Associate Agreement (BAA)](https://aws.amazon.com/compliance/hipaa-eligible-services-reference/). For the current list of all HIPAA-eligible AWS services, see the [AWS HIPAA Eligible Services Reference](https://aws.amazon.com/compliance/hipaa-eligible-services-reference/).
**There is no stdapi.ai BAA to sign.** stdapi.ai runs entirely within your AWS account — you are the data controller. Your existing AWS BAA with Amazon covers the underlying services. No separate agreement with stdapi.ai is required.
Recommended configuration for PHI workloads:
- Configure the [AI services opt-out policy](#aws-ai-service-improvement-opt-out) before processing any PHI through Polly, Transcribe, Comprehend, or Translate
- Restrict `AWS_BEDROCK_REGIONS` to regions covered by your BAA geography requirements
- Enable Bedrock invocation logging to a KMS-encrypted S3 bucket if audit trails are required by your compliance programme
- Confirm `LOG_REQUEST_PARAMS` is disabled (the default) so PHI never appears in application logs
### :material-bank: Financial Services (SOC 2, PCI-DSS, GDPR)
Amazon Bedrock carries **SOC 1/2/3**, **PCI-DSS**, and **GDPR** certifications. See [AWS Compliance Programs](https://aws.amazon.com/compliance/programs/) for the current list.
stdapi.ai itself does not process or store payment card data. Your PCI-DSS scoping decision is determined by what data your application sends to the gateway — not by the gateway itself.
For the legal context around cross-border data access under US law (CLOUD Act, FISA 702), the AWS commitments that apply, and the region and encryption controls available to you, see [US Law and Cloud Provider Obligations](#us-law-and-cloud-provider-obligations).
### :material-office-building: Government / Public Sector (FedRAMP)
Amazon Bedrock has received **FedRAMP Moderate and High authorization**. See the [AWS FedRAMP page](https://aws.amazon.com/compliance/fedramp/) and the [FedRAMP Marketplace](https://marketplace.fedramp.gov/) for current authorization status.
Restrict `AWS_BEDROCK_REGIONS` to US regions and set `AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false` so that inference is served only from US regions and US geography-pinned inference profiles.
### :material-briefcase: Legal & Professional Services
Attorneys, consultants, accountants, and other professionals bound by confidentiality obligations cannot transmit client materials to third-party AI services. stdapi.ai runs inference on the AWS services and regions you enable — no third party sits between your users and your models, and the gateway contacts no vendor endpoint of its own choosing. The outbound paths that can exist are driven by your configuration or your own clients; see [Application Data Flow](#application-data-flow). This makes it a workable basis for AI-assisted document review, contract analysis, and research where client confidentiality is non-negotiable.
---
## :material-arrow-right: Next Steps
- :material-server-network: [**Advanced Deployment**](operations_deploy_advanced.md) — Multi-region Terraform examples
- :material-directions-fork: [**Resilience & Failover**](operations_resilience.md) — Multi-region routing and infrastructure resilience
- :material-cog: [**Configuration Reference**](operations_configuration.md) — Complete list of environment variables
- :material-email-outline: [**Contact**](contact.md) — Questions about the product's controls, configuration or licensing, and private offers. Assessing those controls against your obligations is a judgement for you and your own advisers
---
## Troubleshooting
Source: https://stdapi.ai/md/operations_troubleshooting.md
# :material-wrench: Troubleshooting
Common issues when deploying stdapi.ai for the first time. If your error isn't listed here, see the [Contact](contact.md) page, or open an issue on [GitHub](https://github.com/stdapi-ai/stdapi.ai/issues).
---
## :material-cloud-upload: Terraform / Deployment
??? failure "`terraform apply` fails with AccessDenied on IAM, KMS, or ECS actions"
Your AWS profile does not have sufficient permissions. The stdapi.ai Terraform module provisions IAM roles, KMS keys, ECS, ALB, Route53 records, an optional WAF, and (for some samples) RDS and ElastiCache.
- Use an **administrator-level** AWS profile for the evaluation deployment.
- **Recommended**: deploy into a sandbox/non-production AWS account first, then replicate into your target account with scoped-down principals once validated.
- Verify your active identity: `aws sts get-caller-identity`.
??? failure "`terraform apply` succeeds but nothing is reachable"
Terraform completed but the ECS service is still coming up. The ALB returns `503 Service Unavailable` until tasks pass health checks.
- Wait 2–3 minutes after `terraform apply` completes.
- Check ECS service status: `aws ecs describe-services --cluster --services `.
- Check task logs in CloudWatch: `/aws/ecs/`.
??? failure "Wrong AWS region or profile used by Terraform"
The AWS provider uses the region/profile from your environment, not a Terraform variable.
- Confirm before applying:
```bash
aws sts get-caller-identity
aws configure get region
```
- Set explicitly with `AWS_PROFILE=... AWS_REGION=... terraform apply` if needed.
??? failure "ElastiCache creation failed — insufficient capacity in AZ (Open WebUI sample)"
The ElastiCache Valkey cache occasionally fails to create when the target availability zone is out of capacity.
```text
Error: waiting for ElastiCache Replication Group ... create: unexpected state 'create-failed',
wanted target 'available'
```
- Remove the failed Valkey cache from the ElastiCache console (disable backups first, then wait for full deletion) and re-run `terraform apply`.
- If the problem persists, change `node_type` in `valkey.tf` (e.g. `cache.t4g.micro` → `cache.t3.micro`) and retry.
??? failure "Podman volume mount fails on Fedora/RHEL with SELinux (local Docker)"
SELinux blocks container access to `~/.aws` without a relabel.
- Add `:z` (or `:Z` for exclusive use) to the volume and `--userns=keep-id`:
```bash
podman run --rm -p 8000:8000 \
--userns=keep-id \
-v ~/.aws:/home/nonroot/.aws:ro,z \
-e AWS_BEDROCK_REGIONS=us-east-1,us-west-2 \
-e ENABLE_DOCS=true \
ghcr.io/stdapi-ai/stdapi.ai-community:latest
```
- See [Local Development](operations_getting_started_local.md#run-it) for the full run command.
---
## :material-api: Runtime / First API call
??? failure "`503 Service Unavailable` — on the /docs page or any endpoint"
The ECS service is still starting up. Health checks take a few minutes.
- Wait 2–3 minutes after deployment and refresh.
- Check the ALB target group health in the AWS console.
- If it persists longer than 5 minutes, inspect CloudWatch logs for the ECS task.
??? failure "`ECONNREFUSED` — connection refused, but only from some clients"
The images bind IPv4 only (`GRANIAN_HOST=0.0.0.0`), so a client that resolves the server to an IPv6 address reaches a port nothing is listening on. Clients disagree about which address to try first, which is why the same deployment looks reachable from one language and dead from another: Node.js prefers the `AAAA` record and fails outright, while most Python clients fall back to the `A` record and hide the problem.
- Typically hit with **ECS service discovery**, which publishes an `AAAA` record for every task in an IPv6-enabled subnet. Deployments fronted by an ALB are unaffected — the load balancer terminates the client connection itself and reaches the task over IPv4.
- Set `GRANIAN_HOST=::` for a dual-stack socket answering both families; see the Container Runtime note in [Configuration](operations_configuration.md). The [Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) sets it when the VPC has IPv6 enabled.
- After switching, extend [`PROXY_TRUSTED_HOSTS`](operations_configuration.md#proxy-trusted-hosts) with the IPv4-mapped form of each range (`::ffff:10.0.0.0/112`) — a dual-stack listener reports IPv4 peers in that form, and an untrusted proxy means `X-Forwarded-For` is ignored and the load balancer's own address is logged as the client IP.
??? failure "Targets never become healthy after setting `TRUSTED_HOSTS`"
Host header validation applies to `/health` as well. A load balancer health check addresses the target directly, so its `Host` header carries the target's IP address — which a list of domain names does not match, and every probe is answered with `400`. The target group stays unhealthy and the ALB keeps returning `503`.
- Prefer host validation at the load balancer: an ALB listener rule on the `Host` header, with `TRUSTED_HOSTS` left unset.
- If the application-level allow-list is required, include the address the health check actually sends.
- The container's own `HEALTHCHECK` is unaffected: it derives its `Host` header from `TRUSTED_HOSTS`. See [`TRUSTED_HOSTS`](operations_configuration.md#trusted-hosts).
??? failure "The ECS task never reports healthy, or restarts in a loop"
ECS ignores the image's own `HEALTHCHECK`, so a task definition that declares no `healthCheck` gets no container-level probe at all, and one that declares a probe carried over from an earlier version runs a command the current image no longer provides. Either way the container is reported unhealthy and the service replaces it.
- Copy the `healthCheck` block from the [ECS task definition example](operations_deploy_advanced.md#ecs-task-definition-example), which declares the image's own probe, and re-copy it when upgrading.
- The [Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) declares it for you.
- Do not substitute a `curl` or `urllib` one-liner: it sends an untrusted `Host` and is answered with `400` as soon as [`TRUSTED_HOSTS`](operations_configuration.md#trusted-hosts) is set.
??? failure "Browser TLS warning on the /docs page"
The ALB uses the default `*.elb.amazonaws.com` domain, which has no trusted certificate. This is expected and safe to bypass for testing.
- For a production-grade certificate, configure a custom domain — the [Terraform module](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) supports ACM-managed certificates via `alb_domain_name`.
??? failure "`401 Unauthorized` — client API key missing or wrong"
The API key is missing, wrong, or not configured.
- Pass the key in the `Authorization: Bearer ` header (OpenAI-style) or `X-API-Key` header.
- Retrieve the generated key with `terraform output -raw api_key`.
- If `api_key_create = true` was not set, no API key is configured and requests pass through without authentication by default (useful for testing behind IP-restricted ALB, not for production).
- See [Authentication & Security](operations_authentication_security.md) for all options.
??? failure "`503` saying the feature is not available on the current server — IAM permission denied on an AWS call"
The gateway reached the AWS service, but the **ECS task role** (or your local AWS credentials) lacks permission for the action it used. AWS returns `AccessDeniedException`, which stdapi.ai answers as a feature this deployment cannot run: HTTP `503`, error code `feature_unavailable`, and the same generic message whatever is missing. This is an IAM misconfiguration, **not** a client API-key problem, and the client is deliberately told nothing about it — **the server log names the operation, the model and the permission AWS refused**, under `error_detail`.
- Confirm the task role grants `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` (and the `bedrock:Converse*` actions) for the target model ARNs.
- For models invoked through an inference profile, allow both the profile ARN and the underlying foundation-model ARNs in the policy.
- Some models require one-time activation in the **Bedrock console → Model access** page before they can be invoked.
- Audio, embeddings, and file features need permissions for the relevant services (Polly, Transcribe, Translate, Comprehend, S3) — see [Configuration → IAM Permissions](operations_configuration.md#iam-permissions) for the full IAM reference.
- A `403 permission_error` on a model call means the opposite: [per-user cost attribution](operations_cost_management.md#per-user-attribution) is enabled and the **end user's** role was denied, so the policy to fix is that role's, not the task role's.
- A `400 invalid_request_error` naming an `s3://` input also means the opposite: the denial is on the object *the request pointed at*, in one of the external buckets declared in [`AWS_S3_ACCEPTED_BUCKETS`](operations_configuration.md#aws-s3-accepted-buckets). Only the caller can fix that one — a wrong key, or a bucket policy that does not grant this deployment's role `s3:GetObject` on the object. Objects in the deployment's own buckets keep the `503` above.
??? failure "`401 Unauthorized` — AWS credentials invalid or expired (often local Docker)"
stdapi.ai's **own** AWS credentials are missing, invalid, or expired — AWS returns `UnrecognizedClientException`, `InvalidSignatureException`, or `ExpiredTokenException`, which stdapi.ai maps to HTTP `401` with error type `authentication_error`. This is distinct from the client-facing API-key `401` above (which concerns your `Authorization` / `X-API-Key` header).
- Locally: refresh with `aws sso login` (or update `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_SESSION_TOKEN`) and restart the container.
- On ECS: confirm the task is assuming its IAM role rather than relying on stale static keys.
??? failure "`404 Not Found` — `The model ... does not exist or you do not have access to it`"
The name in the request is not one this deployment serves. Most names need no change when you adopt the gateway: the Anthropic, OpenAI and Cohere models Bedrock serves are published under their providers' own names as well as their Bedrock IDs, derived mechanically from the ID, so `anthropic.claude-opus-5` also answers to `claude-opus-5`, `openai.gpt-oss-120b-1:0` to `gpt-oss-120b` and `cohere.rerank-v3-5:0` to `rerank-v3.5`. This `404` is what a name that *does* differ looks like: one your application hard-codes for a model this deployment does not serve (`gpt-4o`, `gpt-3.5-turbo`, `text-embedding-3-small`, `dall-e-3`) resolves only if you map it yourself with [`MODEL_ALIASES`](operations_configuration.md#model-aliases). Nothing is substituted on your behalf, because a lookalike would serve a different model than the one you asked for.
- List what this deployment actually serves: `GET /search_models` (the default model-discovery endpoint). Filter by capability with query parameters — e.g. `GET /search_models?input_modalities=IMAGE&route=/v1/chat/completions` returns only vision-capable chat models. See the [Search Models API](api_search_models.md) reference.
- `GET /v1/models` is also available for strict OpenAI SDK compatibility (lighter payload, no capability metadata).
- Confirm the pipeline itself with a low-friction model: `amazon.nova-micro-v1:0` (available in all standard Bedrock regions).
- **Keep the name your application already sends** by mapping it onto a served model with [`MODEL_ALIASES`](operations_configuration.md#model-aliases) — an alias only ever points at a model this deployment serves, so what it resolves to is your choice, not a guess.
- Try the name with and without its `anthropic.` / `openai.` / `cohere.` prefix: both forms resolve for those three families, so `anthropic.claude-fable-5` and `claude-fable-5` reach the same model, as do `openai.gpt-oss-120b-1:0` and `gpt-oss-120b`. Cohere additionally spells its versions with a dot, so `cohere.embed-english-v3` answers to `embed-english-v3.0` and `cohere.rerank-v3-5:0` to `rerank-v3.5` — not to `embed-english-v3` or `rerank-v3-5`. Models from other providers are served under their Bedrock IDs only, so name the ID or alias it. A model version Bedrock has retired needs a current one, whichever form you use.
- Only if the model *is* one Bedrock serves and it is still missing: verify `AWS_BEDROCK_REGIONS` includes a region that offers it — see the [Bedrock model availability table](https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html). Adding regions never makes a name Bedrock does not serve resolve.
??? failure "Every Bedrock Mantle model is missing, while the classic Bedrock models are all there"
The catalog lists the classic Bedrock models and none of the ones served through Amazon Bedrock Mantle, in every configured region at once. A per-region or per-model gap looks different: this is the shape of a network policy that reaches one endpoint and not the other.
Bedrock Mantle is served from `bedrock-mantle..api.aws` — a different domain from classic Bedrock's `bedrock-runtime..amazonaws.com`, so an allowlist, firewall rule, proxy exception or VPC endpoint written for `amazonaws.com` does not cover it.
- Allow `bedrock-mantle..api.aws` outbound, for every region in `AWS_BEDROCK_REGIONS`.
- On a private deployment, create the interface VPC endpoint `com.amazonaws..bedrock-mantle` with private DNS enabled.
- Behind a proxy, make sure `HTTPS_PROXY` is set in the task environment and that `NO_PROXY` does not exclude `api.aws`.
- Confirm the task role carries the Bedrock Mantle permissions — without them the models are simply not listed rather than refused. See [IAM Permissions](operations_iam_permissions.md).
- Check what was discovered: `GET /search_models` returns every model the server found, with its regions.
See [Outbound Network Requirements](operations_deploy_advanced.md#outbound-network-requirements) for the full destination list.
The startup warning names the region, the endpoint address and the exception chain behind the failure, so a blocked route (`ConnectionTimeoutError`), a refused connection (`ConnectionRefusedError`), an intercepting proxy's certificate (`SSLCertVerificationError`) and an unresolvable address (`ClientConnectorDNSError`) are told apart without further instrumentation.
??? failure "Bedrock Mantle models are missing in one region only — `bedrock_mantle_regions_without_endpoint`"
The startup log lists the region under `bedrock_mantle_regions_without_endpoint` instead of `unreachable_bedrock_regions`. Bedrock Mantle is offered in fewer regions than classic Bedrock, and where it is not offered `bedrock-mantle..api.aws` has no DNS record at all — nothing to retry, and no network policy to change. See [model availability by endpoint](https://docs.aws.amazon.com/bedrock/latest/userguide/models-endpoint-availability.html).
- Remove the region from [`AWS_BEDROCK_MANTLE_REGIONS`](operations_configuration.md#bedrock-mantle-regions), or unset it to fall back to the regions of `AWS_BEDROCK_REGIONS` that offer Mantle. Classic Bedrock in that region is unaffected either way.
- If no configured region offers Mantle, the log says so as well and no Mantle model is served — set [`AWS_BEDROCK_MANTLE_ENABLED`](operations_configuration.md#bedrock-mantle-enabled) to `false` to stop the warning.
- If AWS has since added the region, list it explicitly in `AWS_BEDROCK_MANTLE_REGIONS`: an explicit list is used as given.
- If the address should resolve because a VPC endpoint provides it, this is instead the private-DNS case covered by *Every Bedrock Mantle model is missing* above.
??? failure "Startup takes tens of seconds — `server_start_time_ms` far above the usual few seconds"
The `start` log event reports `server_start_time_ms` in the tens of thousands where a healthy deployment reports a few thousand. Startup reads the model catalogs of every configured region, and the ECS task metadata endpoint before them; a destination that never answers is only given up on after a timeout.
- Read the `server_warnings` of the same event first. `ECS container metadata endpoint answered after N attempts in X s` accounts for that many seconds on its own: the endpoint is served by the ECS agent over the task ENI and answers slowly when the task is CPU-starved at boot. Raise the task CPU, or the `cpu` of the Fargate task definition, so the agent is scheduled promptly.
- `unreachable_bedrock_regions` and `bedrock_mantle_regions_without_endpoint` each name a region that spent its full timeout budget. Removing the region from `AWS_BEDROCK_REGIONS` or `AWS_BEDROCK_MANTLE_REGIONS` removes the delay.
- [`AWS_CONNECT_TIMEOUT`](operations_configuration.md#aws-connect-timeout) bounds each connection attempt and the model-catalog fetch that follows it, so lowering it lowers what an unreachable region can cost. It also bounds failover between healthy regions, so keep it above your real inter-region latency.
- Model discovery is per region and runs in parallel, so the count of regions costs far less than one unreachable region does.
??? failure "A deploy leaves work unfinished — `abandoned_background_tasks` in the `stop` log event"
Some work is started outside the request that asked for it, so the caller is answered without waiting: temporary file cleanups, vector store file indexing, and the release of live audio sessions. When a task is stopped, the server waits [`SHUTDOWN_DRAIN_TIMEOUT`](operations_configuration.md#shutdown-drain-timeout) seconds for that work, then cancels the rest — and reports the counts as `abandoned_background_tasks` in a `stop` log event raised to `warning`.
- A vector store file reported `failed` after a deployment, with `last_error` saying the indexing was interrupted, or a temporary object that outlived its request, is what those counts look like from the outside. Attach the file again, or let the object's lifecycle rule expire it. A file is never left `in_progress`: one whose indexing is gone is settled the next time it, its store, or the store's file list is read.
- Raise [`SHUTDOWN_DRAIN_TIMEOUT`](operations_configuration.md#shutdown-drain-timeout) **and** the container stop timeout together — the container runtime sends `SIGKILL` a fixed delay after the stop signal (30 seconds by default on Amazon ECS), so raising the wait alone changes nothing past that delay.
- Counts that persist after raising both mean the work itself is long, not that the wait is short: indexing a large file outlasts any stop timeout. Attach large files when a deployment is not in flight, and treat the operation as retryable.
- The wait is best effort in every deployment: a Spot interruption or a hard kill ends the process regardless, so never rely on it for anything whose completion matters.
??? failure "`429 Too Many Requests` — Bedrock throttling / quota"
AWS returned `ThrottlingException`, `TooManyRequestsException`, or `ServiceQuotaExceededException` — mapped to HTTP `429` with error type `rate_limit_error`. You've hit the per-region Bedrock quota.
- Add more regions to `AWS_BEDROCK_REGIONS`. Each region has its own independent quota — three regions ≈ triple the throughput.
- See [Resilience & Failover](operations_resilience.md) for multi-region routing configuration.
- Check quotas in the AWS Service Quotas console for **Amazon Bedrock**.
- When the router put a region on a quota backoff while serving the request, the response carries a `retry-after` header (in seconds) telling the client exactly how long to wait — OpenAI, Anthropic and Cohere SDKs honour it automatically instead of guessing an exponential backoff, up to the 60 s ceiling their retry loops apply to server-supplied delays.
??? failure "`400 Bad Request` — This model is not available under data retention mode 'default'."
A specific model is unavailable or requests to it are rejected because your account's data retention mode is incompatible with what that model requires.
Amazon Bedrock enforces retention compatibility at invocation time: each model declares the retention modes it accepts, and if your effective mode is not among them, the request is blocked.
**Common scenarios:**
- Your account is set to **zero data retention (`none`)** but the model requires `default` or `provider_data_share` for safety or abuse-prevention purposes. Bedrock blocks the request to honour your retention policy. To access the model, either switch to a compatible retention mode or contact your AWS account manager to request ZDR eligibility for that specific model.
- Your account is set to **`default`** but the model exclusively requires `provider_data_share` (typically models with mandatory provider-side safety review). The model will appear as unavailable. Enabling `provider_data_share` grants access but means AWS will share your inference data with the model provider — see [Data Privacy](operations_compliance.md#data-privacy) before enabling it.
??? failure "`400 Bad Request` — invalid parameters from Bedrock"
Bedrock rejected the request parameters (`ValidationException` / `BadRequestException`), mapped to HTTP `400` with error type `invalid_request_error` — for example an unsupported parameter for the chosen model, an out-of-range value, or content that exceeds the model's limits.
- Read the message detail returned in the response (correlate with `x-request-id` in the server logs).
- Confirm the parameter is supported by the model — see the per-API **Feature Compatibility** tables.
??? failure "`400 Bad Request` — a client's own control flag reaches Bedrock as a model parameter"
Request fields stdapi.ai does not declare are forwarded to Amazon Bedrock as provider-specific inference parameters, so any parameter a model accepts can be passed through — including the highly model-specific ones no common API surface exposes. Some OpenAI-SDK-based clients also use that same channel for their *client-side* settings — LiteLLM-derived ones send `drop_params`, `api_key` or `custom_llm_provider` in `extra_body` — and Bedrock answers `ValidationException` for a field no model declares. The symptom is a route that fails for one client and works for every other.
- The known LiteLLM control parameters are stripped by a built-in denylist, so this only appears for a name it does not yet cover. The rejected field is in the Bedrock message detail, correlated via `x-request-id` in the server logs.
- Add that name to [`EXTRA_MODEL_PARAMS_DENYLIST`](operations_configuration.md#extra-model-params-denylist) — it is merged with the built-in list, and every other extra parameter keeps being forwarded.
- If no client needs the passthrough, [`EXTRA_MODEL_PARAMS_DROP_ALL`](operations_configuration.md#extra-model-params-drop-all) disables it outright. Per-model defaults set through [`DEFAULT_MODEL_PARAMS`](operations_configuration.md#default-model-params) are unaffected — only request-supplied extras are dropped.
??? failure "A request's `service_tier`, guardrail or model parameters are ignored"
Some configuration reaches the model that the client did not send, or the value the client sent is not the one applied. Two layers of server-side configuration sit behind every request, and both are deliberate.
- The model name may be an alias carrying its own configuration — check the entry in [`MODEL_ALIASES`](operations_configuration.md#model-aliases-configuration) for that name. Requests naming it get its service tier, guardrail, metadata and model parameters; requests naming the target model directly do not.
- The reverse also happens: alias and server-wide configuration apply to models served through Amazon Bedrock's Converse and InvokeModel operations. A model served through Amazon Bedrock Mantle applies the request's own values only, so a configured service tier, metadata or model parameters are ignored there by design — see [the scope note](operations_configuration.md#model-aliases-configuration).
- A request value is discarded on purpose when its override setting is disabled: [`AWS_BEDROCK_ALLOW_SERVICE_TIER_OVERRIDE`](operations_configuration.md#aws-bedrock-allow-service-tier-override) for the tier, [`AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE`](operations_configuration.md#aws-bedrock-allow-guardrail-override) for the guardrail headers.
- Otherwise the value comes from the server-wide setting for that field — [`DEFAULT_MODEL_SERVICE_TIERS`](operations_configuration.md#default-model-service-tiers), [`DEFAULT_MODEL_PARAMS`](operations_configuration.md#default-model-params) or [`AWS_BEDROCK_GUARDRAIL_IDENTIFIER`](operations_configuration.md#aws-bedrock-guardrail-identifier). The order is always the request, then the alias, then the setting.
??? failure "Startup fails with a validation error naming a model alias"
An alias in [`MODEL_ALIASES`](operations_configuration.md#model-aliases-configuration) maps to an object that is not a valid alias configuration, so the server refuses to start rather than ignore it.
- `Extra inputs are not permitted` names a field that does not exist — check its spelling against the [alias fields](operations_configuration.md#model-aliases-configuration).
- `Field required` on `model` means the object gives configuration but no target model.
- A guardrail needs both `guardrail_id` and `guardrail_version`.
- An alias that only maps a name to a model stays a plain string: `{"my-model": "amazon.nova-lite-v1:0"}`.
??? failure "`400 Bad Request` — text-to-speech rejects a long `input`"
The message states the length the server accepts (`'input' is limited to 3,000 characters…`, or 20,000 with a generative voice). A generative voice speaks up to 20,000 characters unaided; every other voice, and longer generative input, is synthesized into an S3 bucket co-located with the region serving the request, and none is configured there.
- Set [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) for the first region of `AWS_BEDROCK_REGIONS`, and an [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets) entry for every other region that may serve speech — the server log names the one that was missing.
- Grant the task role `polly:StartSpeechSynthesisStream` (generative voices beyond 3,000 characters), `polly:StartSpeechSynthesisTask`, `polly:GetSpeechSynthesisTask`, and S3 read/write/delete on those buckets — see [IAM Permissions](operations_iam_permissions.md#text-to-speech-optional).
- A generative request that still behaves like the others — rejected without a bucket, or slower than expected with one — is missing `polly:StartSpeechSynthesisStream`; the server log names the failure.
- Up to 3,000 characters (6,000 including SSML markup) never needs a bucket; the limits and the expected latency are in [Long Input](api_openai_audio_speech.md#long-input).
??? failure "`400 Bad Request` — a transcription is rejected only for one model"
The message names the condition: `amazon.nova-2-sonic-v1:0` serves `json` and `text` only, and accepts at most 10 minutes of audio per request. It returns no timestamps and does not report a detected language, so subtitles, `verbose_json`, `diarized_json` and `timestamp_granularities` cannot be produced from it.
- Request `json` or `text`, or send the same audio to `amazon.transcribe`, which produces timestamps, SRT/VTT subtitles, speaker diarization and longer recordings — see [Transcriptions](api_openai_audio_transcriptions.md).
- The same model and the same limits apply on [Translations](api_openai_audio_translations.md).
??? failure "`400 Bad Request` — every transcription fails after setting an output encryption key"
The message is Amazon Transcribe's own failure reason for the job, and it names KMS. Only requests that stage audio in a bucket are affected: [`AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-transcribe-output-encryption-key-arn) encrypts the job's output, so streamed transcriptions, which write nothing, keep working — which is what makes the failure look model-specific at first.
- Grant the task role `kms:GenerateDataKey` and `kms:Decrypt` on that key, **and** allow the same role in the key's own policy — a grant on only one of the two denies the job. See [Speech-to-Text](operations_iam_permissions.md#speech-to-text-optional).
- With [`AWS_TRANSCRIBE_REGION`](operations_configuration.md#aws-transcribe-region) unset, a job runs in whichever candidate Region has a co-located bucket, so a single-Region key fails as soon as failover moves the job. Use a [multi-Region key](https://docs.aws.amazon.com/kms/latest/developerguide/multi-region-keys-overview.html), or pin the Region.
- A key policy conditioned on the [encryption context](operations_configuration.md#aws-transcribe-output-encryption-key-arn) must not require `stdapi-ai.user_id`: it is sent only when the request identifies an end user, so requiring it denies every anonymous call.
??? failure "`amazon.nova-2-sonic-v1:0` is missing from `/v1/models` or returns `404`"
The model is not offered in every AWS Region, and the catalog only lists what the configured Regions serve. [Check the model's regions](models.md).
- Add a Region that offers it to [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions); `us-east-1`, `us-west-2`, `ap-northeast-1` and `eu-north-1` did at the time of writing, while `eu-west-3` and `eu-central-1` did not.
- Request model access for it in the Amazon Bedrock console for that Region.
- Grant the task role `bedrock:InvokeModelWithBidirectionalStream`, which this model needs on top of the usual invoke permissions — see [IAM Permissions](operations_iam_permissions.md). Without it the session opens and then ends with a `503` saying live conversation is not available on this server; the server log names the permission.
- `/search_models?route=openai_audio_transcription` lists what this deployment can actually reach.
??? failure "`503 Service Unavailable` — a speech or speech-to-speech request fails a few seconds after it starts"
Real-time audio requests must become ready within [`AWS_CONNECT_TIMEOUT`](operations_configuration.md#aws-connect-timeout) in each candidate region — that budget covers the connection, the initial handshake and the first response together, not the connection alone. On a high-latency or NAT-fronted network the default of 5 seconds can expire in every region, and the request then ends as a `503`.
- Raise [`AWS_CONNECT_TIMEOUT`](operations_configuration.md#aws-connect-timeout) to 10 seconds or more; `AI_RESPONSE_TIMEOUT` governs the response itself and has no effect here.
- The server log names each Region that was abandoned and why.
??? failure "Conversation items are missing, or cannot be added"
A [conversation](api_openai_conversations.md) has a bounded lifetime and a bounded number of writes, and both are reached silently.
- **After 30 days**, a conversation and its items are removed and every route on it returns `404`. Long-lived agents must create a new conversation rather than reusing one indefinitely.
- **1,000 requests that add or delete items** is the per-conversation ceiling; a response bound to a conversation counts as one, whatever its number of output items. Past it, a listing stops early rather than adding failing: the gateway reads at most 1,000 invocation steps, and a single large item spans several. Start a new conversation, seeding it with the items you still need.
- **`503` saying the API is not available on the current server** means the IAM role is missing the [Bedrock Session Storage permissions](operations_iam_permissions.md#bedrock-session-storage-optional), including `bedrock:UpdateSession`, which only the metadata update uses — a deployment created before conversations shipped fails on `POST /v1/conversations/{id}` alone. The client message is the same whichever one is absent; the server log names it.
- **Items added by a streamed response appear when the stream ends**, not while it runs; a client that reads them from a callback fired on the terminal event must wait for the stream to close.
??? failure "S3 error on image generation or audio transcription"
The S3 bucket is missing, unreachable, or in the wrong region.
- The Terraform module creates the bucket automatically unless you pass your own via `aws_s3_bucket`.
- If you're using your own bucket: `AWS_S3_BUCKET` must point to a bucket in the same region as the **first** entry in `AWS_BEDROCK_REGIONS`.
- Verify the ECS task IAM role has `s3:PutObject` / `s3:GetObject` on the bucket.
- **`503` saying transcription is not available on the current server** means no region that can run a transcription has a bucket at all: set [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket), [`AWS_TRANSCRIBE_S3_BUCKET`](operations_configuration.md#aws-transcribe-s3-bucket) or an [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets) entry for it. The server log names the settings.
- **`503` saying the `'url'` response format is not available on the current server** means there is no bucket to host the images a `url` response points at: set [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket), or request `response_format="b64_json"`, which needs no storage. Image requests are refused before anything is generated, so a misconfigured deployment bills nothing for them.
??? failure "Connection timeout to AWS services from ECS"
Outbound traffic to AWS endpoints is blocked.
- Confirm the ECS task's security group allows outbound HTTPS (port 443).
- If using **VPC endpoints** (the Terraform module default), verify the endpoint security groups and policies permit traffic from the ECS task subnet.
- If ECS runs in a private subnet without VPC endpoints, confirm the NAT gateway / route table is configured.
??? failure "`413 Payload Too Large` — request or file rejected as oversized"
Either an attachment exceeds what the chosen model reads, the application-level file-size cap, or an edge control rejected the request.
- `Attachments larger than … are not available on the current server` means the attachment is too large to travel inside the request and there is nowhere to stage it: no region able to serve that model has an S3 bucket. Set [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) for the first region of `AWS_BEDROCK_REGIONS` and an [`AWS_S3_REGIONAL_BUCKETS`](operations_configuration.md#aws-s3-regional-buckets) entry for every other region that may serve the model — the server log names the one that was missing. The same request succeeds unchanged once a bucket exists.
- `An attached file is too large: this model accepts at most … bytes per file` means that model only reads attachments sent inside the request. Send a smaller file, or choose a model that reads attachments from storage — see [Attachment Size](features.md#attachment-size).
- `The attached files are too large: this model accepts at most … bytes of attachments per request` means the same, for the request as a whole: each file fits on its own, but their total does not. Split them across several requests, or choose a model that reads attachments from storage.
- Check [`MAX_INPUT_FILE_SIZE`](operations_configuration.md#max-input-file-size) — it caps the bytes of any single file loaded into memory for model input; disabled by default, so if it's set and the error appears, raise it or reduce the input size.
- If the deployment sits behind the Terraform module's WAF (`alb_waf_enabled=true`), check for a `SizeConstraintStatement` rule on the request body — see [Request Size & Resource Limits](operations_authentication_security.md#request-size-resource-limits).
- If fronted by Amazon API Gateway instead of an ALB, remember its hard 10 MB payload limit.
??? failure "Web search returns nothing, stale results, or is rejected"
The built-in [web search tool](api_openai_responses.md#openai-gpt-web-search) is gated by both an IAM permission and a server setting, and each failure looks different.
- **The model answers from its training data and says it could not search**: the task role is missing `bedrock-websearch:InvokeSearch` / `bedrock-websearch:InvokeFetch`. Add them — see [Web Search IAM](operations_iam_permissions.md#web-search-iam). The request itself still succeeds, so this shows up as a weak answer rather than an error.
- **`400` on `external_web_access`**: the request asked for a value the server does not allow. By default searches stay inside the AWS boundary; set [`AWS_BEDROCK_EXTERNAL_WEB_ACCESS`](operations_configuration.md#bedrock-external-web-access) to change what the server does, or [`AWS_BEDROCK_ALLOW_EXTERNAL_WEB_ACCESS_OVERRIDE`](operations_configuration.md#bedrock-allow-external-web-access-override) to let requests choose. It travels as an extra model parameter (a top-level request field), not as a field of the tool: a client that sets it on the tool changes nothing.
- **`400` on `external_web_access` saying it is not available with this model**: the override is enabled, but only the models that serve web search natively — the OpenAI GPT-5.x family — take a web access choice per request. Everywhere else the parameter must match the configured value, and is refused rather than accepted and ignored. Send it only with those models, or drop it.
- **External web access was enabled but results still look cached**: `bedrock-websearch:ExternalWebAccess` is missing from the task role, so the search falls back to the Amazon Bedrock web index.
- **`400` naming `filters.allowed_domains` or `user_location`**: the model's own search cannot restrict which sources it uses, and running it unrestricted would answer from the very sources the request excluded. Drop the restriction, or send the request to a model that serves web search natively — the OpenAI GPT-5.x family.
- **The tool is rejected outright**: web search is served on `/v1/responses` for the OpenAI GPT-5.x family, in `us-east-1`, `us-east-2` and `us-west-2`. On `/v1/messages` and `/v1/chat/completions` it is not available for these models.
??? failure "`503`/`504`/`408` — request times out mid-stream on long generations"
A slow or hung generation exceeded a timeout somewhere between the model and the client. The status code tells you where: `503` comes from stdapi.ai's own gateway timeout; `504`/`408` come from an edge or proxy timeout in front of it.
- **`503` from the gateway**: Check [`AI_RESPONSE_TIMEOUT`](operations_configuration.md#ai-response-timeout) — it closes stalled upstream model connections; raise it for workloads with long-running generations. The request is **not** retried in another region: the model already ran and AWS bills it either way, so a failover would pay twice for the same generation.
- **`504`/`408` from the edge/proxy**: Check the Terraform module's `alb_idle_timeout` (default: 3600 s) — if you lowered it, or front the deployment with your own load balancer or reverse proxy at a shorter idle timeout, streaming responses can be cut off mid-flight before the gateway's own timeout fires. See [ALB Resilience](operations_resilience.md#alb-resilience).
??? failure "Every request fails after enabling per-user cost attribution"
Model calls run under a session of [`AWS_BEDROCK_USER_ROLE_ARN`](operations_configuration.md#aws-bedrock-user-role-arn), and a session that cannot be opened fails the request rather than silently falling back to the server's identity. The server also reports this at startup, in the `server_warnings` field of its `start` log event. Five causes, in order of likelihood:
- **The trust policy allows only `sts:AssumeRole`.** Tagging the session is a separate action: add `sts:TagSession` to both the trust policy of the end user role and the server's own policy — see [Per-User Cost Attribution IAM](operations_iam_permissions.md#per-user-cost-attribution). Setting [`AWS_BEDROCK_USER_ROLE_TAG_KEY`](operations_configuration.md#aws-bedrock-user-role-tag-key) to null removes the need for it, at the cost of Cost Explorer grouping.
- **The role was just created.** A new or newly-edited trust policy takes a few seconds to propagate; a task started immediately after logs the startup warning and recovers on its own.
- **`403` on every request**: the end user role lacks `bedrock:InvokeModel` or `bedrock:InvokeModelWithResponseStream`, or its `Resource` list misses an ARN form requests actually reach. A cross-region inference profile also needs `arn:aws:bedrock:*::foundation-model/...` for every Region it routes to, and an application inference profile, a prompt router or a prompt ARN each has to be named in its own right — see [Per-User Cost Attribution IAM](operations_iam_permissions.md#per-user-cost-attribution).
- **`403` once a guardrail is configured**: a guardrail applied during an invocation — [`AWS_BEDROCK_GUARDRAIL_IDENTIFIER`](operations_configuration.md#aws-bedrock-guardrail-identifier), a model alias carrying one, or a request-level `moderation` parameter — is evaluated as part of the call the end user signed, so the end user role needs `bedrock:ApplyGuardrail` on the guardrail ARN as well.
- **`400` naming `safety_identifier`**: [`AWS_BEDROCK_USER_ROLE_REQUIRE_IDENTITY`](operations_configuration.md#aws-bedrock-user-role-require-identity) is enabled and the client sends no end user identifier. Either have the client send one, or disable that setting.
??? failure "A batch is refused, stuck, or its requests fail one by one"
The [Batch API](api_openai_batches.md) and the [Message Batches API](api_anthropic_batches.md) refuse at submit what the backend would otherwise fail hours later, so most surprises land in the create call.
- **`503` (or `529` on `/anthropic/...`) on every batch endpoint**: the deployment declares no batch service role. Set [`AWS_BEDROCK_BATCH_ROLE_ARN`](operations_configuration.md#aws-bedrock-batch-role-arn) and grant the policies in [Batch Inference IAM](operations_iam_permissions.md#batch-inference). The server also reports the disabled feature in the `server_warnings` field of its `start` log event.
- **`400` naming a minimum of 100 requests**: batches run on a backend with a floor of 100 requests **per model**. A batch naming several models must reach it for each of them; combine the small ones or send them without batching. The 100 is the default of the per-model Amazon Bedrock quota *Minimum number of records per batch inference job* ([Amazon Bedrock quotas](https://docs.aws.amazon.com/general/latest/gr/bedrock.html)), and it is the value checked here whatever your own account's quota says.
- **The batch stays `validating` (`in_progress`) for several minutes**: expected. Validation alone takes a few minutes before any request runs, and the whole batch has a 24-hour window. Poll rather than resubmit — a resubmission is a second, separately billed batch.
- **`400` naming tool use or a structured output schema**: neither is available in a batch. Remove `tools`/`tool_choice` and `response_format` of type `json_schema`, or send those requests without batching.
- **`400` saying the model is not available for batched requests**: not every model can run batched. Pick another one; when the batch named several models, no sibling job is left running.
- **A model that batches fine is missing from `search_models?batch=true`, or reports `batch: false`**: that flag is a discovery hint published on a best-effort basis and is never used to reject anything — submit the batch and let the answer decide. It is reported for no model at all while [`COST_TRACKING`](operations_configuration.md#cost-tracking) is disabled, and for a few seconds after startup while the catalogue is still being built.
- **`503` on creation, with nothing wrong with the request**: the backend refused the job for a reason that is not the model — the account's batch quota for that model, the service role, or a restriction such as a model the provider marked legacy and the account has not used in the last 30 days. The client message is deliberately generic; the server log carries the reason the backend gave, as a warning.
- **`503` on creation, after the endpoints answered normally**: the task role is missing `bedrock:CreateModelInvocationJob` or the `iam:PassRole` statement on the batch service role; the server log names which. A batch that starts and then fails without results usually means the service role itself cannot read or write the bucket under [`AWS_S3_BATCHES_PREFIX`](operations_configuration.md#aws-s3-batches-prefix) — the reason the backend gives is logged as a warning when a job reports `Failed`.
- **A batch reports no cached tokens, whatever its requests asked for**: prompt caching does not apply to batched requests, on any model. A cache hint — `cache_control` on `/anthropic/v1/messages/batches`, `prompt_cache_key` or `prompt_cache_breakpoint` on `/v1/batches` — is accepted and dropped rather than refused, so the request is answered normally and no cached tokens are reported for it. There is no discount to lose: batched requests are billed at the batch rate already.
- **The first read after the batch ends is slow**: the results are translated and published on that read. Later reads are immediate.
- **The result files are gone, or never go away**: a batch created with `output_expires_after` deletes both files that long after they are written, and one created without it keeps them until they are deleted with the [Files API](api_openai_files.md). The clock starts when the results are published, not when the batch was created.
??? failure "A vector store file stays in progress, fails, or returns nothing"
Indexing runs after the response is sent, so a file is `in_progress` for a moment by design — see [Vector Stores](api_openai_vector_stores.md#indexing-is-asynchronous).
- **`503` on every vector store endpoint**: the deployment declares no vector storage. Set [`AWS_S3_VECTORS_BUCKET`](operations_configuration.md#aws-s3-vectors-bucket) and [`AWS_S3_VECTORS_REGION`](operations_configuration.md#aws-s3-vectors-region), keep [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) set, and grant the [Vector Stores IAM permissions](operations_iam_permissions.md#vector-stores-optional).
- **`503` on one operation only** (creating a store, searching, deleting): a single `s3vectors` action is missing from the task role. The client message is deliberately the same as above; the server log names the action and the bucket.
- **Creating a store fails**: the vector bucket must already exist, in the Region named by [`AWS_S3_VECTORS_REGION`](operations_configuration.md#aws-s3-vectors-region). A vector bucket is a Region-local resource with no failover, so a bucket in another Region is not reachable at all.
- **A file settles as `failed` with `unsupported_file`**: the store does not index that file type. Read `last_error.message` — it names what **this** store indexes, since that differs per store. A store the server owns indexes text only, so a PDF or an office document settles here; convert it first — [RAG Pipelines](use_cases_rag.md#document-parsing) shows a conversion stage — or attach it to a [knowledge base store](api_openai_vector_stores.md#knowledge-base-stores), which indexes those formats as they stand. When the message names formats and the file is already one of them, its bytes are not what the content type claims.
- **A file settles as `failed` with `server_error`**: indexing hit a backend error, or was interrupted before it finished — `last_error.message` says which. Interrupted means the server was replaced, scaled in or killed while it was indexing, and nothing else is wrong. Check the `background` log event sharing the request's `id` for the backend case; attach the file again in both. To stop losing that work at every deployment, give the deployment an indexing queue — [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url) — and another server finishes the job instead.
- **A file stays `in_progress` far longer than the others**: a large file is many passages, each embedded in turn, and indexing is bounded server-wide, so a file attached while others are being indexed waits its turn. A file no server is indexing any more settles as `failed` rather than waiting for good, so an unchanging `in_progress` is work that is still queued.
- **A search returns nothing after attaching**: the store is still indexing (`status` is `in_progress`), the store has passed its expiration (`status` is `expired`), or the `filters` match no file. A filter applies to the file's `attributes`, never to its content.
- **`409` on an update**: several requests are changing the same store at once. Retry the request.
??? failure "Vector store indexing is not picked up by the queue, or a queued file never settles"
Only deployments that set [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url) hand indexing to a queue; without it, indexing runs in the server that accepted the request and everything below is expected behaviour rather than a fault.
- **The server refuses to start, naming the setting**: the URL is not an Amazon SQS queue URL (`https://sqs..amazonaws.com//`), it names a FIFO queue, or [`AWS_S3_VECTORS_BUCKET`](operations_configuration.md#aws-s3-vectors-bucket) is unset. FIFO is refused because its deduplication would silently drop a legitimate re-attach of the same files.
- **A `start` log warning says the queue could not be described**: the queue does not exist, or the task role lacks `sqs:GetQueueAttributes`. The deployment still runs and still queues, but it cannot read your redrive policy, so it falls back to its own retry count. Grant the [Durable Vector Store Indexing permissions](operations_iam_permissions.md#durable-vector-store-indexing).
- **A `start` log warning says the queue has no dead-letter queue**: add a redrive policy. Without one, the message of a file that cannot be indexed is dropped once its retries run out instead of being kept for inspection.
- **Every file still settles as `failed` after a deployment**: the send is failing, which the server log reports at `error` naming `sqs:SendMessage`. A deployment that cannot queue keeps indexing in-process, which is exactly the behaviour the setting was meant to replace, so the symptom looks like the setting doing nothing.
- **Files sit `in_progress` for minutes under load**: a server only takes jobs off the queue while it is not busy answering requests, so indexing yields to clients by design. Scale out, or wait.
- **A file settles as `failed` although the queue is configured**: the job ran out of deliveries. Its message is in your dead-letter queue; the server log says so at `error`. Attach the file again once the underlying cause is fixed.
??? failure "A `vs_kb_...` vector store answers `404`, or refuses a file attached to it"
A [knowledge base store](api_openai_vector_stores.md#knowledge-base-stores) is addressed, never created, so most of these are configuration rather than a bad request.
- **`404` on every route of a `vs_kb_...` identifier**: the knowledge base is not listed in [`AWS_BEDROCK_KNOWLEDGE_BASE_IDS`](operations_configuration.md#aws-bedrock-knowledge-base-ids), or it does not exist in the first [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) entry, or the task role lacks the read permission on it. The three cases answer identically **by design**, so the allowlist cannot be probed by a client; the server log says which one it was. Check the setting first, then grant the [Knowledge Base Vector Stores permissions](operations_iam_permissions.md#knowledge-base-vector-stores) on the knowledge base ARN.
- **`503` when attaching a file, while search and listing work**: the knowledge base has more than one data source, so which one a document belongs to is ambiguous. Name it in the setting as `/`.
- **`400` saying files cannot be attached to the store, while search and listing work**: the allowlisted data source keeps its corpus in sync from somewhere else — a bucket, or another connected service — and takes no file handed to it. Only a **custom** data source does. Point the entry at one, as `/`; the server log names the data source that refused. A knowledge base can hold both kinds, and the store keeps serving search and listing meanwhile.
- **`400` when deleting a file a search returned**: that document belongs to the corpus behind the store rather than to the files attached here, so it is readable and never removable. Remove it where the corpus comes from.
- **`400` on an update, a delete, a chunking strategy, a file batch, a `score_threshold` or a file's `content`**: none of these apply to a store managed outside the server. The [refusal table](api_openai_vector_stores.md#knowledge-base-stores) lists each one and what to do instead.
??? failure "All Bedrock spend still lands on one identity"
[Per-user attribution](operations_cost_management.md#per-user-attribution) reaches the AWS bill through two AWS-side steps that are easy to miss, and neither is instant:
- **The Cost and Usage Report export must include caller identity.** Create a Data Exports CUR 2.0 export with *Include caller identity (IAM principal) allocation data* enabled; an existing export cannot be changed and must be re-created. The identity then appears in `line_item_iam_principal` as `assumed-role//`.
- **The session tag must be activated as a cost allocation tag**, in the AWS Billing console under **Cost allocation tags**, filtered by type **IAM principal**. It is only listed there after that identity has made at least one call, and takes up to 24 hours to appear in Cost Explorer.
- **Requests that identify no end user are billed to the server**, by design. The request log's `aws_role_session_name` field is absent on exactly those requests — use it to find the clients that send no identifier, then enable [`AWS_BEDROCK_USER_ROLE_REQUIRE_IDENTITY`](operations_configuration.md#aws-bedrock-user-role-require-identity).
- **Only model invocations are attributed.** Video generation, guardrail evaluations, speech, transcription and translation stay on the server's own identity.
??? failure "A WebSocket upgrade to `/v1/realtime` answers `404`"
The upgrade request never reached the [Realtime API](api_openai_realtime.md) route — either the URL is wrong, or something in front of the deployment does not forward a WebSocket upgrade at all.
- Check the path: the route lives at `${OPENAI_ROUTES_PREFIX}/v1/realtime` (`/v1/realtime` with the default empty prefix). A client dialling the wrong prefix reaches no route and gets `404` from the framework itself, before authentication is even checked.
- An Application Load Balancer forwards a WebSocket upgrade by default, but anything placed **in front of** it — a CDN, an API Gateway REST API (which has no WebSocket support at all), a reverse proxy that does not pass through `Connection: Upgrade` / `Upgrade: websocket` — answers its own `404` or refuses the upgrade first. See [WebSocket-Capable Deployment](operations_deploy_advanced.md#websocket-capable-deployment-realtime-api).
- Confirm the deployment actually ships the Realtime API: this is a versioned feature, not every earlier release includes it.
??? failure "`POST /v1/realtime/calls` answers `404`"
That endpoint is the upstream API's WebRTC and SIP transport negotiation, and it is **not served here** — every realtime session runs over the WebSocket. Nothing in the deployment enables it: no setting, no IAM permission and no load balancer configuration changes the answer. See [Transports](api_openai_realtime.md#transports).
- **A browser does not need it.** It connects to the same `WS /v1/realtime` with an [ephemeral client secret](api_openai_realtime.md#ephemeral-client-secrets) in the `Sec-WebSocket-Protocol` list, and captures and plays the audio itself.
- **A client that must speak WebRTC or SIP** — a phone line, or a browser on a lossy network that needs jitter and packet-loss handling — belongs behind a voice-agent framework that terminates the media itself and reaches this API over the WebSocket. See [Put WebRTC or a phone line in front of the gateway](api_openai_realtime.md#put-webrtc-or-a-phone-line-in-front-of-the-gateway).
- **Do not try to route UDP through the deployment's ALB**: its listeners carry HTTP and HTTPS only. A media terminator you run yourself needs its own ingress — see [WebRTC and SIP need their own ingress](operations_deploy_advanced.md#webrtc-and-sip-need-their-own-ingress).
??? failure "A realtime session closes exactly at 8 minutes, or unexpectedly earlier"
Two different things end a [Realtime](api_openai_realtime.md) session, and the close code tells them apart — inspect it in the client's WebSocket close handler.
- **Close code `1000`, reason `session_expired`**: expected. Every session is capped at 8 minutes; reconnect to continue the conversation — see [Session Lifecycle and Limits](api_openai_realtime.md#session-lifecycle-and-limits).
- **Closes with no code from the gateway at all, well before 8 minutes**: the load balancer's idle timeout fired on a quiet stretch between spoken turns. Raise `alb_idle_timeout` to at least 8 minutes (480 seconds) — see [The idle timeout bounds a session](operations_deploy_advanced.md#the-idle-timeout-bounds-a-session).
- **Closes during a deployment, scale-in, or Spot interruption**: the ECS task holding the session was replaced. There is no live handoff between tasks — see [A deploy truncates open sessions](operations_deploy_advanced.md#a-deploy-truncates-open-sessions).
- **Close code `1001`, reason `server_shutdown`**: the deployment was shutting down when the session was still open; reconnect once it is back.
- **Close code `3000`**: a fatal error, not a limit. The reason is `.`, and a terminal `error` event carrying the same detail was sent just before the close frame.
??? failure "A raw WebSocket client gets `403` that looks like an authentication failure"
If the deployment's WAF is enabled with the AWS-managed Common Rule Set (`alb_waf_enabled = true`), its `NoUserAgent_HEADER` rule blocks any request — including a WebSocket upgrade — that carries no `User-Agent` header, with a plain `403` that is easy to mistake for a rejected credential.
- Every mainstream WebSocket client library sets a `User-Agent` automatically; this only surfaces with a hand-rolled client (a bespoke SIP/telephony bridge, a minimal test script).
- Check the WAF sampled requests in the console to confirm `NoUserAgent_HEADER` is the rule that matched, before assuming the API key or ephemeral secret is wrong.
- Have the client send any non-empty `User-Agent`, or exclude the rule for the Realtime path. See [AWS WAF's `NoUserAgent_HEADER` rule reads like an auth failure](operations_deploy_advanced.md#aws-wafs-nouseragent_header-rule-reads-like-an-auth-failure).
??? failure "An ephemeral client secret works on one instance and is rejected on another"
A [Realtime API ephemeral client secret](api_openai_realtime.md#ephemeral-client-secrets) is a signed token with nothing stored server-side, verified by re-checking its signature against a shared key — every instance must sign with the **same** key for that to work.
- With no [`API_KEY`](operations_configuration.md#api-key)-family setting configured at all, each instance falls back to a **random signing key generated per process**, so a secret minted by one instance never verifies on another — the symptom is intermittent rejection that tracks which instance the client's connection happened to land on.
- Set [`REALTIME_CLIENT_SECRET_KEY`](operations_configuration.md#realtime-client-secret-key) explicitly to a value shared by every instance; this also covers a deployment with no API key by design (e.g. behind an IP-restricted ALB).
- A deployment that already configures an API key is unaffected: the signing key is derived from it automatically, and that same key is already shared across instances.
- Rotating the API key or `REALTIME_CLIENT_SECRET_KEY` invalidates every client secret minted before the change, the same as an expired one.
### AWS error → HTTP status mapping
stdapi.ai translates upstream AWS error codes into standard HTTP responses with an OpenAI/Anthropic-style error type. Use this table to map a status code back to its likely AWS cause. HTTP status and error type are as returned on OpenAI-compatible routes (`/v1/...`); Anthropic-compatible routes (`/anthropic/...`) diverge on the two footnoted rows.
| HTTP | Error type | AWS error codes | Typical cause |
|-------|------------------------------|---------------------------------------------------------------------------------------------------------|-----------------------------------------------|
| `400` | `invalid_request_error` | `ValidationException`, `BadRequestException` | Unsupported/invalid request parameters |
| `400` | `invalid_request_error` | `AccessDenied` — on the object an `s3://` input named | The caller's own object cannot be read[^4] |
| `401` | `authentication_error` | `UnrecognizedClientException`, `InvalidSignatureException`, `ExpiredTokenException` | stdapi.ai's AWS credentials missing/expired |
| `403` | `permission_error` | `AccessDeniedException` — on a model call an end user's own role signed | That end user is not allowed that model[^3] |
| `404` | `invalid_request_error`[^1] | `ResourceNotFoundException` | Model or resource not available in the region |
| `429` | `rate_limit_error` | `ThrottlingException`, `TooManyRequestsException`, `ServiceQuotaExceededException` | Bedrock quota / throttling |
| `503` | `feature_unavailable` | `AccessDeniedException`, `AccessDenied` — every other denial | IAM task role lacks permission / model access |
| `503` | `server_error`[^2] | `ServiceUnavailableException`, `InternalServerException`, `ServiceFailureException`, `ReadTimeoutError` | Transient AWS-side error — retry |
[^1]: Anthropic-compatible routes return `not_found_error` instead.
[^2]: Anthropic-compatible routes return HTTP `529` with error type `overloaded_error` instead.
[^3]: Only when [per-user cost attribution](operations_cost_management.md#per-user-attribution) is enabled: the call then carries the end user's identity, and AWS evaluated a policy written about them.
[^4]: Only for a bucket declared in [`AWS_S3_ACCEPTED_BUCKETS`](operations_configuration.md#aws-s3-accepted-buckets), which the deployment reads but does not own — so the refused object is the one the request named. The message names that input, and nothing else. A denial on the deployment's own buckets stays `feature_unavailable`.
!!! note "Where to find the detail"
For security, `401`, `403` and `feature_unavailable` responses returned to clients contain only a generic message — the same one whatever is missing, so that the difference between "no permission" and "not configured" is not disclosed. The full diagnostic detail is captured in the server logs under `error_detail` and can be correlated via the `x-request-id` response header (`request-id` on Anthropic-compatible `/anthropic/...` routes) — see [Logging & Monitoring](operations_logging_monitoring.md).
---
## :material-key-variant: Authentication & Identity
??? failure "Bearer token works, but Anthropic SDK requests fail"
The Anthropic SDK uses a different auth header than OpenAI.
- Use `x-api-key: ` (not `Authorization: Bearer`).
- Set the base URL to `https:///anthropic` (not `/v1`).
- See [API Overview → Anthropic-Compatible API](api_overview.md#using-the-anthropic-compatible-api).
??? failure "A valid Amazon Cognito token is rejected with 401"
The response body is always the same opaque `Unauthorized`; the check that failed is in the server's request log (`error_detail`). Work through the checks in order:
- **Wrong app client**: the token's app client must be listed in [`AWS_COGNITO_CLIENT_IDS`](operations_configuration.md#aws-cognito-client-ids).
- **Wrong token**: an identity token is rejected unless [`AWS_COGNITO_ACCEPT_ID_TOKEN`](operations_configuration.md#aws-cognito-accept-id-token) is enabled. Send the **access** token.
- **Wrong issuer**: pools on the Essentials and Plus tiers can issue `https://issuer-cognito-idp..amazonaws.com/...`. Set [`AWS_COGNITO_ISSUER_TYPE`](operations_configuration.md#aws-cognito-issuer-type) to `updated` for those, and confirm the pool ID matches the pool that minted the token.
- **Missing scope**: a token obtained by signing in with a username and password carries only `aws.cognito.signin.user.admin`. Clear [`AWS_COGNITO_REQUIRED_SCOPES`](operations_configuration.md#aws-cognito-required-scopes), or have clients obtain tokens from the pool's OAuth 2.0 token endpoint.
- **Expired token**: tokens are accepted up to one minute past expiry only. Refresh the token, and check the container clock if expiry errors are constant.
- See [Authentication & Security → Amazon Cognito User Pool Tokens](operations_authentication_security.md#amazon-cognito-user-pool-tokens).
??? failure "The server does not start after enabling authentication"
A half-applied credential configuration is refused rather than accepted, so the deployment never runs unauthenticated by accident. The startup log's `error_detail` names the exact rule:
- **Missing allowlist**: [`AWS_COGNITO_CLIENT_IDS`](operations_configuration.md#aws-cognito-client-ids) is required with a pool.
- **Setting without a pool**: any other `AWS_COGNITO_*` variable requires [`AWS_COGNITO_USER_POOL_ID`](operations_configuration.md#aws-cognito-user-pool-id).
- **Mode conflict**: [`AUTHENTICATION_MODE`](operations_configuration.md#authentication-mode) must not demand a method that is unconfigured, nor ignore one that is configured — use `any` to accept both.
- **Empty API key**: the SSM parameter or Secrets Manager secret named by [`API_KEY_SSM_PARAMETER`](operations_configuration.md#api-key-ssm) or [`API_KEY_SECRETSMANAGER_SECRET`](operations_configuration.md#api-key-secretsmanager-secret) exists but holds an empty value — populate it, or unset the setting to run without an API key deliberately.
- **Signing keys unreachable**: the pool's public keys are read at startup over HTTPS. Check the pool ID, and that the task can reach the internet or a suitable endpoint for outbound HTTPS.
??? failure "An MCP client or agent cannot discover how to authenticate"
Discovery is off until it is configured, and a client that finds nothing falls back to asking the user for a key.
- **Nothing published**: `GET /.well-known/oauth-protected-resource` answering `404` means [`OAUTH_RESOURCE_IDENTIFIER`](operations_configuration.md#oauth-resource-identifier) is unset. That is the default, not a fault — set it to turn discovery on. With an [`AWS_COGNITO_USER_POOL_ID`](operations_configuration.md#aws-cognito-user-pool-id) configured it is the only setting needed; otherwise set [`OAUTH_AUTHORIZATION_SERVERS`](operations_configuration.md#oauth-authorization-servers) too.
- **Nothing in the challenge**: a `401` carrying a bare `WWW-Authenticate: Bearer` means the same thing. Once configured it also carries `resource_metadata="…"` and, with [`OAUTH_SCOPES_SUPPORTED`](operations_configuration.md#oauth-scopes-supported), `scope="…"`.
- **Browser-hosted client**: a client running in a page reads the document cross-origin and needs its origin in [`CORS_ALLOW_ORIGINS`](operations_configuration.md#cors-allow-origins).
- **The client asks for `/.well-known/openid-configuration`**: stdapi.ai is a resource server and deliberately does not serve it; the client should follow `authorization_servers` to the issuer's own document.
- See [Authentication & Security → Authentication Discovery for Agents](operations_authentication_security.md#authentication-discovery-for-agents).
??? failure "A client reports the protected resource does not match the expected URL"
Clients compare the published `resource` against the URL they dialled character by character, so any difference aborts the flow.
- **Scheme**: set [`OAUTH_RESOURCE_IDENTIFIER`](operations_configuration.md#oauth-resource-identifier) to `https://…` when clients reach the deployment over TLS, even though the container itself listens on plain HTTP behind the load balancer.
- **Host**: it must be the public hostname clients use, not the internal service or task address.
- **Port**: an explicit `:443` on the client's URL does not match an identifier without one, and vice versa. Drop the default port on both sides.
- **Path**: the identifier is an origin. Do not append `/mcp`, `/v1`, or a trailing slash — one document at the root already covers every surface of the deployment.
??? failure "The server does not start after configuring authentication discovery"
The three settings describe one document, so an incomplete set is refused rather than published half-formed. The startup log's `error_detail` names the rule:
- **No authorization server**: [`OAUTH_AUTHORIZATION_SERVERS`](operations_configuration.md#oauth-authorization-servers) is required with [`OAUTH_RESOURCE_IDENTIFIER`](operations_configuration.md#oauth-resource-identifier) unless [`AWS_COGNITO_USER_POOL_ID`](operations_configuration.md#aws-cognito-user-pool-id) is set, which supplies the issuer; a document naming none leaves a client unable to obtain a token.
- **Issuer contradicting the pool**: with a user pool configured, the published issuers must include the pool's own — otherwise clients are sent to an authorization server whose tokens every request refuses. Leave [`OAUTH_AUTHORIZATION_SERVERS`](operations_configuration.md#oauth-authorization-servers) empty to publish exactly the pool issuer, or list it alongside the others.
- **Setting without an identifier**: the authorization servers and the scopes describe a document that is not published without [`OAUTH_RESOURCE_IDENTIFIER`](operations_configuration.md#oauth-resource-identifier).
- **Malformed value**: the identifier is an origin with no path or query, each issuer is an `https` URL with no query or fragment, and a scope carries no space or quote.
??? failure "OIDC/Cognito redirect loop or 401 from the ALB"
Authentication is enforced by the ALB listener, not stdapi.ai.
- Verify the OIDC issuer URL, client ID, client secret, and redirect URI in the ALB listener rule.
- For Cognito, confirm the app client is configured as a "confidential" client with a client secret.
- See [Authentication & Security → via Application Load Balancer (ALB)](operations_authentication_security.md#via-application-load-balancer-alb).
---
## :material-arrow-right: Next Steps
- :material-cog: [**Configuration Reference**](operations_configuration.md) — Every environment variable
- :material-server-network: [**Advanced Deployment**](operations_deploy_advanced.md) — VPC integration, manual ECS, multi-region
- :material-lock: [**Authentication & Security**](operations_authentication_security.md) — API keys, OIDC/Cognito, IAM
- :material-terraform: [**Terraform Module Docs**](https://github.com/stdapi-ai/terraform-aws-stdapi-ai) — All module inputs and outputs
- :material-github: [**GitHub Issues**](https://github.com/stdapi-ai/stdapi.ai/issues) — Report a bug or ask a question
- :material-email-outline: [**Contact**](contact.md) — Reach the team directly
---
## Overview
Source: https://stdapi.ai/md/use_cases.md
# :material-puzzle: Use Cases
Discover how to integrate stdapi.ai with popular AI applications and tools. stdapi.ai's OpenAI, Anthropic, and Cohere-compatible APIs are already spoken by hundreds of applications and tools, and adopting it takes **three client-side changes**: the base URL, the API key, and — where the name differs from what the client already sends — the model name, now picked from every provider in the catalogue rather than one vendor's list.
**Why use stdapi.ai for integrations?**
- **Three client-side changes** - Point your application at the new base URL and API key, and update the model name if the one it already sends isn't in this deployment's catalogue
- **Access [100+ models](models.md)** - Claude, OpenAI GPT, xAI Grok, Kimi, DeepSeek, Qwen, GLM, Nova, Llama, Stability AI, and more
- **Enterprise data control** - The gateway runs in your own AWS account — no third party sits between your users and your models
- **Pay-per-use pricing** - Pay Amazon Bedrock rates for actual usage, with no markup and no per-seat fees
- **AWS-native features** - Leverage prompt caching, reasoning modes, and guardrails through standard OpenAI, Anthropic, and Cohere APIs
- **Three-dialect API compatibility** - Use the OpenAI, Anthropic, or Cohere SDK with the same deployment
!!! tip "Try it before committing to anything"
Run stdapi.ai on your laptop with the [free community Docker image](operations_getting_started_local.md) — or deploy to AWS with a [14-day free trial](operations_getting_started.md). Both editions expose the same API surface, so the integrations below are configured the same way against either.
## :material-lightning-bolt: How Integration Works
Every integration on this page follows the same four steps, and that's the whole process:
1. **Deploy** stdapi.ai — [on AWS](operations_getting_started.md) or [locally with Docker](operations_getting_started_local.md)
2. **Copy** your endpoint URL and API key
3. **Paste** them into your tool's AI-provider settings (or your SDK's `base_url`)
4. **Name** a model this deployment serves — the [Models](models.md) page shows them and [`GET /search_models`](api_search_models.md) lists what your deployment actually serves; a name the catalogue does not hold returns `404` rather than a lookalike
```python
from openai import OpenAI
client = OpenAI(base_url="https://your-endpoint/v1", api_key="YOUR_KEY")
# Then name a model from the catalogue — the rest of your application is unchanged
```
## :material-view-grid: Choose Your Integration
Pick the category that matches your goal — categories marked :material-book-open-variant: have a dedicated step-by-step guide.
Throughout this page, tools in **bold** are driven end to end by an automated test suite against a real deployment; the others have a documented setup.
- :material-code-braces: **[AI Coding Assistants](#coding-assistants)** :material-book-open-variant:
Frontier coding models in your IDE and terminal
- :material-brain: **[Autonomous Agents](#autonomous-agents)** :material-book-open-variant:
Self-directed agents on infrastructure you control
- :material-chat: **[Chat Interfaces](#chat-interfaces)** :material-book-open-variant:
Private ChatGPT alternative for your organization
- :material-graph-outline: **[Workflow Automation](#workflow-automation)** :material-book-open-variant:
AI steps in business processes, no code required
- :material-microphone-message: **[Voice & Audio](#voice-audio)** :material-book-open-variant:
Voice agents, transcription, and subtitles
- :material-magnify: **[RAG & Semantic Search](#rag-semantic-search)** :material-book-open-variant:
Ground AI answers in your own documents
- :material-image-multiple: **[Content & Media Generation](#media-generation)**
Images and video for creative pipelines
- :material-note-text: **[Knowledge Management](#knowledge-management)**
Private AI inside your notes and research
- :material-robot: **[Team Chatbots](#team-chatbots)**
Assistants in Slack, Discord, and Teams
### :material-code-braces: Developer Tools — AI Coding Assistants { #coding-assistants }
*For teams that want frontier coding models in their IDEs and terminals — without sending code to third-party AI clouds.*
Enhance your development workflow with AI-powered coding assistants. stdapi.ai integrates with popular IDEs and AI development frameworks, allowing you to leverage Amazon Bedrock models (Claude, Kimi thinking, Qwen3 Coder Next) for code completion, generation, and intelligent assistance.
**What you can do:**
- **Code completion** - Real-time suggestions as you type in VS Code, JetBrains IDEs
- **Code generation** - Natural language to code with Claude and specialized coding models
- **Codebase understanding** - Chat with your codebase, explain functions, refactor code
**Popular tools:** **Claude Code**, **Codex**, **Qwen Code**, **pi**, Cline, OpenCode, Zed, JetBrains AI Assistant
**[AI Coding Assistants Guide](use_cases_coding_assistants.md)** — Universal setup for IDEs and development frameworks
---
### :material-brain: Autonomous Agents — Research & Task Automation { #autonomous-agents }
*For builders of agents that must run on infrastructure you control.*
Build self-directed AI agents that can plan, execute, and refine complex tasks autonomously. Integrate stdapi.ai with agent frameworks to create intelligent systems powered by Amazon Bedrock that can conduct research, automate workflows, and solve multi-step problems.
**What you can build:**
- **Personal AI assistants** - Autonomous agents connected to messaging, email, and smart home
- **Research agents** - Autonomous web research, data gathering, and analysis
- **Multi-agent systems** - Collaborative agents for complex problem-solving
- **Task automation** - Self-improving workflows that adapt to results
- **Code agents** - Autonomous development and testing systems
- **Grounded agents** - Retrieval the agent calls for itself, over [vector stores](api_openai_vector_stores.md) holding your own documents
- **Resumable sessions** - Threads kept server-side with the [Conversations API](api_openai_conversations.md) and continued by id, rather than resent every turn
- **Per-caller identity** - Agents that [discover how to authenticate](operations_authentication_security.md#authentication-discovery-for-agents) and whose spend is [reported per end user](operations_cost_management.md#per-user-attribution)
**Compatible frameworks:** **OpenClaw**, **Hermes**, **LangChain**, **Pydantic AI**, **OpenAI Agents SDK**, **Agno**, **LlamaIndex**, **LiteLLM**, LangGraph, CrewAI, Strands Agents
All agent frameworks that support OpenAI or Anthropic SDKs work immediately — point the SDK's base URL to stdapi.ai. See the [API overview](api_overview.md) for connection details.
**[Python Client Libraries Guide](use_cases_python_libraries.md)** — Configuring LangChain, pydantic-ai and the OpenAI Agents SDK directly against stdapi.ai
**[Autonomous Agent CLIs Guide](use_cases_autonomous_agents.md)** — Configuring Hermes and OpenClaw directly against stdapi.ai
!!! tip "Give your agents AI capabilities via MCP"
stdapi.ai is also a native [MCP server](api_overview.md#mcp-model-context-protocol): agents can call image generation, speech synthesis, transcription, file management, and model discovery as MCP tools — no custom integration code needed.
---
### :material-chat: Chat Interfaces — Private ChatGPT Alternative { #chat-interfaces }
*For organizations replacing per-seat AI subscriptions with a private, pay-per-use assistant.*
Build ChatGPT-like experiences with Amazon Bedrock models and complete privacy control. Deploy feature-rich web interfaces that provide familiar chat experiences while keeping all data within your AWS environment.
**What you can build:**
- **Private team chat** - ChatGPT-style interface for your organization
- **Customer support assistant** - AI-powered help desk with your data
- **Internal knowledge base** - RAG-enabled chat with document search
- **Multi-modal applications** - Process text, voice, images, and documents
- **Voice chat & image generation** - Speech input/output and in-chat image creation through the same endpoint
**Popular tools:** **Open WebUI**, LobeHub, AnythingLLM, LibreChat
**[Open WebUI Integration Guide](use_cases_openwebui.md)** — Complete setup with Terraform deployment examples
**[LobeHub Integration Guide](use_cases_lobehub.md)** — Complete setup with Terraform deployment examples
---
### :material-graph-outline: Workflow Automation — AI-Powered Business Processes { #workflow-automation }
*For teams adding AI steps to business processes without writing code.*
Integrate Amazon Bedrock AI into your business processes and automation workflows. Connect models to hundreds of services and APIs through visual workflow builders, enabling sophisticated AI-powered automation without writing code.
**What you can automate:**
- **Customer support** - Auto-classify tickets, generate responses, route intelligently
- **Content creation** - Automated blog posts, social media, email campaigns
- **Data processing** - Extract, transform, and analyze data with AI
- **Document workflows** - Automated summarization, translation, and classification
- **Bulk runs** - Push a backlog through the [Batch API](api_openai_batches.md) asynchronously, at the Amazon Bedrock batch price
- **Content safety** - Screen user-generated content with the [Moderations API](api_openai_moderations.md)
**Popular tools:** **n8n**, **Haystack**, Langflow, Dify, Flowise
!!! note "Make & Zapier"
Make and Zapier can call stdapi.ai through their generic HTTP/webhook modules, but their native OpenAI modules do not support custom endpoints.
**[n8n Integration Guide](use_cases_n8n.md)** — Complete setup for AI workflow automation
---
### :material-microphone-message: Voice & Audio — Speech Applications & Voice Agents { #voice-audio }
*For products that need speech in and speech out — without adding a second AI vendor.*
Build voice-first applications on the same OpenAI-compatible endpoint: text-to-speech with Amazon Polly voices, speech-to-text with Amazon Transcribe and Bedrock audio models (including streaming and speaker diarization), and speech translation with subtitle output.
**What you can build:**
- **Speech-to-speech agents** - Hold a spoken conversation over one WebSocket with the [Realtime API](api_openai_realtime.md) — the model handles turn taking and interruption, and a live transcript comes back with the audio
- **Voice agents** - Real-time conversational agents for phone, web, and support lines
- **Meeting intelligence** - Transcription with speaker diarization and AI summaries
- **Live transcription** - Return each phrase as it is recognized instead of after the whole recording, with [streamed transcriptions](api_openai_audio_transcriptions.md#streaming)
- **Subtitles & dubbing** - Transcribe and translate audio with SRT/VTT subtitle output
- **Long-form narration** - Speak up to [100,000 characters](api_openai_audio_speech.md#long-input) per request, streamed as it is synthesized
- **Voice interfaces** - Add speech input/output to chat interfaces and internal tools
**Popular frameworks:** **Pipecat**, **LiveKit Agents**, TEN Framework — all accept a custom OpenAI-compatible base URL for LLM, speech-to-text, and text-to-speech services, and the first two are also what put [WebRTC or a phone line](api_openai_realtime.md#transports) in front of a realtime session
**Popular tools:** Home Assistant Assist (via the **wyoming-openai** proxy)
**[Home Assistant Voice Guide](use_cases_home_assistant.md)** — Complete setup for local voice assistants backed by Amazon Transcribe and Amazon Polly
!!! tip "Getting started"
Point the framework's OpenAI plugin at your stdapi.ai `/v1` URL. See the [Audio Speech](api_openai_audio_speech.md), [Audio Transcriptions](api_openai_audio_transcriptions.md), and [Audio Translations](api_openai_audio_translations.md) APIs for supported models and formats.
---
### :material-magnify: RAG & Semantic Search — Embeddings and Reranking { #rag-semantic-search }
*For teams grounding AI answers in their own documents and data.*
Build retrieval-augmented generation and semantic search pipelines with Bedrock embedding models and Cohere-compatible reranking — two-stage retrieval (embed, then rerank) through one deployment.
**What you can build:**
- **Managed retrieval** - Attach files to a [vector store](api_openai_vector_stores.md) and search it by meaning, with no chunker, embedder, or vector database of your own to run
- **Retrieval the model runs itself** - Name a store as a `file_search` tool and the model searches it mid-answer, citing the files it drew on
- **Your existing knowledge base** - Address an Amazon Bedrock knowledge base you already operate [as a vector store](api_openai_vector_stores.md#knowledge-base-stores) — searched and extended, never recreated
- **Document ingestion** - Parse PDFs and office documents into Markdown with Docling before embedding
- **RAG pipelines** - Ground model answers in your documents with [embeddings](api_openai_embeddings.md)
- **Two-stage retrieval** - Improve relevance with the [Rerank API](api_cohere_rerank.md) on top of vector search
- **Semantic search** - Search by meaning across documents, tickets, and knowledge bases
- **Multimodal search** - Embed text and images with models like Cohere Embed v4
**Popular tools:** **Docling Serve** for document parsing, **Haystack**, **Agno**, **LlamaIndex**, RAGFlow, LightRAG for retrieval — the managed stores need no vector database, and an assembled pipeline works with any of them (pgvector, Qdrant, and others store the vectors; stdapi.ai serves the embeddings)
**[RAG Pipelines Guide](use_cases_rag.md)** — Configuring document parsing, embeddings, Cohere-compatible reranking, and generation together
**[RAGFlow Integration Guide](use_cases_ragflow.md)** — Complete setup with Terraform deployment examples
---
### :material-image-multiple: Content & Media Generation — Images and Video { #media-generation }
*For creative and marketing pipelines that generate visuals at scale.*
Generate and edit visual content with Amazon Bedrock media models through the standard OpenAI Images and Videos APIs — from marketing assets to fully automated content pipelines.
**What you can build:**
- **Image generation** - Text-to-image with Amazon Nova Canvas and Stability AI models via [Images Generations](api_openai_images_generations.md)
- **Image editing** - Inpainting, outpainting, and style transfer via [Images Edits](api_openai_images_edits.md)
- **Video generation** - Asynchronous text/image-to-video with Amazon Nova Reel and Luma Ray via the [Videos API](api_openai_videos.md)
- **Safe publishing pipelines** - Combine generation with the [Moderations API](api_openai_moderations.md) for automated content review
**Popular tools:** Open WebUI (built-in image generation), n8n media workflows, or the APIs directly
!!! tip "Getting started"
Any tool that supports the OpenAI Images API works by pointing it at your stdapi.ai `/v1` URL. Video generation requires S3 storage. No dedicated guide yet — see the [Videos API](api_openai_videos.md) for setup and the [API overview](api_overview.md) for connection details.
---
### :material-note-text: Knowledge Management — AI-Enhanced Notes & Research { #knowledge-management }
*For individuals and teams adding private AI to their notes and research.*
Transform your knowledge base with AI-powered insights and generation. Integrate stdapi.ai with note-taking applications to add semantic search, writing assistance, and intelligent content organization.
**What you can do:**
- **AI writing assistance** - Generate, edit, and improve your writing
- **Semantic search** - Find notes by meaning, not just keywords
- **Auto-summarization** - Extract key points from long documents
- **Smart organization** - Automatic tagging, linking, and categorization
**Compatible tools:** Obsidian (Copilot plugin), Khoj (self-hosted), SiYuan
!!! tip "Getting started"
These tools accept a custom OpenAI-compatible endpoint for both chat and embedding models. Point them to your stdapi.ai `/v1` URL. No dedicated guide yet — [see the API overview](api_overview.md) for connection details.
---
### :material-robot: Team Chatbots & Assistants — Slack, Discord, Teams Integration { #team-chatbots }
*For support and ops teams meeting users where they already chat.*
Deploy intelligent AI assistants to your team's communication platforms powered by Amazon Bedrock models.
**What you can build:**
- **Team Q&A bot** - Answer common questions in Slack or Teams
- **Documentation assistant** - Search and cite internal docs in real-time
- **Task automation** - Create tickets, schedule meetings, update databases via chat
- **Moderated channels** - Screen messages with the [Moderations API](api_openai_moderations.md)
**Compatible platforms:** Dify, Chatwoot (Captain, self-hosted), Typebot — or build directly for Slack, Discord, and Microsoft Teams with the OpenAI or Anthropic SDK
!!! tip "Getting started"
Build bots using the OpenAI or Anthropic SDK, pointing to your stdapi.ai endpoint. No dedicated guide yet — [see the API overview](api_overview.md) for connection details.
---
## :material-help-circle: Common Questions
- **Where does my data go?** The gateway runs in your own AWS account, so no third party sits between your users and your models: inference runs on the AWS services and regions you enable, and Amazon Bedrock does not share prompts with model providers or use them for training. [Data sovereignty & compliance →](operations_compliance.md)
- **What does it cost?** $0.10/container-hour for the gateway — the Terraform module runs one container per Availability Zone by default — plus Amazon Bedrock rates, with no markup and no per-seat fees. Each end user's share can be reported separately in Cost Explorer, from the invoice rather than an estimate. [Licensing & pricing →](operations_licensing.md) · [Cost management →](operations_cost_management.md)
- **Am I locked in?** No — stdapi.ai speaks the standard OpenAI, Anthropic, and Cohere APIs. Leaving is the same client-side change that got you in.
## :material-arrow-right: Ready to Get Started?
- :material-rocket-launch: [**Deploy to AWS**](operations_getting_started.md) — Production-ready with two Terraform commands (14-day free trial)
- :material-docker: [**Try Locally with Docker**](operations_getting_started_local.md) — Free community image for development and testing
- :material-book-open-variant: [**API Overview**](api_overview.md) — Endpoints, parameters, and usage examples
- :material-email-outline: [**Contact**](contact.md) — Integration questions, sales, and private offers
---
## Open WebUI Integration
Source: https://stdapi.ai/md/use_cases_openwebui.md
# :material-chat: Open WebUI Integration
Connect Open WebUI to stdapi.ai as an OpenAI-compatible backend. Access Amazon Bedrock models through Open WebUI's chat interface—it works out of the box as a private ChatGPT alternative running on your AWS infrastructure.
## :material-information-outline: About Open WebUI
**🔗 Links:** [Website](https://openwebui.com/) | [GitHub](https://github.com/open-webui/open-webui) | [Documentation](https://docs.openwebui.com/)
Open WebUI is the leading open-source ChatGPT alternative. It provides a feature-rich, self-hosted web interface that operates entirely under your control, offering a ChatGPT-like experience while maintaining complete data privacy.
**Key Features:**
- ⭐ **140,000+ GitHub stars** - Most popular open-source AI chat interface
- **ChatGPT-like UI** - Familiar interface your team already knows
- **Multi-modal capabilities** - Text, voice, images, and document processing
- **RAG & embeddings** - Upload documents, search with semantic understanding
- **Extensible platform** - Plugins, custom functions, and community tools
- **Complete privacy** - Self-hosted, all data stays in your infrastructure
## :material-help-circle-outline: Why Open WebUI + stdapi.ai?
- :material-swap-horizontal: __Point It at stdapi.ai__
stdapi.ai provides an OpenAI-compatible API. Set the endpoint URL and API key, and change the model only if the one you want isn't already in Open WebUI's list—it shows whatever your deployment serves, across every provider in the catalogue.
- :material-aws: __Access Amazon Bedrock Models__
Claude with reasoning, Nova, Llama, DeepSeek, Stable Diffusion, and 100+ models through Open WebUI's familiar chat interface.
- :material-application-cog: __Full Multi-Modal Support__
Text chat, voice input/output, image generation/editing, document RAG—all AWS AI services unified through one interface.
- :material-lock: __Enterprise Data Privacy__
All processing stays in your AWS account. Complete infrastructure control with AWS security, compliance, and data sovereignty.
- :material-currency-usd-off: __Pay-Per-Use Pricing__
No ChatGPT subscriptions. Pay only Amazon Bedrock rates for actual usage—no monthly minimums or per-user fees.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
openwebui[" Open WebUI"] --> stdapi[" stdapi.ai"]
stdapi --> bedrock[" Amazon Bedrock"]
stdapi --> transcribe[" Amazon Transcribe"]
stdapi --> polly[" Amazon Polly"]
```
## :material-connection: Connect Your Own Instance
Point any running Open WebUI instance—wherever you host it—at your stdapi.ai gateway. Nothing below requires the AWS sample in [Part 2](#deploy-the-full-stack-on-aws).
### :material-check-circle: Prerequisites
!!! info "What You'll Need"
- ✓ **stdapi.ai deployed** - [See deployment guide](operations_getting_started.md)
- ✓ **Your stdapi.ai URL** - e.g., `https://api.example.com`
- ✓ **Your API key** - From Terraform output or configuration
- ✓ **Open WebUI instance** - Running or ready to deploy (see [Part 2](#deploy-the-full-stack-on-aws) for a Terraform-deployed option)
### :material-cog: Configuration
Open WebUI is configured entirely through environment variables. The sections below focus on the stdapi.ai integration. Use the same stdapi.ai key for all `*_OPENAI_API_KEY` entries. For more details on Open WebUI settings, refer to the official [Open WebUI Environment Variable Configuration](https://docs.openwebui.com/getting-started/env-configuration/) documentation.
!!! warning "Each section needs its own connection settings"
Open WebUI does not fall back from `RAG_OPENAI_*`, `IMAGES_OPENAI_*`, or `AUDIO_*_OPENAI_*` to the core `OPENAI_API_*` pair — a missing pair disables that feature instead of inheriting the Core Connection. Set the base URL, key, and model explicitly for every section you enable.
!!! warning "These settings are read once, on first boot"
Open WebUI reads its connection settings from the environment only the first time it starts against a given data directory, then stores them in its own database. Changing an environment variable afterwards has no effect until you either update the setting from the admin UI or start from a fresh `DATA_DIR`.
!!! note "Model choice"
In every section below, pick any Bedrock-available model that matches the operation's modality — a chat model for the Core Connection, an embedding model for RAG Embeddings, and so on.
#### :material-connection: Core Connection
Enables: Chat completions and Open WebUI background tasks (titles, summarization).
!!! example "Environment Variables"
```bash
OPENAI_API_BASE_URL=https://YOUR_STDAPI_URL/v1
OPENAI_API_KEY=YOUR_STDAPI_KEY
TASK_MODEL_EXTERNAL=amazon.nova-micro-v1:0
```
Use a fast, low-cost chat model for `TASK_MODEL_EXTERNAL`. Open WebUI calls `POST /v1/chat/completions` for chat and background tasks (see [Chat Completions API](api_openai_chat_completions.md)).
#### :material-database: RAG Embeddings
Enables: Document ingestion and semantic search for RAG.
!!! example "Environment Variables"
```bash
RAG_EMBEDDING_ENGINE=openai
RAG_OPENAI_API_BASE_URL=https://YOUR_STDAPI_URL/v1
RAG_OPENAI_API_KEY=YOUR_STDAPI_KEY
RAG_EMBEDDING_MODEL=cohere.embed-v4:0
```
Open WebUI calls `POST /v1/embeddings` (see [Embeddings API](api_openai_embeddings.md)).
#### :material-sort-variant: RAG Reranking
Enables: Hybrid search, with retrieved documents reordered by relevance before they reach the model.
!!! example "Environment Variables"
```bash
ENABLE_RAG_HYBRID_SEARCH=true
RAG_RERANKING_ENGINE=external
RAG_EXTERNAL_RERANKER_URL=https://YOUR_STDAPI_URL/cohere/v2/rerank
RAG_EXTERNAL_RERANKER_API_KEY=YOUR_STDAPI_KEY
RAG_RERANKING_MODEL=cohere.rerank-v3-5:0
```
Open WebUI's external reranker speaks the Cohere dialect, so it targets the Cohere-compatible route instead of `/v1` (see [Cohere Rerank API](api_cohere_rerank.md)). Give the full endpoint path: Open WebUI sends the request to the URL as-is and appends nothing.
!!! tip "Regional availability"
Amazon Bedrock serves reranking from a subset of regions only. Keep at least one of them in [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions); stdapi.ai fails over to it automatically.
Without an external reranker, Open WebUI falls back to a local Sentence-Transformers cross-encoder that it downloads from Hugging Face at startup—unavailable when `OFFLINE_MODE` is enabled.
!!! tip "Chunk size determines whether reranking has anything to do"
Set `CHUNK_SIZE` small enough that each retrieval chunk covers one self-contained idea. A chunk size large enough to fold a whole document into a single chunk leaves the reranker nothing to reorder—there is only one candidate to rank.
#### :material-image: Image Generation
Enables: Text-to-image creation inside chats.
!!! example "Environment Variables"
```bash
ENABLE_IMAGE_GENERATION=true
IMAGE_GENERATION_ENGINE=openai
IMAGES_OPENAI_API_BASE_URL=https://YOUR_STDAPI_URL/v1
IMAGES_OPENAI_API_KEY=YOUR_STDAPI_KEY
IMAGE_GENERATION_MODEL=stability.stable-image-core-v1:1
```
Open WebUI calls `POST /v1/images/generations` (see [Images Generations API](api_openai_images_generations.md)).
#### :material-image-edit: Image Editing
Use Open WebUI's image editor to upload an image and describe the change. Masking is not configured.
Enables: Image edits and transformations in the editor.
!!! example "Environment Variables"
```bash
ENABLE_IMAGE_EDIT=true
IMAGE_EDIT_ENGINE=openai
IMAGES_EDIT_OPENAI_API_BASE_URL=https://YOUR_STDAPI_URL/v1
IMAGES_EDIT_OPENAI_API_KEY=YOUR_STDAPI_KEY
IMAGE_EDIT_MODEL=stability.stable-image-control-structure-v1:0
```
Pick any image-editing model that supports edits without a mask. Open WebUI calls `POST /v1/images/edits` (see [Images Edits API](api_openai_images_edits.md)).
#### :material-microphone: Speech to Text (STT)
Enables: Voice input and audio transcription.
!!! example "Environment Variables"
```bash
AUDIO_STT_ENGINE=openai
AUDIO_STT_OPENAI_API_BASE_URL=https://YOUR_STDAPI_URL/v1
AUDIO_STT_OPENAI_API_KEY=YOUR_STDAPI_KEY
AUDIO_STT_MODEL=amazon.transcribe
```
Open WebUI calls `POST /v1/audio/transcriptions` (see [Audio Transcriptions API](api_openai_audio_transcriptions.md)).
!!! tip "A cheaper transcription model"
Setting `AUDIO_STT_MODEL=amazon.nova-2-sonic-v1:0` transcribes through [Amazon Nova Sonic](api_openai_audio_transcriptions.md#amazon-nova-sonic), the lowest-cost option here, punctuated and in the language spoken. It returns plain text with no timestamps and takes recordings up to 10 minutes — ample for chat voice input, but keep `amazon.transcribe` if you also transcribe long meeting recordings from the same setting.
#### :material-volume-high: Text to Speech (TTS)
Enables: Spoken responses from chat outputs.
!!! example "Environment Variables"
```bash
AUDIO_TTS_ENGINE=openai
AUDIO_TTS_OPENAI_API_BASE_URL=https://YOUR_STDAPI_URL/v1
AUDIO_TTS_OPENAI_API_KEY=YOUR_STDAPI_KEY
AUDIO_TTS_MODEL=amazon.polly-neural
```
Open WebUI calls `POST /v1/audio/speech` (see [Audio Speech API](api_openai_audio_speech.md)).
!!! warning "TTS language detection"
Open WebUI generates audio in small chunks, which makes language auto-detection inconsistent. Disable auto-detection by setting the stdapi.ai environment variable `DEFAULT_TTS_LANGUAGE` to a fixed language (for example, `en-US`).
#### :material-tools: MCP Tool Server
Enables: Chat models calling stdapi.ai endpoints as tools, including the ones Open WebUI has no native feature for—such as video generation.
Open WebUI supports MCP servers over Streamable HTTP (v0.6.31 and later), the transport stdapi.ai exposes at `/mcp`. Turn it on with the stdapi.ai environment variables:
!!! example "Environment Variables (stdapi.ai)"
```bash
ENABLE_MCP_STREAMABLE_HTTP=true
MCP_INCLUDE_TOOLS=openai_video_generation,openai_video_get,search_models
```
Every endpoint is exposed as a tool by default. Restrict the list with [`MCP_INCLUDE_TOOLS`](operations_configuration.md#mcp-include-tools) so models see only the tools they need, and keep the ones already wired natively—chat, images, speech, embeddings—out of it.
Then register the server in Open WebUI. MCP connections are admin-only and configured in the interface, not through environment variables:
1. Open **Admin Settings → External Tools**
2. Click **+ (Add Server)** and set **Type** to **MCP (Streamable HTTP)**
3. Set the server URL to `https://YOUR_STDAPI_URL/mcp`
4. Set **Auth** to **Bearer** and paste your stdapi.ai API key
5. Save, then enable the tool in a chat with **+ → Integrations → Tools**
See [MCP tools](api_overview.md#mcp-model-context-protocol) for the full tool list.
### :material-alert-outline: Known Issues
!!! warning "The model selector lists every model, not only the chat ones"
Open WebUI populates its selector from `GET /v1/models`, and that endpoint
answers with every model this deployment serves across every modality — as
OpenAI's own does. So embedding, image, speech and moderation models appear
beside the chat ones, and picking one fails at the first message.
Turn off the ones you do not want in the Open WebUI admin panel. To see which
models answer which route before choosing, query
[`/v1/search_models`](api_search_models.md), which reports each model's
capabilities rather than just its name.
!!! note "Per-user cost attribution needs an identifier Open WebUI does not send"
Open WebUI identifies the signed-in user to its backend with `X-OpenWebUI-User-*` headers (`ENABLE_FORWARD_USER_INFO_HEADERS`), not with the OpenAI `safety_identifier` field. [Per-user attribution](operations_cost_management.md#per-user-attribution) reads that field, or an authenticated caller — neither of which one shared connection provides — so every chat is billed to the deployment's own identity. Where the split matters, give each team its own [model alias](operations_configuration.md#model-aliases-configuration) as a separate Open WebUI connection, and read the totals from Amazon Bedrock model invocation logs.
## :material-rocket-launch: Deploy the Full Stack on AWS
The Terraform sample below is one worked example of a credible AWS deployment, not the only architecture that works. stdapi.ai's gateway is a normal HTTP service, and Open WebUI can run wherever you already operate it—your own ECS, EKS, EC2, another cloud, or a laptop.
### :material-sitemap: Architecture
The diagram below is the topology the [sample](#whats-included) builds: a browser-facing chat application and its AI gateway, both on ECS Fargate, in one VPC you own.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true, 'nodeSpacing': 20, 'rankSpacing': 40, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart TB
user["👤 Your users (browser)"]
subgraph public["Your VPC · public subnets"]
alb[" Application Load Balancer HTTPS · ACM certificate"]
end
subgraph private["Your VPC · private app subnets — no inbound route from the internet"]
openwebui[" Open WebUI ECS Fargate"]
stdapi[" stdapi.ai ECS Fargate"]
tools[" SearXNG · Playwright ECS Fargate"]
aurora[" Aurora PostgreSQL Serverless v2 + pgvector"]
valkey[" ElastiCache Valkey TLS + auth token"]
egress[" NAT gateways · one per AZ + free S3 gateway endpoint"]
aurora ~~~ tools
end
subgraph regional["AWS service endpoints · your account, the regions you configure"]
bedrock[" Amazon Bedrock"]
polly[" Amazon Polly"]
transcribe[" Amazon Transcribe"]
s3[" Amazon S3 SSE-KMS"]
cw[" Amazon CloudWatch logs · metrics · alarms"]
bedrock ~~~ transcribe ~~~ cw
polly ~~~ s3
end
user -->|"HTTPS · TLS 1.2+ (HTTP until a domain is set)"| alb
alb -->|"HTTP · private subnet"| openwebui
openwebui -->|"OpenAI + Cohere API · API key private DNS, no public endpoint"| stdapi
openwebui --> aurora
openwebui --> valkey
openwebui --> tools
openwebui --> egress
stdapi --> egress
egress -->|"HTTPS · SigV4"| bedrock
egress -->|"HTTPS · SigV4"| polly
egress -->|"HTTPS · SigV4"| transcribe
egress -->|"S3 gateway endpoint"| s3
egress --> cw
```
Two properties of this topology are worth reading off the picture. The gateway has **no listener of its own**: Open WebUI resolves it through AWS Cloud Map private DNS inside the VPC, so the load balancer is the only thing with a public address, and it only ever forwards to Open WebUI. And every store that holds your users' content — the Aurora database, the Valkey cache, the S3 bucket — sits inside the account boundary; nothing in the picture is operated by a third party.
#### What Each AWS Service Does Here
| AWS service | Role in this integration | Where it is configured |
| --- | --- | --- |
| **Amazon ECS on AWS Fargate** | Runs Open WebUI, the stdapi.ai gateway, SearXNG and Playwright as separate services with independent auto-scaling | Terraform sample |
| **Elastic Load Balancing** | The single public entry point; terminates TLS with an ACM certificate and forwards only to Open WebUI | Terraform sample |
| **AWS Cloud Map** | Private DNS name that lets Open WebUI reach the gateway without exposing it | Terraform sample (`service_discovery_dns_name`) |
| **Amazon Bedrock** | Chat completions, embeddings, reranking, image generation and editing | [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) |
| **Amazon Transcribe** | Voice input, behind `POST /v1/audio/transcriptions` | `AUDIO_STT_MODEL` (above) |
| **Amazon Polly** | Spoken replies, behind `POST /v1/audio/speech` | `AUDIO_TTS_MODEL` (above) |
| **Amazon Aurora PostgreSQL** | Open WebUI's own database, and the pgvector store its RAG pipeline queries; Serverless v2, two instances across Availability Zones, storage encrypted, TLS enforced (`rds.force_ssl`) | Terraform sample |
| **Amazon ElastiCache (Valkey)** | WebSocket session state and model-list cache, reached over `rediss://` with an auth token; a primary and a cross-AZ replica with automatic failover | Terraform sample |
| **Amazon S3** | Open WebUI file uploads under an `openwebui/` prefix, plus the gateway's temporary multimodal objects | [S3 storage](operations_compliance.md#s3-data-storage) |
| **AWS KMS** | Customer-managed keys encrypting the S3 bucket and the Aurora cluster storage | Terraform sample |
| **AWS Secrets Manager** | Holds the RDS-managed Aurora master password, used only by the apply-time RDS Data API initialization. The tasks' own credentials are Systems Manager Parameter Store SecureStrings created by the ECS module | Terraform sample |
| **Amazon CloudWatch** | Container logs, Container Insights, gateway request logs and EMF usage metrics | [Logging & monitoring](operations_logging_monitoring.md) |
| **AWS IAM** | Separate least-privilege task roles for each service; the gateway's role grants only the model and AI-service actions it invokes | [IAM permissions](operations_iam_permissions.md) |
#### Security Measures in This Flow
- **Authentication** — Open WebUI signs in its own users; every call it then makes to the gateway carries a stdapi.ai [API key](operations_authentication_security.md#api-key-authentication) that Terraform generates and injects into the container environment.
- **Encryption in transit** — HTTPS from the browser to the ALB, whose listener supports TLS 1.2 and 1.3; private-VPC traffic from the ALB to the container; HTTPS with SigV4 from the gateway to each AWS service.
- **Encryption at rest** — SSE-KMS on the S3 bucket, encrypted Aurora storage, and TLS plus an auth token on the Valkey connection.
- **Least privilege** — each ECS task assumes its own role; the gateway's role carries no permission for the Aurora cluster, and Open WebUI's role carries none for Amazon Bedrock.
- **Content policy** — a [Bedrock guardrail](operations_configuration.md#bedrock-guardrails) configured on the gateway applies to each route Open WebUI uses, not only to chat, and stays in force unless the deployment explicitly allows a per-request override.
- **Data handling** — the gateway is stateless and holds request bodies in memory only; CloudWatch receives request metadata, not prompts, unless payload logging is explicitly turned on for debugging.
### :material-cube-outline: What's Included
Deploy Open WebUI + stdapi.ai together with production infrastructure:
**📦 [stdapi-ai/samples/getting_started_openwebui](https://github.com/stdapi-ai/samples/tree/main/getting_started_openwebui)**
**What's included:**
- Open WebUI on ECS Fargate with auto-scaling
- stdapi.ai gateway connected to Amazon Bedrock
- ElastiCache Valkey for caching
- Aurora PostgreSQL with pgvector extension for RAG, with hybrid search and reranking
- SearXNG for web search integration
- Playwright for web scraping
- HTTPS with ALB on your own domain
- All environment variables pre-configured
- Official container images used as published — no local Docker build and no registry credential
**Deploy:**
```bash
git clone https://github.com/stdapi-ai/samples.git
cd samples/getting_started_openwebui/terraform
tofu init
tofu apply
```
### :material-gauge: What It Costs to Run
| Charge | Driver |
| --- | --- |
| stdapi.ai licence | $0.10 per gateway container-hour, metered through AWS Marketplace, with a 14-day free trial on the licence |
| ECS Fargate | Four services — Open WebUI, the gateway, SearXNG, Playwright — each sized and auto-scaled independently |
| Load balancing and networking | One ALB, plus the NAT gateways the private subnets egress through — the S3 gateway endpoint alongside them carries no charge |
| Aurora Serverless v2 | Scales with query load; the sample floors it at 0.5 ACU rather than zero, because auto-pause needs no open connections and the application holds an idle pool |
| ElastiCache Valkey | A standing node cost |
| Model and AI-service usage | Amazon Bedrock, Polly and Transcribe at AWS rates, billed to your account with no markup |
Read a model's price before you send anything to it with [`GET /model_pricing`](api_model_pricing.md). Setting [`COST_TRACKING=true`](operations_cost_management.md#cost-tracking-real-time-aws-pricing) additionally puts a per-request cost on each usage entry — estimated from published AWS prices, not read back from your invoice.
### :material-eye-outline: What to Watch
Both containers log to CloudWatch: Open WebUI writes its audit trail to stdout (`ENABLE_AUDIT_STDOUT=true`), and the gateway writes one structured `request` event per call carrying the request id, path, status code, `execution_time_ms`, the model that served it, and the token counts AWS billed. Turning on [`CLOUDWATCH_METRICS`](operations_logging_monitoring.md#cloudwatch-metrics-emf) republishes those counts as CloudWatch metrics in the `stdapi` namespace, dimensioned by `Model`, so a dashboard can plot chat, embedding, image and speech consumption side by side.
For a chat deployment the useful first question is which feature is consuming the models — the Core Connection, RAG, images or voice all arrive on different paths:
```sql
fields path, execution_time_ms
| filter type = "request"
| stats count(*) as calls, pct(execution_time_ms, 95) as p95_ms by path
| sort calls desc
```
Amazon Bedrock [model invocation logging](operations_compliance.md#amazon-bedrock-invocation-logging) is the AWS-side counterpart — off by default, and the record to enable when you need the prompts and completions themselves rather than metadata.
## :material-arrow-right: Next Steps
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy stdapi.ai to AWS with Terraform
- :material-docker: [**Local Development**](operations_getting_started_local.md) — Run stdapi.ai locally with Docker
- :material-puzzle: [**More Use Cases**](use_cases.md) — Explore other integrations and tools
- :material-api: [**API Overview**](api_overview.md) — Explore supported endpoints
---
## LobeHub Integration
Source: https://stdapi.ai/md/use_cases_lobehub.md
# :material-chat-processing-outline: LobeHub Integration
Connect LobeHub to stdapi.ai as its OpenAI-compatible backend. Chat, vision, image generation, and knowledge-base embeddings all run through Amazon Bedrock with a single connection.
## :material-information-outline: About LobeHub
**🔗 Links:** [Website](https://lobehub.com/) | [GitHub](https://github.com/lobehub/lobehub) | [Documentation](https://lobehub.com/docs/self-hosting/start)
LobeHub (formerly LobeChat) is an open-source, self-hosted AI chat platform with a ChatGPT-like interface, a plugin/agent marketplace, and a built-in knowledge base for retrieval-augmented chat.
**Key Features:**
- **Modern chat UI** - Multi-session, multi-agent chat with Markdown, code, and file rendering
- **Knowledge base** - Upload documents and search them with semantic retrieval
- **Vision and image generation** - Analyze images in chat and generate new ones with the AI Image tool
- **Single provider model** - One "OpenAI" connection covers chat, vision, image generation, and embeddings
- **Real accounts** - Email/password (or SSO) registration, not a shared access code
## :material-help-circle-outline: Why LobeHub + stdapi.ai?
- :material-swap-horizontal: __One Connection, Every Modality__
LobeHub treats "OpenAI" as a single provider. Set one base URL and key, and chat, vision, image generation, and embeddings all reach Amazon Bedrock.
- :material-aws: __Access Amazon Bedrock Models__
Claude, Nova, Stable Diffusion, Cohere Embed, and 100+ models through LobeHub's chat interface and knowledge base.
- :material-application-cog: __Multi-Modal Chat__
Text chat, vision, and in-chat image generation, all backed by Bedrock models through the same endpoint.
- :material-lock: __Enterprise Data Privacy__
All processing stays in your AWS account. Complete infrastructure control with AWS security, compliance, and data sovereignty.
- :material-currency-usd-off: __Pay-Per-Use Pricing__
No LobeHub Cloud subscription. Pay only Amazon Bedrock rates for actual usage—no monthly minimums or per-user fees.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
lobehub[" LobeHub"] --> stdapi[" stdapi.ai"]
stdapi --> bedrock[" Amazon Bedrock"]
```
## :material-connection: Connect Your Own Instance
Point any running LobeHub instance—wherever you host it—at your stdapi.ai gateway. Nothing below requires the AWS sample in [Part 2](#deploy-the-full-stack-on-aws).
### :material-check-circle: Prerequisites
!!! info "What You'll Need"
- ✓ **stdapi.ai deployed** - [See deployment guide](operations_getting_started.md)
- ✓ **Your stdapi.ai URL** - e.g., `https://api.example.com`
- ✓ **Your API key** - From Terraform output or configuration
- ✓ **LobeHub instance** - Running or ready to deploy (see [Part 2](#deploy-the-full-stack-on-aws) for a Terraform-deployed option), in **server DB** mode — LobeHub 2.x has no client-storage mode
### :material-cog: Configuration
LobeHub is configured through environment variables. Unlike Open WebUI, LobeHub has no per-feature connection settings: one "OpenAI" provider covers chat, vision, image generation, and embeddings.
!!! example "Environment Variables"
```bash
ENABLED_OPENAI = "1"
OPENAI_API_KEY = YOUR_STDAPI_KEY
OPENAI_PROXY_URL = https://YOUR_STDAPI_URL/v1
OPENAI_MODEL_LIST = "-all,+anthropic.claude-sonnet-4-5-20250929-v1:0=Claude Sonnet 4.5<200000:vision:fc>,+stability.stable-image-core-v1:1=Stable Image Core<4096:imageOutput>"
```
`OPENAI_MODEL_LIST` both selects which models appear in the UI and tags their capabilities (`vision`, `fc` for tool calling, `imageOutput`), following LobeHub's [model list syntax](https://lobehub.com/docs/self-hosting/advanced/model-list). Any model can be swapped for another Bedrock model available through stdapi.ai, as long as its capability tags match its modality.
Set the default models for chat and background tasks separately:
!!! example "Default Models"
```bash
DEFAULT_AGENT_CONFIG = "model=anthropic.claude-sonnet-4-5-20250929-v1:0;provider=openai"
SYSTEM_AGENT = "default=openai/amazon.nova-micro-v1:0"
DEFAULT_FILES_CONFIG = "embedding_model=openai/cohere.embed-v4:0"
```
- `DEFAULT_AGENT_CONFIG` — the default assistant's chat/vision model
- `SYSTEM_AGENT` — a fast, low-cost model for background tasks (topic naming, translation)
- `DEFAULT_FILES_CONFIG` — the embedding model backing the knowledge base
LobeHub calls `POST /v1/chat/completions` for chat and vision (see [Chat Completions API](api_openai_chat_completions.md)), `POST /v1/embeddings` for the knowledge base (see [Embeddings API](api_openai_embeddings.md)), and `POST /v1/images/generations` for the AI Image tool (see [Images Generations API](api_openai_images_generations.md)).
For the full reference, see LobeHub's [environment variables documentation](https://lobehub.com/docs/self-hosting/environment-variables/basic).
#### :material-volume-high: Voice (TTS/STT)
LobeHub has no server-side environment variable for the default voice provider. In an agent's settings, under **Text-to-Speech**/**Speech-to-Text**, select **OpenAI Audio** — it reuses the same stdapi.ai connection already configured, with no extra key needed.
Two gateway-side details make that pairing more comfortable than it is upstream:
- **Reading a long answer aloud takes one request.** [`/v1/audio/speech`](api_openai_audio_speech.md#long-input) accepts up to 100,000 characters — 24× the upstream limit — and speaks long input as it is synthesized rather than after the whole job finishes, so an essay-length reply needs no client-side splitting. Past 3,000 characters it needs a bucket for the serving region, except on generative voices, which reach 20,000 without one.
- **Transcription can be cheaper.** Naming `amazon.nova-2-sonic-v1:0` as the speech-to-text model transcribes through [Amazon Nova Sonic](api_openai_audio_transcriptions.md#amazon-nova-sonic), the lowest-cost option here; it returns punctuated text with no timestamps, for recordings up to 10 minutes.
#### :material-account-key: SSO and Authentication
LobeHub supports SSO providers (Google, GitHub, Microsoft, AWS Cognito, and others) and restricting registration to specific email domains via `AUTH_ALLOWED_EMAILS` / `AUTH_DISABLE_EMAIL_PASSWORD`. See LobeHub's [authentication environment variables](https://lobehub.com/docs/self-hosting/environment-variables/auth).
## :material-rocket-launch: Deploy the Full Stack on AWS
The Terraform sample below is one worked example of a credible AWS deployment, not the only architecture that works. stdapi.ai's gateway is a normal HTTP service, and LobeHub can run wherever you already operate it—your own ECS, EKS, EC2, another cloud, or a laptop.
### :material-sitemap: Architecture
The diagram below is the topology the [sample](#whats-included) builds: LobeHub sits behind a public load balancer with its own self-hosted Postgres database, while the stdapi.ai gateway has no public endpoint of its own.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true, 'nodeSpacing': 20, 'rankSpacing': 40, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart TB
user["👤 Your users (browser)"]
subgraph public["Your VPC · public subnets"]
alb[" Application Load Balancer fronts LobeHub only"]
end
subgraph private["Your VPC · private app subnets — no inbound route from the internet"]
lobehub2[" LobeHub ECS Fargate"]
stdapi2[" stdapi.ai ECS Fargate"]
postgres["ParadeDB Postgres ECS Fargate · EFS-backed"]
valkey[" ElastiCache Valkey TLS + auth token"]
egress[" NAT gateways · one per AZ + free S3 gateway endpoint"]
end
subgraph regional["AWS service endpoints · your account, the regions you configure"]
bedrock2[" Amazon Bedrock"]
s3[" Amazon S3 SSE-KMS"]
end
user -->|"HTTPS or HTTP · restricted to your IP"| alb
alb -->|"HTTP · private subnet"| lobehub2
lobehub2 -->|"OpenAI API · API key Cloud Map private DNS no public endpoint"| stdapi2
lobehub2 -->|"TLS · verify-full"| postgres
lobehub2 -->|"TLS · auth token"| valkey
lobehub2 -->|"S3 gateway endpoint IAM user access key"| s3
stdapi2 --> egress
egress -->|"HTTPS · SigV4"| bedrock2
stdapi2 -->|"S3 gateway endpoint · task role"| s3
```
Two things stand out. LobeHub is the only service with a public address — the gateway resolves entirely through AWS Cloud Map private DNS, so no third party, and no public listener of its own, sits between your users and the models. And Amazon S3 is the one destination both applications reach directly through the account's S3 gateway endpoint rather than the NAT gateways, each under its own credential: LobeHub's static IAM user key for its uploads bucket, the gateway's task role for its own bucket of temporary multimodal objects.
#### What Each AWS Service Does Here
| AWS service | Role in this integration | Where it is configured |
| --- | --- | --- |
| **Amazon ECS on AWS Fargate** | Runs LobeHub, the stdapi.ai gateway and the self-hosted ParadeDB Postgres task as independent services | Terraform sample |
| **Elastic Load Balancing** | The only public entry point; optionally terminates TLS with an ACM certificate on your domain, otherwise serves plain HTTP; forwards only to LobeHub | Terraform sample (`alb.tf`) |
| **AWS Cloud Map** | Private DNS names LobeHub uses to reach the gateway and its Postgres task — neither is exposed outside the VPC, and the Postgres name is also the one its TLS certificate is issued for | Terraform sample (`service_discovery_dns_name`) |
| **Amazon Bedrock** | Chat, vision, image generation and knowledge-base embeddings | [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) |
| **Amazon S3** | Two separate buckets: LobeHub's file, avatar and knowledge-base uploads, and the gateway's own temporary multimodal objects | [S3 storage](operations_compliance.md#s3-data-storage) |
| **Amazon ElastiCache (Valkey)** | LobeHub's cache, session state and the stream fan-out that lets a reply generated by one task reach a browser attached to another; reached over `rediss://` with an auth token, and deployed as a primary with a cross-AZ replica, automatic failover and daily snapshots | Terraform sample (`valkey.tf`) |
| **Self-hosted PostgreSQL (ParadeDB)** | LobeHub's application database and the `pg_search`/`pgvector` store behind its knowledge base; not an AWS managed service — see [below](#why-this-sample-self-hosts-postgres) | Terraform sample (`postgres.tf`) |
| **AWS KMS** | Customer-managed keys encrypting the S3 buckets, the Valkey replication group, and the EFS volume behind Postgres | Terraform sample |
| **AWS IAM** | Separate least-privilege ECS task roles per service, plus a dedicated IAM user whose static access key for LobeHub's own S3 client is rotated every 90 days, on the next `tofu apply` after that interval elapses | [IAM permissions](operations_iam_permissions.md) |
| **Amazon CloudWatch** | Container logs, Container Insights, gateway request logs and EMF usage metrics | [Logging & monitoring](operations_logging_monitoring.md) |
#### Why This Sample Self-Hosts Postgres { #why-this-sample-self-hosts-postgres }
LobeHub's server DB mode requires ParadeDB's `pg_search` extension, and neither Amazon RDS nor Aurora PostgreSQL can load it — `shared_preload_libraries` on both engines is restricted to an AWS allow-list that does not include it, and the migration that enables the extension is unconditional, so LobeHub refuses to start without it. This sample therefore runs the official `paradedb/paradedb` image on ECS Fargate with its data directory on Regional EFS: one task, replaced stop-before-start on every deployment, because Postgres does not detect two postmasters sharing the same data directory. The sample's README, ["The database: why this sample does not use Aurora"](https://github.com/stdapi-ai/samples/blob/main/getting_started_lobehub/README.md#the-database-why-this-sample-does-not-use-aurora), has the full mechanism, the alternatives considered, and what running Postgres on EFS does and does not cost you.
#### Security Measures in This Flow
- **Authentication** — LobeHub signs in its own users; every call it makes to the gateway carries a stdapi.ai [API key](operations_authentication_security.md#api-key-authentication) that Terraform generates and injects as an ECS `secrets` entry, never a plain environment variable.
- **Encryption in transit** — HTTPS to the ALB when a domain and certificate are configured (plain HTTP otherwise, restricted to the deployer's IP), private-subnet HTTP from the ALB to LobeHub, TLS with an auth token to Valkey, and HTTPS with SigV4 from the gateway to Amazon Bedrock and S3. The Postgres hop is TLS too: Terraform creates a private certificate authority, issues a server certificate for the database's private DNS name — delivered to the container as a KMS-encrypted SSM `SecureString` parameter, never as a file on the shared volume — and LobeHub connects with `sslmode=verify-full` trusting that one authority, which makes the connection authenticated rather than merely encrypted.
- **Encryption at rest** — SSE-KMS on both S3 buckets, an encrypted Valkey replication group, and a KMS-encrypted, transit-encrypted EFS volume behind the self-hosted Postgres.
- **Least privilege** — each ECS task assumes its own role; the ALB's security group admits only the deployer's current IP, not the open internet; LobeHub's S3 access runs through a dedicated IAM user scoped to one bucket and one KMS key, because LobeHub's own S3 client needs a static access key and cannot assume the task role.
- **Content policy** — a [Bedrock guardrail](operations_configuration.md#bedrock-guardrails) configured on the gateway applies to chat, vision and image generation alike, since LobeHub reaches all three through the same connection.
- **Data handling** — the gateway is stateless and holds request bodies in memory only; LobeHub's own content — chat history, uploaded files, knowledge-base embeddings — lives in the Postgres and S3 resources this sample creates, not in a service operated by LobeHub or stdapi.ai.
### :material-cube-outline: What's Included
Deploy LobeHub + stdapi.ai together with production infrastructure:
**📦 [stdapi-ai/samples/getting_started_lobehub](https://github.com/stdapi-ai/samples/tree/main/getting_started_lobehub)**
**What's included:**
- LobeHub on ECS Fargate, server DB mode
- stdapi.ai gateway connected to Amazon Bedrock, exposed as LobeHub's single "OpenAI" provider
- Self-hosted ParadeDB PostgreSQL (`pg_search` + `pgvector`) on EFS — see [above](#why-this-sample-self-hosts-postgres)
- Amazon S3 for file, avatar and knowledge-base uploads, private and SSE-KMS encrypted, served through presigned URLs
- ElastiCache Valkey for cache and sessions
- HTTPS with ALB on your own domain
- All environment variables pre-configured
**Deploy:**
```bash
git clone https://github.com/stdapi-ai/samples.git
cd samples/getting_started_lobehub/terraform
tofu init
tofu apply
```
**Manual steps after apply:**
- **First account** — register through the UI; there is no pre-provisioned admin user, unlike the n8n sample
- **Image generation model** — verify the model appears in the AI Image tool's model picker; this wiring was not exercised against a live deployment while building the sample
### :material-gauge: What It Costs to Run
| Charge | Driver |
| --- | --- |
| stdapi.ai licence | $0.10 per gateway container-hour, metered through AWS Marketplace, with a 14-day free trial on the licence |
| ECS Fargate | Three services — LobeHub, the gateway, and the self-hosted ParadeDB Postgres task, which is pinned to exactly one instance |
| Load balancing and networking | One ALB fronting LobeHub, plus the NAT gateways the private subnets egress through — the S3 gateway endpoint alongside them carries no charge |
| Amazon EFS | Backing storage for the Postgres data directory |
| ElastiCache Valkey | A standing cost for two nodes — a primary and its failover replica (`cache.t4g.micro` each in the sample) |
| Amazon S3 | Storage for LobeHub's uploads and the gateway's temporary multimodal objects |
| Model and AI-service usage | Amazon Bedrock at AWS rates, billed to your account with no markup |
Check a model's price before routing traffic to it with [`GET /model_pricing`](api_model_pricing.md). Turning on [`COST_TRACKING`](operations_cost_management.md#cost-tracking-real-time-aws-pricing) puts a per-request cost on each usage entry — estimated from published AWS prices, not read back from your invoice — and is off by default.
### :material-eye-outline: What to Watch
The gateway writes one structured `request` (or `request_stream` for streamed chat) event per call, carrying the request id, path, status code, `execution_time_ms`, the model that served it, and the AWS-billed token, character or second counts recorded in its nested `usage` list. Turning on [`CLOUDWATCH_METRICS`](operations_logging_monitoring.md#cloudwatch-metrics-emf) republishes those same counts as flat CloudWatch EMF metrics in the `stdapi` namespace, dimensioned by `Model` — with a second `[Model, Currency]` set for the `Cost` metric — so a dashboard can plot token consumption without parsing the JSON logs.
```sql
fields Model, InputTokens, OutputTokens
| filter _aws.CloudWatchMetrics is not null
| stats sum(InputTokens) as input_tokens, sum(OutputTokens) as output_tokens, count(*) as calls by Model
| sort input_tokens + output_tokens desc
```
For the prompts and completions themselves rather than metadata, turn on Amazon Bedrock's own [model invocation logging](operations_compliance.md#amazon-bedrock-invocation-logging), which stays off by default and is configured entirely on the AWS side.
## :material-arrow-right: Next Steps
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy stdapi.ai to AWS with Terraform
- :material-docker: [**Local Development**](operations_getting_started_local.md) — Run stdapi.ai locally with Docker
- :material-puzzle: [**More Use Cases**](use_cases.md) — Explore other integrations and tools
- :material-api: [**API Overview**](api_overview.md) — Explore supported endpoints
---
## n8n Integration
Source: https://stdapi.ai/md/use_cases_n8n.md
# :material-sitemap: n8n Integration
Connect n8n automation workflows to Amazon Bedrock models through stdapi.ai's OpenAI-compatible or Anthropic-compatible interfaces. Existing OpenAI and Anthropic templates from the n8n marketplace need the credential repointed at your stdapi.ai instance—base URL and API key—and, only where a template's model name isn't one your deployment serves, the node's model changed to one that is, chosen from every provider in the catalogue rather than one vendor's list.
## :material-information-outline: About n8n
**🔗 Links:** [Website](https://n8n.io/) | [GitHub](https://github.com/n8n-io/n8n) | [Documentation](https://docs.n8n.io/)
n8n is a powerful workflow automation platform that enables you to connect any app with an API to build intelligent automations. With its intuitive visual interface, you can create complex AI-powered workflows without writing code, connecting Amazon Bedrock models to hundreds of services including Slack, Google Sheets, Salesforce, and more.
**Key Features:**
- ⭐ **190,000+ GitHub stars** - Leading open-source workflow automation platform
- **Hundreds of integrations** - Pre-built nodes for popular services and APIs
- **Visual no-code builder** - Drag-and-drop interface with JavaScript customization
- **Self-hosted or cloud** - Deploy on your infrastructure or use n8n Cloud
- **AI-native platform** - Built-in OpenAI nodes work instantly with Amazon Bedrock via stdapi.ai
- **Template marketplace** - Thousands of pre-built workflows ready to use
## :material-help-circle-outline: Why n8n + stdapi.ai?
- :material-puzzle: __Use Existing OpenAI Templates__
stdapi.ai works with n8n's OpenAI nodes. Thousands of marketplace templates and workflows designed for OpenAI run on Amazon Bedrock—zero modifications needed.
- :material-robot: __Use Existing Anthropic Templates__
stdapi.ai works with n8n's Anthropic nodes. Templates and workflows designed for Anthropic Claude run on Amazon Bedrock—zero modifications needed.
- :material-aws: __Access Amazon Bedrock Models__
Claude, Nova, Llama, DeepSeek, Stable Diffusion, and 100+ models available in n8n workflows. Switch models without changing automation logic.
- :material-graph-outline: __Automate Business Processes__
Connect Amazon Bedrock AI to Slack, Salesforce, Google Workspace, databases, and hundreds of other services. Build intelligent automation with no-code drag-and-drop.
- :material-lock: __Enterprise Data Control__
All AI processing stays in your AWS account. Self-host n8n and stdapi.ai for complete data sovereignty and compliance.
- :material-currency-usd-off: __Pay-Per-Use Pricing__
No OpenAI subscriptions or per-automation fees. Pay only Amazon Bedrock rates for actual AI usage in your workflows.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
n8n[" n8n"] --> stdapi[" stdapi.ai"]
stdapi --> bedrock[" Amazon Bedrock"]
stdapi --> transcribe[" Amazon Transcribe"]
stdapi --> polly[" Amazon Polly"]
```
## :material-connection: Connect Your Own Instance
Point any running n8n instance—self-hosted or n8n Cloud—at your stdapi.ai gateway. Nothing below requires the AWS sample in [Part 2](#deploy-the-full-stack-on-aws).
### :material-check-circle: Prerequisites
!!! info "What You'll Need"
- ✓ **stdapi.ai deployed** - [See deployment guide](operations_getting_started.md)
- ✓ **Your stdapi.ai URL** - e.g., `https://api.example.com`
- ✓ **Your API key** - From Terraform output or configuration
- ✓ **n8n instance** - Self-hosted or [n8n Cloud](https://n8n.io/cloud/)
### :material-cog: Configuration
!!! note "Model modality"
In every section below, the selected model must match the operation's modality — chat nodes need a chat model, embedding nodes an embedding model, and so on.
#### { style="height: 1.2em; vertical-align: text-bottom;" } OpenAI Nodes
##### :material-key: Set Up Your Credentials
The foundation of any n8n integration is configuring your API credentials. This one-time setup unlocks all AI capabilities.
!!! example "Creating Your stdapi.ai Credential"
**In your n8n interface:**
1. Navigate to **Credentials** menu
2. Click **Create Credential**
3. Search and select **"OpenAI"** in the credential list
4. Configure the following fields:
```
API Key: YOUR_STDAPI_KEY
Base URL: https://YOUR_STDAPI_URL/v1
```
!!! tip "What This Does"
By setting a custom Base URL, you redirect all OpenAI API calls to your stdapi.ai instance. n8n will use this credential to authenticate and route requests to Amazon Bedrock models instead of OpenAI's servers.
##### :material-cog-outline: Configure Nodes
For each node, first select the credentials you previously created in the node parameters. Then, select the model you want to use. If you want to use a model that is not listed, you can enter its ID as an expression in the `Model` parameter.
Operation names below match V2 of the n8n OpenAI node (n8n 1.117.0 and later); older versions use slightly different names (e.g. `Message a model` instead of `Generate a Model Response`).
##### :material-chat-outline: Chat Completions
Enables: Text generation and conversational AI in workflows.
!!! example "Supported Node"
**`OpenAI Chat Model`**
- Model can be selected directly in the `Model` parameter
- Sub-node for AI Agent and chain nodes
- The **Use Responses API** option works either way: enabled calls `POST /v1/responses` (see [Responses API](api_openai_responses.md)), disabled calls `POST /v1/chat/completions` (see [Chat Completions API](api_openai_chat_completions.md))
##### :material-message-text: Text Generation
Enables: Text generation using the OpenAI Responses or Chat Completions APIs.
!!! example "Supported Nodes"
**`OpenAI/Generate a Model Response`**
- Model can be selected directly in the `Model` parameter
n8n calls `POST /v1/responses` (see [Responses API](api_openai_responses.md)).
---
**`OpenAI/Generate a Chat Completion`**
- Model can be selected directly in the `Model` parameter
n8n calls `POST /v1/chat/completions` (see [Chat Completions API](api_openai_chat_completions.md)).
##### :material-text-box-outline: Legacy Completions
Enables: raw prompt completion in LangChain-based chains, as an alternative to the chat-based nodes above.
!!! example "Supported Node"
**`OpenAI Model`** (`@n8n/n8n-nodes-langchain.lmOpenAi`)
- Sub-node feeding a Basic LLM Chain or similar LangChain node — distinct from the **`OpenAI Chat Model`** sub-node above
- Model can be selected directly in the `Model` parameter
n8n calls `POST /v1/completions` (see [Completions API](api_openai_completions.md)).
##### :material-shield-check: Text Moderation
Enables: Content safety classification in workflows.
!!! example "Supported Node"
**`OpenAI/Classify Text for Violations`**
- Works out of the box with OpenAI's default moderation model names
- `omni-moderation-latest` maps to your configured Bedrock guardrail (or to Amazon Comprehend toxicity detection when no guardrail is set); `text-moderation-latest` maps to Amazon Comprehend toxicity detection
n8n calls `POST /v1/moderations` (see [Moderations API](api_openai_moderations.md)).
##### :material-database: Embeddings
Enables: Vector embeddings for semantic search and RAG workflows.
!!! example "Supported Node"
**`Embeddings OpenAI`**
- Model can be selected directly in the `Model` parameter
n8n calls `POST /v1/embeddings` (see [Embeddings API](api_openai_embeddings.md)).
##### :material-image-search: Image Analysis
Enables: Image understanding and analysis in workflows.
!!! example "Supported Node"
**`OpenAI/Analyze Image`**
- Model can be selected directly in the `Model` parameter
n8n calls `POST /v1/responses` with image input (see [Responses API](api_openai_responses.md)); the model must support vision.
##### :material-image: Image Generation
Enables: Text-to-image creation in workflows.
!!! example "Supported Node"
**`OpenAI/Generate an Image`**
- Model ID can be entered as expression in the `Model` parameter
n8n calls `POST /v1/images/generations` (see [Images Generations API](api_openai_images_generations.md)).
##### :material-image-edit: Image Editing
Enables: Image transformation and editing in workflows.
!!! example "Supported Node"
**`OpenAI/Edit an Image`**
- Model ID can be entered as expression in the `Model` parameter
n8n calls `POST /v1/images/edits` (see [Images Edits API](api_openai_images_edits.md)).
##### :material-video: Video Generation
Enables: Asynchronous text-to-video generation in workflows.
!!! example "Supported Node"
**`OpenAI/Generate a Video`**
- Model ID can be entered as expression in the `Model` parameter
- Set **Wait Time** high enough for the job to finish—the node polls the job until it completes before returning, so the workflow blocks for the full generation time
n8n calls `POST /v1/videos` (see [Videos API](api_openai_videos.md)), including status polling and content download.
##### :material-volume-high: Audio Generation (TTS)
Enables: Text-to-speech audio generation in workflows.
!!! example "Supported Node"
**`OpenAI/Generate Audio`**
- Model ID can be entered as expression in the `Model` parameter
- **Or use OpenAI model names directly:** `tts-1` and `tts-1-hd` work by default thanks to built-in model aliases
n8n calls `POST /v1/audio/speech` (see [Audio Speech API](api_openai_audio_speech.md)).
A single node call takes up to [100,000 characters](api_openai_audio_speech.md#long-input) — 24× the upstream limit — so a whole article or report becomes one audio file instead of a split-and-concatenate branch. Past 3,000 characters the deployment needs a bucket for the serving region; generative voices reach 20,000 without one.
##### :material-microphone: Audio Transcription (STT)
Enables: Speech-to-text transcription in workflows.
!!! example "Supported Node"
**`OpenAI/Transcribe a Recording`**
- Works out of the box with OpenAI's `whisper-1` model name
- The model alias automatically maps to `amazon.transcribe`
n8n calls `POST /v1/audio/transcriptions` (see [Audio Transcriptions API](api_openai_audio_transcriptions.md)).
##### :material-translate: Audio Translation
Enables: Translating speech in any supported language into English text.
!!! example "Supported Node"
**`OpenAI/Translate a Recording`**
- Works out of the box with OpenAI's `whisper-1` model name
- The model alias automatically maps to `amazon.transcribe`; Bedrock speech-to-text models (e.g. Mistral Voxtral) also work
n8n calls `POST /v1/audio/translations` (see [Audio Translations API](api_openai_audio_translations.md)).
##### :material-file-upload: Files
Enables: Upload files once and reference them across multiple chat completion requests without resending the raw bytes each time.
n8n calls the `/v1/files` endpoints (see [Files API](api_openai_files.md)). Set **Resource** to **"Files"** in the OpenAI node for all operations below.
!!! example "Upload a file — `OpenAI/Upload a File`"
Uploads a file to S3 and returns a `file_id` for use in subsequent requests.
**Node parameters:**
- **Resource:** Files
- **Operation:** Upload a File
- **Input Data Field Name:** name of the binary field containing the file
- **Purpose:** intended purpose (e.g. `assistants`, `user_data`)
**Typical workflow pattern:**
1. Receive or fetch a file (PDF, image, etc.) in an earlier node
2. Pass the binary data to this node
3. Store the returned `file_id` in a variable or database
4. Pass `file_id` in `OpenAI Chat Model` messages via the `type: "file"` content part for repeated analysis without re-uploading
!!! example "Delete a file — `OpenAI/Delete a File`"
Permanently deletes a file from S3 by its `file_id`.
**Node parameters:**
- **Resource:** Files
- **Operation:** Delete a File
- **File ID:** the `file_id` of the file to delete
!!! example "List files — `OpenAI/List Files`"
Returns a paginated list of uploaded files, optionally filtered by purpose.
**Node parameters:**
- **Resource:** Files
- **Operation:** List Files
- **Purpose:** _(optional)_ filter results to a specific purpose
- **Return All / Limit:** control pagination; enable **Return All** or set **Limit** for the first page
Files are returned in descending order (newest first) by default.
##### :material-package-variant-closed: Bulk Runs (Batch API)
Enables: running thousands of chat completion or embedding requests asynchronously, at the Amazon Bedrock batch price rather than the synchronous one.
n8n ships no node for the [Batch API](api_openai_batches.md) — the OpenAI node's resources stop at text, images, audio and files — so drive it with **HTTP Request** nodes, as the Cohere routes are reached [below](#known-limitations):
1. Build the request set as JSONL, one line per item with its own `custom_id`, and upload it with `purpose=batch` (`OpenAI/Upload a File` when its **Purpose** list offers it, an HTTP Request against `POST /v1/files` otherwise)
2. `POST /v1/batches` with that file id, then poll `GET /v1/batches/{id}` behind a **Wait** node until it reports `completed`
3. Download the output file and match each line back to its `custom_id` — results may arrive in any order
Worth the extra nodes when a workflow classifies, enriches or summarizes a large backlog overnight and nothing is waiting on the answer. Interactive workflows stay on the synchronous nodes above.
---
#### { style="height: 1.2em; vertical-align: text-bottom;" } Anthropic Nodes
##### :material-key: Set Up Your Credentials
!!! example "Creating Your stdapi.ai Anthropic Credential"
**In your n8n interface:**
1. Navigate to **Credentials** menu
2. Click **Create Credential**
3. Search and select **"Anthropic"** in the credential list
4. Configure the following fields:
```
API Key: YOUR_STDAPI_KEY
Base URL: https://YOUR_STDAPI_URL/anthropic
```
!!! tip "Anthropic Base URL"
By default, all Anthropic-compatible routes are prefixed with `/anthropic`, so the Base URL must end with `/anthropic`. You can customize this prefix using the `ANTHROPIC_ROUTES_PREFIX` configuration variable documented in [Operations Configuration](operations_configuration.md#anthropic-routes-prefix).
##### :material-cog-outline: Configure Nodes
For each node, first select the credentials you previously created in the node parameters. Then, select the model you want to use. The model can be selected directly in the `Model` parameter for all supported nodes.
##### :material-chat-outline: Chat Completions
Enables: Text generation and conversational AI in workflows.
!!! example "Supported Nodes"
**`Anthropic Chat Model`**
- Model can be selected directly in the `Model` parameter
- Sub-node for AI Agent and chain nodes
---
**`Anthropic/Message a Model`**
- Model can be selected directly in the `Model` parameter
n8n calls `POST /anthropic/v1/messages` (see [Anthropic Messages API](api_anthropic_messages.md)).
##### :material-image-search: Image Analysis
Enables: Image understanding and analysis in workflows.
!!! example "Supported Node"
**`Anthropic/Analyze Image`**
- Model can be selected directly in the `Model` parameter
n8n calls `POST /anthropic/v1/messages` with image content (see [Anthropic Messages API](api_anthropic_messages.md)); the model must support vision.
##### :material-file-document: Document Analysis
Enables: Document understanding and extraction in workflows.
!!! example "Supported Node"
**`Anthropic/Analyze Document`**
- Model can be selected directly in the `Model` parameter
n8n calls `POST /anthropic/v1/messages` with document content (see [Anthropic Messages API](api_anthropic_messages.md)); the model must support document processing.
##### :material-file-upload: Files
Enables: Upload files once and reference them across multiple Messages requests as document or image sources.
n8n calls the `/anthropic/v1/files` endpoints (see [Anthropic Files API](api_anthropic_files.md)). Set **Resource** to **"Files"** in the Anthropic node for all operations below.
!!! example "Upload a file — `Anthropic/Upload File`"
Uploads a file to S3 and returns a `file_id` for use in subsequent Messages requests.
**Node parameters:**
- **Resource:** Files
- **Operation:** Upload File
- **Input Data Field Name:** name of the binary field containing the file
**Typical workflow pattern:**
1. Receive or fetch a file (PDF, image, etc.) in an earlier node
2. Pass the binary data to this node
3. Store the returned `file_id` in a variable or database
4. Pass `file_id` as a `source: {type: "file"}` in `Anthropic/Message a Model` document or image blocks
!!! example "Get file metadata — `Anthropic/Get File Metadata`"
Retrieves metadata (filename, MIME type, size, creation date) for a file by its `file_id`.
**Node parameters:**
- **Resource:** Files
- **Operation:** Get File Metadata
- **File ID:** the `file_id` of the file to retrieve
!!! example "List files — `Anthropic/List Files`"
Returns a paginated list of uploaded files.
**Node parameters:**
- **Resource:** Files
- **Operation:** List Files
- **Return All / Limit:** control pagination; enable **Return All** or set **Limit** for the first page
Files are returned most recently created first. Use `after_id` / `before_id` cursors for bidirectional pagination.
!!! example "Delete a file — `Anthropic/Delete File`"
Permanently deletes a file from S3 by its `file_id`.
**Node parameters:**
- **Resource:** Files
- **Operation:** Delete File
- **File ID:** the `file_id` of the file to delete
##### :material-lightbulb-outline: Prompt Resource
The Anthropic node's **Prompt** resource (`Generate Prompt`, `Improve Prompt`, `Templatize Prompt`) calls Anthropic's experimental prompt tools endpoints, which are not part of the Amazon Bedrock API surface and are not available through stdapi.ai. Use a `Message a Model` node with prompt-engineering instructions instead.
### :material-alert-outline: Known Issues { #known-limitations }
n8n's Cohere sub-nodes—**Cohere Reranker** (used by vector store nodes for hybrid search) and **Embeddings Cohere**—cannot be pointed at stdapi.ai. Their shared `cohereApi` credential exposes only an API key: its base URL is a hidden field pinned to Cohere's own endpoint, and both nodes build their client from the API key and the model alone, so that URL never reaches the request anyway.
- **Reranking:** use an **HTTP Request** node against `POST /cohere/v2/rerank` (see [Cohere Rerank API](api_cohere_rerank.md)), or a framework with a configurable reranker base URL—see [RAG Pipelines](use_cases_rag.md).
- **Embeddings:** use the **Embeddings OpenAI** node described [above](#embeddings), which reaches the same Amazon Bedrock models through `/v1/embeddings`; an **HTTP Request** node against `POST /cohere/v2/embed` (see [Cohere Embed API](api_cohere_embed.md)) works too, but its output cannot be fed to a vector store sub-node.
## :material-rocket-launch: Deploy the Full Stack on AWS
The Terraform sample below is one worked example of a credible AWS deployment, not the only architecture that works. stdapi.ai's gateway is a normal HTTP service, and n8n can run wherever you already operate it—your own ECS, EKS, EC2, another cloud, or a laptop. The sample's README documents where you would reasonably diverge from what it builds (networking, worker capacity, database, queue broker).
### :material-sitemap: Architecture
The diagram below is the topology the [sample](#whats-included) builds: n8n and the stdapi.ai gateway both on ECS Fargate in one VPC you own, with n8n's own workflow state in Aurora PostgreSQL alongside them.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true, 'nodeSpacing': 20, 'rankSpacing': 40, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart TB
user["👤 Your users (browser)"]
subgraph public["Your VPC · public subnets"]
alb[" Application Load Balancer HTTPS · ACM certificate"]
end
subgraph private["Your VPC · private app subnets — no inbound route from the internet"]
n8n[" n8n ECS Fargate"]
stdapi[" stdapi.ai ECS Fargate"]
aurora[" Aurora PostgreSQL n8n workflows · credentials · executions"]
egress[" NAT gateways · one per AZ multi-region Bedrock access"]
end
subgraph regional["AWS service endpoints · your account, the regions you configure"]
bedrock[" Amazon Bedrock"]
transcribe[" Amazon Transcribe"]
polly[" Amazon Polly"]
comprehend[" Amazon Comprehend"]
s3[" Amazon S3 SSE-KMS"]
cw[" Amazon CloudWatch logs · metrics"]
bedrock ~~~ polly ~~~ s3
transcribe ~~~ comprehend ~~~ cw
end
user -->|"HTTPS · TLS 1.2+"| alb
alb -->|"HTTP · private subnet"| n8n
n8n -->|"OpenAI + Anthropic dialects · API key HTTP over Cloud Map private DNS no public endpoint"| stdapi
n8n -->|"TLS, no cert verification security-group restricted"| aurora
stdapi --> egress
egress -->|"HTTPS · SigV4"| bedrock
egress -->|"HTTPS · SigV4"| transcribe
egress -->|"HTTPS · SigV4"| polly
egress -->|"HTTPS · SigV4"| comprehend
egress -->|"HTTPS · SigV4"| s3
egress --> cw
```
Two properties are worth reading off the picture. n8n is the only service with a public address — the ALB forwards nothing but n8n traffic, and the stdapi.ai gateway has no listener of its own, reachable only through Cloud Map private DNS inside the VPC. Customer data then splits in two: n8n's own workflow definitions, credentials and execution history live in Aurora, inside the account boundary, while whatever a workflow sends to a model passes through the gateway straight to Amazon Bedrock and the other AWS AI services behind it — no third party sits between your workflows and your models.
#### What Each AWS Service Does Here
| AWS service | Role in this integration | Where it is configured |
| --- | --- | --- |
| **Amazon ECS on AWS Fargate** | Runs n8n and the stdapi.ai gateway as separate services, plus a one-shot "import" task that seeds the credential and sample workflows | Terraform sample (`n8n.tf`, `main.tf`) |
| **Elastic Load Balancing** | The single public entry point; terminates TLS with an ACM certificate and forwards only to n8n | Terraform sample (`alb.tf`) |
| **AWS Cloud Map** | Private DNS name that lets n8n reach the gateway without exposing it | Terraform sample (`service_discovery_dns_name`) |
| **Amazon Bedrock** | Chat, text and image generation, embeddings, video generation | [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) |
| **Amazon Transcribe** | Speech-to-text, behind `POST /v1/audio/transcriptions` and `/v1/audio/translations` | [Audio Transcription (STT)](#audio-transcription-stt) |
| **Amazon Polly** | Spoken replies, behind `POST /v1/audio/speech` | [Audio Generation (TTS)](#audio-generation-tts) |
| **Amazon Comprehend** | Toxicity fallback for `POST /v1/moderations` when no guardrail is configured | [Comprehend Moderation](operations_iam_permissions.md#comprehend-moderation) |
| **Amazon Aurora PostgreSQL** | n8n's own database — workflow definitions, credentials and execution history; Serverless v2, storage encrypted | Terraform sample (`postgres.tf`) |
| **Amazon S3** | The gateway's Files API uploads, long text-to-speech input, and generated image/video output | [S3 storage](operations_compliance.md#s3-data-storage) |
| **AWS KMS** | Two customer-managed keys: one for the gateway's S3 bucket and CloudWatch Logs, another for Aurora storage and the Postgres secret | Terraform sample |
| **AWS Secrets Manager** | Holds the Aurora master password, read only at deploy time — via the RDS Data API — to provision n8n's database role | Terraform sample (`postgres.tf`) |
| **Amazon CloudWatch** | Container logs, gateway request logs and, optionally, EMF usage metrics | [Logging & monitoring](operations_logging_monitoring.md) |
| **AWS IAM** | Separate least-privilege task roles for n8n and the gateway; the gateway's role grants only the AI-service and S3 actions it calls | [IAM permissions](operations_iam_permissions.md) |
#### Security Measures in This Flow
- **Authentication** — n8n reaches the gateway with a stdapi.ai [API key](operations_authentication_security.md#api-key-authentication) that Terraform generates (`api_key_create = true`) and seeds directly into n8n's own credential store at first start.
- **Encryption in transit** — HTTPS with TLS 1.2+ from the browser to the ALB; HTTP from n8n to the gateway, confined to the private subnet and reachable only through Cloud Map private DNS; TLS without certificate verification from n8n to Aurora, a connection that never leaves the VPC and is already restricted by security group; HTTPS with SigV4 from the gateway to each AWS service.
- **Encryption at rest** — SSE-KMS on the gateway's S3 bucket and CloudWatch Logs; separately, encrypted Aurora storage and the Postgres secret under the VPC module's own KMS key.
- **Least privilege** — each ECS task assumes its own role; the gateway's role carries no permission for Aurora or its Secrets Manager secret, and n8n's role carries none for Amazon Bedrock, Transcribe, Polly or Comprehend.
- **Content policy** — a [Bedrock guardrail](operations_configuration.md#bedrock-guardrails) configured on the gateway applies to every route n8n uses, including the mapping [Text Moderation](#text-moderation) already documents for `omni-moderation-latest`.
- **Data handling** — the gateway is stateless and holds request bodies in memory only, so nothing a workflow sends to a model is persisted outside Amazon Bedrock's own call; n8n's own workflow data — credentials, executions — stays in Aurora inside the account boundary.
### :material-cube-outline: What's Included
Deploy n8n + stdapi.ai together, with a credential and thirteen sample workflows already imported:
**📦 [stdapi-ai/samples/getting_started_n8n](https://github.com/stdapi-ai/samples/tree/main/getting_started_n8n)**
**What's included:**
- n8n on ECS Fargate, backed by Aurora PostgreSQL
- stdapi.ai gateway connected to Amazon Bedrock, preconfigured as both an OpenAI and an Anthropic n8n credential
- Thirteen sample workflows — one per stdapi.ai route family — imported automatically on first start
- Owner account pre-provisioned non-interactively, no signup screen to click through
- HTTPS-only ALB on your own domain (required — n8n's session cookie needs it)
- No local image build — the official `n8nio/n8n` image is pulled directly from Docker Hub
**Deploy:**
```bash
git clone https://github.com/stdapi-ai/samples.git
cd samples/getting_started_n8n/terraform
tofu init
tofu apply
```
### :material-gauge: What It Costs to Run
| Charge | Driver |
| --- | --- |
| stdapi.ai licence | $0.10 per gateway container-hour, metered through AWS Marketplace, with a 14-day free trial on the licence |
| ECS Fargate | Two steady services — n8n and the gateway — each sized independently, plus the one-shot import task that seeds credentials and workflows |
| Load balancing and networking | One ALB, plus the NAT gateways the private subnets egress through for multi-region Bedrock access |
| Aurora Serverless v2 | Scales with query load; the sample sets a minimum capacity of zero ACUs |
| Model and AI-service usage | Amazon Bedrock, Transcribe, Polly and Comprehend at AWS rates, billed to your account with no markup |
Read a model's price before a workflow sends anything to it with [`GET /model_pricing`](api_model_pricing.md). Setting [`COST_TRACKING=true`](operations_cost_management.md#cost-tracking-real-time-aws-pricing) additionally puts a per-request cost on each usage entry — estimated from published AWS prices, not read back from your invoice.
### :material-eye-outline: What to Watch
The gateway writes one structured `request` event per call, carrying the request id, path, status code, `execution_time_ms`, the model that served it (`model_id`), and the token, character and second counts AWS billed. A workflow that fires on a schedule has no one watching its output in real time, so whether its calls are succeeding matters more than for an interactive chat session:
```sql
fields model_id, path, status_code
| filter type = "request" and status_code >= 400
| stats count(*) as failures by model_id, path, status_code
| sort failures desc
```
Turning on [`CLOUDWATCH_METRICS`](operations_logging_monitoring.md#cloudwatch-metrics-emf) republishes the same counts as CloudWatch metrics in the `stdapi` namespace, dimensioned by `Model`, so an alarm can fire on a rising failure count without a scheduled query.
## :material-arrow-right: Next Steps
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy stdapi.ai to AWS with Terraform
- :material-docker: [**Local Development**](operations_getting_started_local.md) — Run stdapi.ai locally with Docker
- :material-puzzle: [**More Use Cases**](use_cases.md) — Explore other integrations and tools
- :material-api: [**API Overview**](api_overview.md) — Explore supported endpoints
---
## AI Coding Assistants
Source: https://stdapi.ai/md/use_cases_coding_assistants.md
# :material-code-braces: AI Coding Assistants Integration
Connect your favorite AI coding assistants to Amazon Bedrock models through stdapi.ai—three client-side changes: the base URL, the API key, and — where the name differs from what the assistant already sends — the model name, now picked from every provider in the catalogue rather than one vendor's list. Get intelligent code completions, chat assistance, and codebase understanding with AWS models like Claude, Kimi thinking, and Qwen3 Coder Next.
## :material-information-outline: About AI Coding Assistants
AI coding assistants are IDE extensions and terminal tools that leverage large language models to enhance developer productivity. These tools provide real-time code completions, intelligent suggestions, natural language code generation, and interactive chat capabilities directly within your coding environment—acting as AI pair programmers that understand your codebase context.
**What AI coding assistants can do:**
- **Real-time completions** - Autocomplete code as you type with context awareness
- **Interactive chat** - Ask questions about your codebase, get explanations
- **Code generation** - Natural language to code conversion
- **Refactoring** - Intelligent code improvements and optimization suggestions
- **Documentation** - Auto-generate comments, docstrings, and READMEs
- **Testing** - Create unit tests, debug issues, suggest fixes
- **Git integration** - Generate commit messages, review diffs
- **Multi-language** - Support for Python, JavaScript, TypeScript, Go, Rust, Java, and more
## :material-help-circle-outline: Why AI Coding Assistants + stdapi.ai?
- :material-puzzle: __Works with Your IDE__
Almost any coding assistant that supports OpenAI or Anthropic compatible APIs works with stdapi.ai. Claude Code, Cline, OpenCode, Pi Agent, Zed, OpenAI Codex CLI—all compatible with Amazon Bedrock models.
- :material-brain: __Best-in-Class Coding Models__
Claude for reasoning and architecture, Kimi thinking for complex problem-solving, Qwen3 Coder Next for specialized coding tasks. Choose the right model for each task.
- :material-lock: __No Third-Party AI Cloud__
Your code goes from your IDE to your own deployment to Amazon Bedrock — no vendor endpoint in between, and Bedrock does not use prompts to train models. Suited to proprietary codebases and compliance-sensitive projects.
- :material-server-network: __Flexible Deployment Options__
Run stdapi.ai in AWS for production or locally with Docker for development. Test locally, deploy to cloud—same API, same experience.
- :material-currency-usd-off: __No Per-Developer Licence__
The gateway is licensed per container-hour, not per developer, and model usage is billed by Amazon Bedrock at its published rates with no markup, on your own AWS account. Enable per-user attribution and the invoice shows which developer spent what. A flat-rate subscription can still cost less for one developer coding all day; this is for the calls a subscription does not cover, and for seeing who made them.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
ide[" IDE + AI Assistant"] --> stdapi[" stdapi.ai"]
stdapi --> bedrock[" Amazon Bedrock"]
```
## :material-connection: Connect Your Own Instance
Point any coding assistant—CLI, IDE plugin, or terminal tool—at your stdapi.ai gateway, wherever it runs. Nothing below requires the AWS sample in [Part 2](#deploy-the-gateway-on-aws).
### :material-check-circle: Prerequisites
!!! info "What You'll Need"
- ✓ **stdapi.ai deployed** - [See deployment guide](operations_getting_started.md) or [run locally with Docker](operations_getting_started_local.md); see [Part 2](#deploy-the-gateway-on-aws) for a Terraform-deployed gateway
- ✓ **Your stdapi.ai URL** - e.g., `https://api.example.com` or `http://localhost:8000` for local
- ✓ **Your API key** - From Terraform output or configuration (optional for local development)
- ✓ **IDE with AI assistant** - VS Code, JetBrains IDEs, Zed, or your preferred editor with an AI coding extension
### { style="height: 1.2em; vertical-align: text-bottom;" } OpenAI-Compatible Coding Assistants
**Popular Tools:** [Cline](https://github.com/cline/cline) | [OpenCode](https://opencode.ai/) | [Pi Agent](https://github.com/earendil-works/pi) | [OpenAI Codex CLI](https://developers.openai.com/codex) | [Qwen Code](https://github.com/QwenLM/qwen-code) | [Zed](https://zed.dev/) | [JetBrains AI Assistant](https://www.jetbrains.com/ai/)
Most IDE coding assistants use the OpenAI-compatible API. Configure them by pointing to stdapi.ai's `/v1` endpoint.
#### :material-cog: Configuration
Most AI coding assistants follow a similar configuration pattern. The exact menu location and field names may vary, but the core settings remain consistent.
!!! example "Generic Configuration Steps"
**In your coding assistant settings:**
1. Navigate to **Settings** or **Preferences**
2. Find the **AI Provider** or **Model Provider** section
3. Select **"OpenAI Compatible"** or **"Custom OpenAI"** as the provider type
4. Configure the connection:
```
API Base URL: https://YOUR_STDAPI_URL/v1
(or sometimes just: https://YOUR_STDAPI_URL)
API Key: YOUR_STDAPI_KEY
Model: anthropic.claude-fable-5
(or select from detected models if available)
```
!!! example "OpenAI Codex CLI"
Codex CLI only speaks the [Responses API](api_openai_responses.md) wire format, which stdapi.ai serves at `/v1/responses`. Declare a custom provider in `~/.codex/config.toml` with `wire_api = "responses"`:
```toml
[model_providers.stdapi]
name = "stdapi.ai"
base_url = "https://YOUR_STDAPI_URL/v1"
env_key = "STDAPI_API_KEY"
wire_api = "responses"
[profiles.stdapi]
model_provider = "stdapi"
model = "anthropic.claude-fable-5"
# Codex asks for web search on every request unless this is set.
web_search = "disabled"
```
Codex declares the built-in `web_search` tool by default, and it is served only where the model can actually search:
- **Amazon Nova 2 and Nova Premier, US regions** — mapped to Amazon Nova's grounding tool, which AWS bills per request on top of tokens (see [Cost Management](operations_cost_management.md)). Leave `web_search` enabled to use it.
- **OpenAI GPT-5.x, US regions** — served by the [built-in web search](api_openai_responses.md#openai-gpt-web-search), billed per query on top of tokens, **on the Amazon Bedrock Mantle endpoint only**. Models offered on both endpoints resolve to their bedrock-runtime twin by default, which refuses the tool with a `400` naming what to change; route them to Mantle to use it.
- **EU inference profiles** — the grounding tool is not offered there, so the request is rejected.
- **Any other model** — no hosted search exists behind it, and the tool reaches the model as an ordinary function the gateway cannot execute, so answers come back ungrounded.
Set `web_search = "disabled"` unless your model and region are in the first group. Codex's own shell, file and patch tools are unaffected either way.
A model that is not in Codex's own catalog logs `Model metadata for '' not found. Defaulting to fallback metadata`. That is expected for every Bedrock model ID and does not affect the run.
!!! example "Qwen Code"
[Qwen Code](https://github.com/QwenLM/qwen-code) authenticates against any OpenAI-compatible endpoint through three environment variables:
```bash
export OPENAI_API_KEY=YOUR_STDAPI_KEY
export OPENAI_BASE_URL=https://YOUR_STDAPI_URL/v1
export OPENAI_MODEL=anthropic.claude-fable-5
```
Qwen Code refuses to run non-interactively unless an auth type is selected explicitly. Add it to `~/.qwen/settings.json`:
```json
{
"security": {
"auth": {
"selectedType": "openai"
}
}
}
```
(or pass `--auth-type openai` on the command line). With those three variables and the auth type set, Qwen Code calls `POST /v1/chat/completions` (see [Chat Completions API](api_openai_chat_completions.md)) like any other OpenAI-compatible client.
**Reasoning effort:** set `model.reasoningEffort` in the same settings file (for example `"low"` or `"high"`) to control how hard a reasoning-capable model thinks. Qwen Code is also one of the few coding assistants that keeps a reasoning model's thinking text across turns of the same session and replays it back, rather than discarding it once displayed.
!!! example "pi"
[pi](https://github.com/earendil-works/pi) registers custom providers declaratively in `~/.pi/agent/models.json`. Only `baseUrl`, `api`, `apiKey` and one `id` per model are required:
```json
{
"providers": {
"stdapi": {
"baseUrl": "https://YOUR_STDAPI_URL/v1",
"api": "openai-completions",
"apiKey": "YOUR_STDAPI_API_KEY",
"models": [
{ "id": "anthropic.claude-fable-5" }
]
}
}
}
```
Then select the model, qualified by the provider name:
```bash
pi --model stdapi/anthropic.claude-fable-5
```
`api` selects the wire format, and `baseUrl` has to match the route serving it:
| `api` | `baseUrl` | API |
|---|---|---|
| `openai-completions` | `https://YOUR_STDAPI_URL/v1` | [Chat Completions](api_openai_chat_completions.md) |
| `openai-responses` | `https://YOUR_STDAPI_URL/v1` | [Responses](api_openai_responses.md) |
| `anthropic-messages` | `https://YOUR_STDAPI_URL/anthropic` | [Anthropic Messages](api_anthropic_messages.md) |
Set `api` on the provider to apply it to every model under it, or on an individual model to override it. Declare several providers side by side in the same file to reach more than one route.
List one entry in `models` per model you want to select. Each entry also accepts pi's own client-side accounting — `contextWindow` and `maxTokens` — and setting them generously lets the gateway report the model's real limit instead of pi truncating the prompt first.
!!! tip "Model Selection for Coding"
**Recommended models for different tasks:**
- **Advanced reasoning & architecture**: Anthropic Claude Opus or Fable
- **Complex problem-solving**: Kimi thinking models
- **Specialized coding tasks**: Qwen3 Coder, Mistral Devstral
- **Fast completions**: Amazon Nova Micro or Nova Lite
**Configuration tips:**
- **Auto-detect**: Some assistants query `/v1/models` and show a dropdown
- **Manual entry**: Use full Bedrock model ID (e.g., `anthropic.claude-fable-5`)
- **Multi-model setup**: Use fast, cheap models for secondary tasks (autocomplete, summaries) and stronger models for complex generation
#### :material-chat-outline: Chat Completions
All coding assistants use chat completions for interactive conversations, code generation, and explanations.
!!! example "How It Works"
Your coding assistant calls `POST /v1/chat/completions` (see [Chat Completions API](api_openai_chat_completions.md)) to:
- Answer questions about your code
- Generate new code from natural language
- Explain complex functions or algorithms
- Suggest refactoring and improvements
- Debug issues and propose fixes
The model must be a text/chat-capable model from the correct family for your Bedrock region.
#### :material-tools: Tool Calling Support
stdapi.ai fully supports tool calling (function calling) through the chat completions API, which is essential for autonomous and efficient coding agents.
!!! success "Advanced Agent Capabilities"
**Tool calling enables your coding assistant to:**
- Execute terminal commands and see results
- Read and write files in your codebase
- Search through code and documentation
- Run tests and analyze output
- Interact with external APIs and services
Most modern autonomous agents like Cline or OpenCode rely heavily on tool calling to perform complex, multi-step coding tasks. stdapi.ai's tool calling support (see [Chat Completions API - Tool Calling](api_openai_chat_completions.md#feature-compatibility)) ensures these agents can work at their full potential with Amazon Bedrock models.
#### :material-lightning-bolt: Code Completions
Some coding assistants support dedicated code completion endpoints for real-time suggestions as you type.
!!! example "Completion Support"
Advanced assistants may call `POST /v1/completions` for:
- Inline code suggestions
- Auto-completion while typing
- Context-aware code snippets
Not all models or assistants support this mode. Chat-based assistants handle completions through the chat API instead.
### { style="height: 1.2em; vertical-align: text-bottom;" } Anthropic-Compatible Coding Assistants
**Popular Tools:** [Claude Code](https://code.claude.com/docs/en/overview) | [OpenCode](https://opencode.ai/) | [Zed](https://zed.dev/) | [Factory Droid](https://factory.ai/)
Tools that use the Anthropic messages API natively can be connected to stdapi.ai's `/anthropic` endpoint, enabling them to use Claude models via Amazon Bedrock.
#### Claude Code
Claude Code is Anthropic's agentic coding tool that runs in the terminal.
##### :material-cog: Configuration
Create or edit `~/.claude/settings.json`:
```json
{
"env": {
"ANTHROPIC_AUTH_TOKEN": "YOUR_API_KEY",
"ANTHROPIC_BASE_URL": "https://YOUR_STDAPI_URL/anthropic",
"ANTHROPIC_DEFAULT_FABLE_MODEL": "anthropic.claude-fable-5",
"ANTHROPIC_DEFAULT_OPUS_MODEL": "anthropic.claude-opus-5",
"ANTHROPIC_DEFAULT_SONNET_MODEL": "anthropic.claude-sonnet-5",
"ANTHROPIC_DEFAULT_HAIKU_MODEL": "anthropic.claude-haiku-4-5-20251001-v1:0"
}
}
```
- Replace `YOUR_STDAPI_URL` with your stdapi.ai deployment URL (e.g., `https://api.example.com` or `http://localhost:8000` for local)
- Replace `YOUR_API_KEY` with your stdapi.ai API key
- The `/anthropic` path prefix is configured via the [`ANTHROPIC_ROUTES_PREFIX`](operations_configuration.md#anthropic-routes-prefix) setting (default: `/anthropic`)
- The `ANTHROPIC_DEFAULT_*_MODEL` variables pin each model tier to a specific Bedrock model ID, so Claude Code stops resolving the `fable`/`opus`/`sonnet`/`haiku` aliases itself — an alias moves to a new model whenever Anthropic ships one, while a pinned model ID only changes when you edit it. Haiku's current Bedrock ID carries a dated snapshot (`claude-haiku-4-5-20251001-v1:0`) for the most granular pin; Sonnet, Opus, and Fable 5 don't yet have a separate dated ID in the Bedrock catalog, so pinning to their generation ID (e.g. `anthropic.claude-sonnet-5`) is the most specific option available today. stdapi.ai also accepts the short alias names (e.g. `claude-sonnet-5`) as a convenience.
!!! note "No API key authentication? `ANTHROPIC_AUTH_TOKEN` is still required"
Claude Code refuses to start without a non-empty `ANTHROPIC_AUTH_TOKEN`, even if your stdapi.ai deployment has no API-key authentication configured. In that case, set it to any non-empty placeholder, e.g. `"ANTHROPIC_AUTH_TOKEN": "1"`.
!!! tip "Beta Flag Compatibility"
stdapi.ai automatically filters unsupported `anthropic_beta` flags, so Claude Code works without needing `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1`. Bedrock-supported flags (like `Interleaved-thinking-2025-05-14` and `token-efficient-tools-2025-02-19`) are preserved while unsupported ones are silently removed. See [`ANTHROPIC_BETA_FILTER`](operations_configuration.md#anthropic-beta-filter) and [`ANTHROPIC_BETA_ALLOWLIST`](operations_configuration.md#anthropic-beta-allowlist) for details.
##### :material-brain: Effort-Based Reasoning
Claude Code supports [effort levels](https://code.claude.com/docs/en/model-config#adjust-effort-level) that control how much reasoning the model applies — lower effort is faster and cheaper; higher effort provides deeper thinking for complex tasks.
**Supported models via stdapi.ai:**
| Model | Effort levels | Notes |
|-------------------------------|-----------------------------|---------------------------------------------|
| Claude Sonnet 4.6 / Opus 4.6+ | `low` `medium` `high` `max` | Full adaptive reasoning; `max` is Opus-only |
| Amazon Nova 2 | `low` `medium` `high` | Maps to `maxReasoningEffort` in Bedrock |
| DeepSeek V3 | `low` `medium` `high` | Passed as a string literal to Bedrock |
**Setting effort level:**
```bash
# Per session at launch
claude --model sonnet --effort high
# Persist across sessions (env var takes precedence over all other settings)
export CLAUDE_CODE_EFFORT_LEVEL=high
# Or add to claude.json
```
```json
{
"effortLevel": "medium"
}
```
During a session, use `/effort low`, `/effort medium`, `/effort high`, or `/effort max` to change levels on the fly.
##### :material-key: Declaring Model Capabilities
When you pin a non-Claude Bedrock model ID, Claude Code may not recognize it and will silently disable effort and thinking features. Use `ANTHROPIC_DEFAULT_*_MODEL_SUPPORTED_CAPABILITIES` to declare what the model actually supports:
| Capability value | Enables |
|------------------------|------------------------------------------|
| `effort` | Effort levels and the `/effort` command |
| `max_effort` | The `max` effort level (Opus 4.6+ only) |
| `thinking` | Extended thinking blocks |
| `adaptive_thinking` | Dynamic token budget allocation |
| `interleaved_thinking` | Thinking between tool calls |
**Example — Nova 2 with effort enabled:**
```json
{
"env": {
"ANTHROPIC_AUTH_TOKEN": "YOUR_API_KEY",
"ANTHROPIC_BASE_URL": "https://YOUR_STDAPI_URL/anthropic",
"ANTHROPIC_DEFAULT_SONNET_MODEL": "amazon.nova-2-lite-v1:0",
"ANTHROPIC_DEFAULT_SONNET_MODEL_NAME": "Nova 2 Lite",
"ANTHROPIC_DEFAULT_SONNET_MODEL_SUPPORTED_CAPABILITIES": "effort",
"DISABLE_PROMPT_CACHING": "1"
}
}
```
**Example — Claude with full capabilities declared (e.g. for a Bedrock ARN or inference profile):**
```json
{
"env": {
"ANTHROPIC_DEFAULT_OPUS_MODEL": "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/my-opus",
"ANTHROPIC_DEFAULT_OPUS_MODEL_NAME": "Opus via Bedrock",
"ANTHROPIC_DEFAULT_OPUS_MODEL_SUPPORTED_CAPABILITIES": "effort,max_effort,thinking,adaptive_thinking,interleaved_thinking"
}
}
```
##### :material-robot: Using Non-Claude Models
Claude Code is optimized for Claude models and enables reasoning by default. When routing non-Claude models through stdapi.ai, incompatible reasoning parameters are silently ignored — no special configuration is needed to avoid API errors.
**Models with effort support** (Nova 2, DeepSeek V3) — declare `effort` capability:
```json
{
"env": {
"ANTHROPIC_DEFAULT_SONNET_MODEL": "amazon.nova-2-lite-v1:0",
"ANTHROPIC_DEFAULT_SONNET_MODEL_SUPPORTED_CAPABILITIES": "effort"
}
}
```
Common configuration issues with non-Claude models:
- **Prompt caching** — Claude Code sends `cache_control` headers that can cause errors on models that handle caching differently. Set `DISABLE_PROMPT_CACHING=1` to suppress them.
- **Output token limit** — Claude Code defaults to requesting up to 32,000 output tokens, which exceeds the maximum for many non-Claude models. Set `CLAUDE_CODE_MAX_OUTPUT_TOKENS` to a value within the model's limit to avoid `max_tokens` validation errors.
```json
{
"env": {
"ANTHROPIC_DEFAULT_SONNET_MODEL": "moonshot.kimi-k2-thinking",
"DISABLE_PROMPT_CACHING": "1",
"CLAUDE_CODE_MAX_OUTPUT_TOKENS": "128000"
}
}
```
!!! warning "Don't Switch Models Mid-Conversation"
Avoid switching between Claude models and non-Claude models (like Nova, Kimi, Qwen) within the same conversation. Claude Code and other tools may cache conversation context in a format specific to the model family, and switching can cause errors or unexpected behavior. Start a new conversation when changing model families.
##### :material-format-list-bulleted: Adding a Model to the Picker
Use `ANTHROPIC_CUSTOM_MODEL_OPTION` to add a single custom entry to the `/model` picker without replacing the built-in aliases. Useful for testing a specific Bedrock model ID alongside the standard Claude tiers:
```json
{
"env": {
"ANTHROPIC_CUSTOM_MODEL_OPTION": "moonshot.kimi-k2-thinking",
"ANTHROPIC_CUSTOM_MODEL_OPTION_NAME": "Kimi K2 Thinking",
"ANTHROPIC_CUSTOM_MODEL_OPTION_DESCRIPTION": "Moonshot Kimi K2 Thinking via stdapi.ai"
}
}
```
Claude Code skips validation for this model ID, so any Bedrock model ID accepted by stdapi.ai works here.
#### Other Anthropic-Compatible Tools
Any tool using the Anthropic SDK or messages API can be configured the same way—set the `ANTHROPIC_BASE_URL` to `https://YOUR_STDAPI_URL/anthropic` and `ANTHROPIC_API_KEY` (or equivalent) to your stdapi.ai API key.
[pi](https://github.com/earendil-works/pi) also speaks this API: register its provider with `api: "anthropic-messages"` and the `/anthropic` base URL, as shown in [the pi configuration above](#configuration).
### :material-connection: MCP (Model Context Protocol)
stdapi.ai can act as an MCP server, exposing its API endpoints as tools that MCP-capable clients call directly using the Model Context Protocol. Enable MCP on your stdapi.ai deployment by setting the appropriate environment variable:
| Transport | Endpoint | Config variable | Notes |
|-----------------|----------|-----------------------------------|---------------------------|
| Streamable HTTP | `/mcp` | `ENABLE_MCP_STREAMABLE_HTTP=true` | Recommended |
| SSE | `/sse` | `ENABLE_MCP_SSE=true` | Legacy, for older clients |
#### :material-cog: Configuration
Many MCP clients—including Claude Code and Cline—configure servers via a `mcpServers` JSON block:
```json
{
"mcpServers": {
"stdapi": {
"type": "http",
"url": "https://YOUR_STDAPI_URL/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
- Replace `YOUR_STDAPI_URL` with your stdapi.ai deployment URL (e.g., `https://api.example.com` or `http://localhost:8000` for local)
- The `Authorization` header (and `YOUR_API_KEY`) is only required if your deployment uses API key authentication; omit the `headers` block for local development without a key
- For { style="height: 1.2em; vertical-align: text-bottom;" } **Claude Code**, add this to `~/.claude.json`
!!! tip "SSE Transport (Legacy)"
For older MCP clients that do not support Streamable HTTP, use `"type": "sse"` with the `/sse` endpoint instead. Requires `ENABLE_MCP_SSE=true` on the server.
#### :material-filter: Tool Selection
By default, all tools are exposed. Restrict the tool set for better performance — LLMs work better with fewer choices, and many providers cap active tools per session.
**Choose the model-calling tool by task shape:**
- **`openai_completion`** (`/v1/completions`) — the smallest schema and smallest token footprint per tool call. **Recommended for text-first coding agents**: code generation, completion, refactoring, explanation, Q&A. Supports batch prompts, streaming, and a single-request multimodal collapse (`["instruction", , …]`) for analysing screenshots or reference documents.
- **`openai_chat_completion`** (`/v1/chat/completions`) — use when the agent needs multi-turn conversations with system prompts, built-in function calling, or structured multimodal messages.
- **`openai_response`** (`/v1/responses`) — modern API with tool calling, structured output, and optional server-side storage.
- **`anthropic_message`** (`/anthropic/v1/messages`) — Anthropic SDK compatibility; same Bedrock models, different protocol.
Always include `search_models` — it lets the agent discover available model IDs dynamically rather than relying on hardcoded values. Use it instead of `openai_model_list` or `anthropic_model_list`: it returns richer metadata and supports capability-based filtering (by modality, route, region, and more).
The examples below use the OpenAI tools; replace with the matching `anthropic_*` tools if you prefer the Anthropic protocol.
**Coding agent (text and code only)** — completions, model discovery, and file operations; no image/audio, no destructive tools:
```bash
export MCP_INCLUDE_TOOLS="openai_completion,search_models,openai_embedding,openai_file,openai_file_list,openai_files_get,openai_file_content"
```
**Coding agent with tool calling / multi-turn chat** — when your agent uses OpenAI-style function calling or stateful conversations:
```bash
export MCP_INCLUDE_TOOLS="openai_chat_completion,search_models,openai_embedding,openai_file,openai_file_list,openai_files_get,openai_file_content"
```
**Coding agent with image support** — completions plus image generation and editing:
```bash
export MCP_INCLUDE_TOOLS="openai_completion,search_models,openai_embedding,openai_file,openai_file_list,openai_files_get,openai_file_content,openai_image_generation,openai_image_edit,openai_image_variation"
```
**Coding agent with audio support** — completions plus transcription, translation, and speech synthesis:
```bash
export MCP_INCLUDE_TOOLS="openai_completion,search_models,openai_embedding,openai_file,openai_file_list,openai_files_get,openai_file_content,openai_audio_transcription,openai_audio_translation,openai_audio_speech"
```
**Coding agent with your own documentation** — completions plus semantic search over a [vector store](api_openai_vector_stores.md) holding your design docs, runbooks, or an internal API reference:
```bash
export MCP_INCLUDE_TOOLS="openai_completion,search_models,openai_vector_store_list,openai_vector_store_search,openai_file,openai_file_list,openai_files_get,openai_file_content"
```
`openai_vector_store_search` returns the matching passages with their file names and scores, so the agent looks a fact up instead of guessing at it — and it works whatever wire format the assistant chats with, since the search is a tool call rather than a request field. Indexing the corpus is a one-off, covered in the [RAG Pipelines guide](use_cases_rag.md#managed-retrieval).
In all cases, file deletion tools (`openai_files_delete`, `anthropic_files_delete`) are intentionally omitted — add them only when your workflow explicitly requires cleanup.
!!! warning "Token usage — complex API tools"
`openai_chat_completion`, `openai_response`, and `anthropic_message` expose large schemas (messages, tool definitions, multimodal content parts). Each tool invocation can cost hundreds of extra tokens just to describe the schema. Select them only when your workflow actually needs multi-turn chat, function calling, or structured output — for text-first code Q&A, `openai_completion` is significantly cheaper per call.
See [Configuration Reference → MCP](operations_configuration.md#mcp-model-context-protocol) for the full tool list and selection guidance.
### :material-account-key: One Key for the Team, or One Identity per Developer
An assistant configured with the deployment's API key makes every developer's calls indistinguishable on the AWS bill. A deployment can [authenticate callers with Amazon Cognito user pool tokens](operations_configuration.md#cognito-authentication) instead — alongside the API key or in place of it — so each developer reaches the gateway with a credential of their own. With [per-user cost attribution](operations_cost_management.md#per-user-attribution) enabled, their model calls then run under a short-lived role session of their own, and Cost Explorer and the Cost and Usage Report show what each of them spent, from the invoice rather than an estimate.
The assistant needs no feature for this beyond sending the token it was given: whichever field holds the API key today (`ANTHROPIC_AUTH_TOKEN`, `OPENAI_API_KEY`, a provider entry) carries the access token instead. Renewal is the thing to plan for — an access token expires where an API key does not — so favour a tool that reads its credential from the environment, or from a helper command, on each run over one that stores a key once in a settings file.
### :material-docker: Running stdapi.ai Locally
stdapi.ai works well when running locally with Docker, making it ideal for your development environment.
!!! tip "Running Locally"
For complete local deployment instructions, see the [Local Development Guide](operations_getting_started_local.md).
**OpenAI-compatible tools:**
```
API Base URL: http://localhost:8000/v1
API Key: your_stdapi_key
```
**Anthropic-compatible tools:**
```
ANTHROPIC_BASE_URL: http://localhost:8000/anthropic
ANTHROPIC_AUTH_TOKEN: your_stdapi_key
```
## :material-rocket-launch: Deploy the Gateway on AWS
There is no application to deploy here, and no per-developer infrastructure either — every IDE and terminal assistant on the team points at the same shared gateway. The Terraform sample below is one worked example of a credible AWS deployment for that gateway, not the only architecture that works — it is a normal HTTPS service, so any deployment that gets a URL and a credential to every developer's machine works the same way.
### :material-sitemap: Architecture
The diagram below is the topology the [`getting_started_production` sample](https://github.com/stdapi-ai/samples/tree/main/getting_started_production) builds, putting a public Application Load Balancer in front of the gateway so it is reachable from outside the VPC.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true, 'nodeSpacing': 20, 'rankSpacing': 40, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart TB
dev[" Developer workstations IDE / terminal + coding assistant"]
waf[" AWS WAF (optional) rate limit · anonymous-IP block"]
subgraph public["Your VPC · public subnets"]
alb[" Application Load Balancer HTTPS · ACM certificate custom domain via Route 53"]
end
subgraph private["Your VPC · private app subnets — no inbound route from the internet"]
gateway[" stdapi.ai gateway ECS Fargate · stateless"]
egress[" NAT gateways · one per AZ + free S3 gateway endpoint"]
end
subgraph regional["AWS service endpoints · your account, the regions you configure"]
bedrock[" Amazon Bedrock"]
s3[" Amazon S3 SSE-KMS"]
cw[" Amazon CloudWatch logs · metrics · alarms"]
bedrock ~~~ s3
end
dev -->|"HTTPS · TLS 1.2+ API key or Cognito token"| alb
waf -.->|"optional · alb_waf_enabled"| alb
alb -->|"HTTP · private subnet"| gateway
gateway -->|"S3 gateway endpoint always provisioned"| s3
gateway --> egress
egress -->|"HTTPS · SigV4"| bedrock
egress --> cw
```
The Application Load Balancer, optionally fronted by AWS WAF, is the only address any developer's machine ever reaches — everything past it lives in private app subnets with no inbound route from the internet. Source code sent for a completion or a chat turn crosses the ALB, reaches the gateway task over the private network, and leaves again over SigV4-signed HTTPS straight to Amazon Bedrock; the gateway container is stateless and keeps no request body or completion on disk once the response is sent.
#### What Each AWS Service Does Here
| AWS service | Role in this integration | Where it is configured |
| --- | --- | --- |
| **Elastic Load Balancing** | The single public entry point for every developer's IDE or terminal; terminates TLS | `alb_enabled`, `alb_public` |
| **AWS WAF** | Optional edge protection in front of the ALB — rate limiting and anonymous-IP blocking | `alb_waf_enabled`, `alb_waf_rate_limit`, `alb_waf_block_anonymous_ips` |
| **AWS Certificate Manager / Route 53** | Issues and DNS-validates the TLS certificate for your own domain, and publishes the record that resolves to it | `alb_domain_name` |
| **Amazon ECS on AWS Fargate** | Runs the stdapi.ai gateway container in private app subnets, with at least one task per Availability Zone | `autoscaling_min_capacity` |
| **Amazon Bedrock** | Serves chat completions, tool calling and reasoning for every model the team's assistants call | [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) |
| **Amazon S3** | Holds the gateway's temporary multimodal objects, KMS-encrypted, reached through the always-provisioned S3 gateway endpoint | `aws_s3_bucket_create` |
| **AWS KMS** | Customer-managed keys encrypting the S3 bucket(s) | Terraform module |
| **Amazon CloudWatch** | Container logs, gateway request logs, and optional EMF usage metrics | [Logging & monitoring](operations_logging_monitoring.md) |
| **AWS IAM** | Least-privilege task role restricted to the model and AI-service actions the gateway actually invokes | [IAM permissions](operations_iam_permissions.md) |
#### Security Measures in This Flow
- **Authentication** — see [One Key for the Team, or One Identity per Developer](#one-key-for-the-team-or-one-identity-per-developer) above for the choice between a shared API key and a per-developer credential.
- **Encryption in transit** — HTTPS from every workstation to the ALB, whose listener supports TLS 1.2 and 1.3; a private-subnet hop from the ALB to the gateway task; SigV4-signed HTTPS from the gateway to Amazon Bedrock.
- **Encryption at rest** — SSE-KMS on the S3 bucket(s), with automatic key rotation.
- **Least privilege** — the gateway's task role carries only the model and AI-service actions it calls, and its security group accepts inbound traffic only from the ALB's security group, on the container port.
- **Content policy** — an optional [Bedrock guardrail](operations_configuration.md#bedrock-guardrails) applies to every route a coding assistant reaches, chat included, and stays in force unless the deployment explicitly allows a per-request override.
- **Data handling** — the gateway is stateless and holds request bodies in memory only; CloudWatch receives request metadata, not the source code or the model's replies, unless payload logging is explicitly turned on for debugging.
### :material-gauge: What It Costs to Run
| Charge | Driver |
| --- | --- |
| stdapi.ai licence | $0.10 per gateway container-hour, metered through AWS Marketplace, with a 14-day free trial |
| ECS Fargate | One shared gateway service for the whole team, sized and auto-scaled independently of developer count |
| Elastic Load Balancing | One ALB serving every developer's connections |
| NAT gateways | Standing charge, plus data processing, for the private-subnet egress path — every AWS service call except S3 |
| S3 gateway endpoint | Nothing: a gateway endpoint has no hourly or data charge, and keeps the S3 traffic off the NAT gateways |
| AWS WAF | Optional — only when `alb_waf_enabled = true` |
| Amazon Bedrock usage | Model tokens at AWS rates — the variable charge, and the one prompt caching reduces most |
Coding assistants resend large amounts of repeated context — the system prompt, tool definitions, and often the same files — on every turn, which makes prompt caching the main lever on that variable charge; see the caching notes under [Using Non-Claude Models](#using-non-claude-models) above for when `cache_control` helps and when `DISABLE_PROMPT_CACHING=1` is the right call instead. Read a model's price before you send anything to it with [`GET /model_pricing`](api_model_pricing.md). Setting [`COST_TRACKING=true`](operations_cost_management.md#cost-tracking-real-time-aws-pricing) additionally puts a per-request cost on each usage entry — estimated from published AWS prices, not read back from your invoice.
### :material-eye-outline: What to Watch
The gateway writes one structured `request` (or `request_stream` for streamed replies) event per call, carrying the request id, path, status code, `execution_time_ms`, the model that served it, and the AWS-billed token counts — including `cached_tokens` and `cache_write_tokens` from Bedrock prompt caching. Turning on [`CLOUDWATCH_METRICS`](operations_logging_monitoring.md#cloudwatch-metrics-emf) republishes those counts as EMF metrics in the `stdapi` namespace, dimensioned by `Model`. On a shared deployment, the useful first question is which developer and which model are driving the traffic:
```sql
fields aws_role_session_name, model_id
| filter type = "request" and ispresent(aws_role_session_name)
| stats count(*) as calls, pct(execution_time_ms, 95) as p95_ms by aws_role_session_name, model_id
| sort calls desc
```
`aws_role_session_name` is the identity AWS billed the call under, and it is populated by [per-user cost attribution](operations_cost_management.md#per-user-attribution) — which is an enforced boundary only with `AUTHENTICATION_MODE=cognito`. Under a shared API key the field is absent, and the `request_user_id` an assistant declares is a label rather than a boundary; on that setup, group by `model_id` alone and read the per-team split from Cost Explorer instead.
## :material-arrow-right: Next Steps
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy stdapi.ai to AWS with Terraform
- :material-docker: [**Local Development**](operations_getting_started_local.md) — Run stdapi.ai locally with Docker
- :material-puzzle: [**More Use Cases**](use_cases.md) — Explore other integrations and tools
- :material-cog: [**Configuration Reference**](operations_configuration.md) — Complete list of environment variables
---
## Python Client Libraries
Source: https://stdapi.ai/md/use_cases_python_libraries.md
# :material-language-python: Python Client Libraries Integration
Build Python applications and agents directly on Amazon Bedrock models with stdapi.ai, using the same LangChain, pydantic-ai and OpenAI Agents SDK client classes you would use against OpenAI or Anthropic directly—three client-side changes: the base URL, the API key, and — where the name differs from what the client already sends — the model name, now picked from every provider in the catalogue rather than one vendor's list.
## :material-information-outline: About These Libraries
**🔗 Links:** [LangChain](https://python.langchain.com/) | [pydantic-ai](https://ai.pydantic.dev/) | [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/)
These are three of the most widely used Python frameworks for building LLM-backed applications and agents. All of them ship OpenAI-compatible client classes that accept a custom base URL and API key as constructor arguments—no plugin, wrapper, or extension needed.
**What you can build:**
- **Custom agents** - Tool-calling loops, structured output, and multi-turn conversations in your own Python code
- **RAG applications** - Combine chat models with `OpenAIEmbeddings` for retrieval, see [RAG Pipelines](use_cases_rag.md)
- **Stateful and voice agents** - Server-side conversations, hosted retrieval and spoken sessions through the OpenAI Agents SDK
- **Internal services** - Backend applications and scripts that call Bedrock models without a UI or CLI in between
## :material-help-circle-outline: Why Python Libraries + stdapi.ai?
- :material-swap-horizontal: __Standard Client Classes, No Fork__
`ChatOpenAI`, `OpenAIEmbeddings`, `ChatAnthropic`, pydantic-ai's `OpenAIChatModel` and the Agents SDK's `OpenAIResponsesModel` all accept a custom base URL directly—no gateway-specific SDK to install.
- :material-aws: __Access Amazon Bedrock Models__
Claude, Nova, DeepSeek, Qwen, and 100+ models, called through the same classes your code already imports.
- :material-tools: __Tool Calling and Structured Output__
`bind_tools`, `with_structured_output`, and pydantic-ai's typed tool registration all work end to end against Bedrock models.
- :material-currency-usd-off: __Pay-Per-Use Pricing__
No per-request markup. Pay only Amazon Bedrock rates for the calls your application makes.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
app[" Your Python App (LangChain, pydantic-ai, Agents SDK)"] --> stdapi[" stdapi.ai"]
stdapi --> bedrock[" Amazon Bedrock"]
```
## :material-connection: Connect Your Own Instance
Point any Python process—wherever it runs—at your stdapi.ai gateway. Nothing below requires the AWS sample in [Part 2](#deploy-the-gateway-on-aws).
### :material-check-circle: Prerequisites
!!! info "What You'll Need"
- ✓ **stdapi.ai deployed** - [See deployment guide](operations_getting_started.md) or [run locally with Docker](operations_getting_started_local.md); see [Part 2](#deploy-the-gateway-on-aws) for a Terraform-deployed gateway
- ✓ **Your stdapi.ai URL** - e.g., `https://api.example.com` or `http://localhost:8000` for local
- ✓ **Your API key** - From Terraform output or configuration (optional for local development)
### :material-link-variant: LangChain
#### :material-chat: Chat — `langchain-openai`
`ChatOpenAI` takes the gateway's `/v1` base URL directly. `.invoke()`, `.stream()`, `bind_tools()`, and `with_structured_output()` all work unchanged against Bedrock models.
!!! example "ChatOpenAI"
```python
from langchain_openai import ChatOpenAI
model = ChatOpenAI(
model="anthropic.claude-fable-5",
base_url="https://YOUR_STDAPI_URL/v1",
api_key="YOUR_STDAPI_KEY",
)
response = model.invoke("Name the largest planet in the solar system.")
print(response.content)
```
See [Chat Completions API](api_openai_chat_completions.md) for the full parameter and model reference.
#### :material-vector-polyline: Embeddings — `langchain-openai`
`OpenAIEmbeddings` also takes the `/v1` base URL, but its default behavior needs one extra setting.
!!! warning "Disable client-side tokenization"
`OpenAIEmbeddings` tokenizes its input with `tiktoken` by default and sends the gateway a token-ID array instead of text—an artifact the embeddings endpoint rejects with a `400` error rather than silently embedding something other than what was asked for. Set `check_embedding_ctx_length=False` to send plain text instead:
```python
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(
model="amazon.titan-embed-text-v2:0",
base_url="https://YOUR_STDAPI_URL/v1",
api_key="YOUR_STDAPI_KEY",
check_embedding_ctx_length=False,
)
vector = embeddings.embed_query("Your text here")
```
Without this setting, every call to `embed_query` or `embed_documents` fails—this is the first thing to check if `OpenAIEmbeddings` returns a `400` against stdapi.ai but works against OpenAI directly.
See [Embeddings API](api_openai_embeddings.md) for supported models.
#### :material-robot: Chat — `langchain-anthropic`
`ChatAnthropic` takes the gateway's `/anthropic` base URL and works with every model the route serves, not only Claude.
!!! example "ChatAnthropic"
```python
from langchain_anthropic import ChatAnthropic
model = ChatAnthropic(
model_name="anthropic.claude-fable-5",
base_url="https://YOUR_STDAPI_URL/anthropic",
api_key="YOUR_STDAPI_KEY",
)
response = model.invoke("Name the largest planet in the solar system.")
print(response.content)
```
See [Anthropic Messages API](api_anthropic_messages.md) for the full parameter and model reference.
### :material-robot-outline: pydantic-ai
pydantic-ai's `OpenAIChatModel` reaches the gateway through an `OpenAIProvider` carrying the base URL and API key, and works through the same [Chat Completions API](api_openai_chat_completions.md) route as `ChatOpenAI` above—including reasoning models and multi-turn tool-calling loops.
!!! example "Agent with a custom base URL"
```python
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider
model = OpenAIChatModel(
"anthropic.claude-fable-5",
provider=OpenAIProvider(
base_url="https://YOUR_STDAPI_URL/v1", api_key="YOUR_STDAPI_KEY"
),
)
agent = Agent(model, system_prompt="You are a helpful assistant.")
result = agent.run_sync("Name the largest planet in the solar system.")
print(result.output)
```
Reasoning-capable models (Claude, DeepSeek, and others) work through the same agent, including a full tool-calling loop that reasons on one turn and calls a registered tool on the next. Request a reasoning effort level per call with `model_settings`:
```python
from pydantic_ai.models.openai import OpenAIChatModelSettings
result = agent.run_sync(
"Call the registered tool, then answer.",
model_settings=OpenAIChatModelSettings(openai_reasoning_effort="low"),
)
```
### :material-account-group: OpenAI Agents SDK
The [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) speaks the [Responses API](api_openai_responses.md) natively, so pointing its client at the gateway also hands its agents the server-side surfaces the Responses route serves — stored conversations, hosted retrieval and the realtime voice session included.
!!! example "An agent bound to the gateway"
```python
from agents import Agent, Runner, set_tracing_disabled
from agents.models.openai_responses import OpenAIResponsesModel
from openai import AsyncOpenAI
set_tracing_disabled(True)
client = AsyncOpenAI(base_url="https://YOUR_STDAPI_URL/v1", api_key="YOUR_STDAPI_KEY")
agent = Agent(
name="assistant",
instructions="Answer in one short sentence.",
model=OpenAIResponsesModel(model="anthropic.claude-fable-5", openai_client=client),
)
result = Runner.run_sync(agent, "Name the largest planet in the solar system.")
print(result.final_output)
```
`set_tracing_disabled(True)` matters here: left on, the SDK exports every run to OpenAI's tracing backend with its own key, which is exactly the third party this deployment exists to remove.
#### :material-history: Server-Side Sessions
`OpenAIConversationsSession` keeps an agent's turns in a [conversation](api_openai_conversations.md) on the gateway instead of in the process, so a second run replays what the gateway stored rather than a history you carried:
```python
from agents.memory import OpenAIConversationsSession
session = OpenAIConversationsSession(openai_client=client)
```
Pass it as `Runner.run_sync(..., session=session)`; the conversation id is what makes a run resumable from another process.
#### :material-file-search: Hosted Retrieval
`FileSearchTool(vector_store_ids=[...])` attaches a [vector store](api_openai_vector_stores.md) to the agent, and the search happens inside the response — the SDK's own loop never sees a tool call. Build the store first, as in the [RAG Pipelines guide](use_cases_rag.md#managed-retrieval), then:
```python
from agents import FileSearchTool
agent = Agent(
name="librarian",
instructions="Answer only from the attached notes.",
tools=[FileSearchTool(vector_store_ids=["vs_abc123"], include_search_results=True)],
)
```
#### :material-microphone: Voice Agents
`RealtimeRunner` opens a spoken session against [`WS /v1/realtime`](api_openai_realtime.md): give its `model_config` the `wss://YOUR_STDAPI_URL/v1/realtime?model=` URL and either the API key or a [minted client secret](api_openai_realtime.md#ephemeral-client-secrets). Sessions last up to 8 minutes and call no tools — see the [Realtime API](api_openai_realtime.md#feature-compatibility) for what a session does and does not emit.
## :material-rocket-launch: Deploy the Gateway on AWS
There is no application sample to deploy here — your LangChain, pydantic-ai or OpenAI Agents SDK process is code you already own. The Terraform sample below is one worked example of a credible AWS deployment for the gateway itself, not the only architecture that works: it is a normal HTTPS service, and where your Python process runs relative to its VPC is your choice, not a fixed part of the architecture.
### :material-sitemap: Architecture
The decision that shapes this diagram is where your application process runs relative to the gateway's VPC: deployed as another ECS Fargate service (or any other workload) inside the same VPC, it reaches the gateway over private DNS and needs no public endpoint at all; running anywhere else — a laptop, a different account, another cloud — it reaches the gateway through a public Application Load Balancer instead.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true, 'nodeSpacing': 20, 'rankSpacing': 40, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart TB
extapp[" Your Python App (outside the VPC)"]
subgraph public["Your VPC · public subnets"]
alb[" Application Load Balancer HTTPS · ACM certificate WAF (optional)"]
end
subgraph private["Your VPC · private app subnets — no inbound route from the internet"]
intapp[" Your Python App (in-VPC, e.g. ECS Fargate)"]
stdapi[" stdapi.ai ECS Fargate"]
egress[" NAT gateways · one per AZ + free S3 gateway endpoint"]
end
subgraph regional["AWS service endpoints · your account, the regions you configure"]
bedrock[" Amazon Bedrock"]
s3[" Amazon S3 SSE-KMS"]
cw[" Amazon CloudWatch logs · metrics · traces"]
end
extapp -->|"HTTPS · TLS 1.2+ · bearer token"| alb
alb -->|"HTTP · private subnet"| stdapi
alb ~~~ intapp
intapp -.->|"OpenAI/Anthropic API · bearer token Cloud Map private DNS, no public endpoint"| stdapi
stdapi --> egress
egress -->|"HTTPS · SigV4"| bedrock
egress -->|"S3 gateway endpoint"| s3
egress --> cw
```
The solid path is the public one: an application outside the VPC has no route to the private subnets, so it can only reach the gateway through the ALB, over HTTPS. The dotted path is the in-VPC alternative: an application co-located in the private app subnets resolves the gateway through AWS Cloud Map private DNS and never touches the ALB — for that path, the ALB, its ACM certificate and any WAF module do not need to exist at all. A deployment picks one path or the other for a given application; both are shown here only because the choice is yours to make, not because both run at once.
#### What Each AWS Service Does Here
| AWS service | Role in this integration | Where it is configured |
| --- | --- | --- |
| **Amazon ECS on AWS Fargate** | Runs the stdapi.ai gateway container, and — on the in-VPC path — your Python application as its own service | Terraform module (default) |
| **Elastic Load Balancing** | Public entry point on the out-of-VPC path only; terminates TLS with an ACM certificate | `alb_enabled`, `alb_public` |
| **AWS Cloud Map** | Private DNS name your in-VPC application resolves instead of a public endpoint | `service_discovery_dns_namespace_id`, `service_discovery_dns_name` |
| **Amazon Bedrock** | Chat completions, embeddings, tool calling and reasoning for every client library on this page | [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) |
| **Amazon S3** | Temporary storage for multimodal request and response payloads | Terraform module (default) |
| **AWS KMS** | Customer-managed key encrypting the S3 bucket | Terraform module (default) |
| **AWS Secrets Manager / SSM Parameter Store** | Holds the API key when one is generated or referenced | `api_key_create`, `api_key_ssm_parameter`, `api_key_secretsmanager_secret` |
| **Amazon CloudWatch** | Gateway request logs, EMF usage metrics, and OpenTelemetry trace export | [Logging & monitoring](operations_logging_monitoring.md) |
| **AWS IAM** | Least-privilege task role for the gateway; scoped to the model and AI-service actions it invokes | [IAM permissions](operations_iam_permissions.md) |
#### Security Measures in This Flow
- **Authentication** — every client class on this page sends its `api_key` argument as an `Authorization: Bearer` header, whichever mechanism validates it on the gateway: a stdapi.ai [API key](operations_authentication_security.md#api-key-authentication), an [Amazon Cognito user-pool token](operations_authentication_security.md#amazon-cognito-user-pool-tokens) the gateway verifies itself, or a token from an [OIDC / IAM Identity Center](operations_authentication_security.md#oidc-cognito-iam-identity-center) flow terminated at the ALB before the request reaches stdapi.ai. [AWS IAM SigV4](operations_authentication_security.md#aws-iam) is available only through an API Gateway integration in front of the gateway — these SDKs' bearer-token clients cannot sign a SigV4 request themselves.
- **Encryption in transit** — HTTPS from an out-of-VPC application to the ALB; private-subnet HTTP from the ALB to the gateway container, or Cloud Map DNS with no ALB hop at all on the in-VPC path; HTTPS with SigV4 from the gateway to every AWS service it calls.
- **Encryption at rest** — SSE-KMS on the S3 bucket that holds multimodal payloads, with a customer-managed key.
- **Least privilege** — the gateway's ECS task role carries only the model and AI-service actions its configuration enables, not a blanket Bedrock or S3 grant.
- **Content policy** — a [Bedrock guardrail](operations_configuration.md#bedrock-guardrails) configured on the gateway applies to every chat request your application sends, independent of which client library issued it.
- **Data handling** — the gateway is stateless and holds request bodies in memory only; no third party sits between your application and the models it calls, so a Bedrock request made through stdapi.ai carries no telemetry back to another vendor unless your own client library adds it — see the `set_tracing_disabled(True)` note under [OpenAI Agents SDK](#openai-agents-sdk) above.
### :material-gauge: What It Costs to Run
| Charge | Driver |
| --- | --- |
| stdapi.ai licence | $0.10 per gateway container-hour, metered through AWS Marketplace, with a 14-day free trial on the licence |
| ECS Fargate | The gateway service, sized and auto-scaled independently of your application's own compute |
| Load balancing and networking | An ALB plus the NAT gateways the private subnets egress through, next to the S3 gateway endpoint, which carries no charge — the ALB drops out entirely when your application runs in the same VPC and reaches the gateway over Cloud Map private DNS |
| Model and AI-service usage | Amazon Bedrock at AWS rates, billed to your account with no markup |
Read a model's price before your application sends anything to it with [`GET /model_pricing`](api_model_pricing.md). Setting [`COST_TRACKING=true`](operations_cost_management.md#cost-tracking-real-time-aws-pricing) additionally puts a per-request cost on each usage entry — estimated from published AWS prices, not read back from your invoice.
### :material-eye-outline: What to Watch
The gateway writes one structured `request` event per call to CloudWatch, carrying the request id, path, status code, `execution_time_ms`, the model that served it, and the token counts AWS billed; streaming calls add a matching `request_stream` event. When [`OTEL_ENABLED=true`](operations_configuration.md#otel-enabled) and [`OTEL_EXPORTER_ENDPOINT`](operations_configuration.md#otel-exporter-endpoint) point at a collector, the gateway also exports OpenTelemetry traces for the same calls. Because your application is the client here, the most direct link between the two is the `x-request-id` response header: read it from every gateway response and log it alongside your own request handling, so a failure your application sees can be traced straight back to the gateway event that produced it.
```sql
fields id, path, model_id, status_code
| filter type = "request" and status_code >= 400
| stats count(*) as errors by path, model_id, status_code
| sort errors desc
```
Pair this with your application's own logs, keyed on the `x-request-id` it received, to tell a gateway-side failure apart from one your code introduced before or after the call.
## :material-arrow-right: Next Steps
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy stdapi.ai to AWS with Terraform
- :material-docker: [**Local Development**](operations_getting_started_local.md) — Run stdapi.ai locally with Docker
- :material-magnify: [**RAG Pipelines**](use_cases_rag.md) — Combine embeddings and reranking in a retrieval pipeline
- :material-puzzle: [**More Use Cases**](use_cases.md) — Explore other integrations and tools
---
## Autonomous Agent CLIs
Source: https://stdapi.ai/md/use_cases_autonomous_agents.md
# :material-robot-excited: Autonomous Agent CLIs
Run autonomous agent CLIs against Amazon Bedrock models with stdapi.ai, using the same provider configuration you would point at OpenAI or Anthropic directly — three client-side changes: the base URL, the API key, and the model name, the last only where it differs from what the client already sends, now chosen from every provider in the catalogue rather than one vendor's list.
## :material-information-outline: About Autonomous Agent CLIs
Unlike IDE coding assistants, autonomous agent CLIs plan and execute multi-step tasks on their own—reading files, calling tools, and iterating toward a goal without a human approving each step. They typically run on infrastructure you control (a server, a container, a scheduled job) rather than inside an editor.
**What you can build:**
- **Personal assistants** - Agents that read, search, and act on your behalf from the command line
- **Autonomous research and task loops** - Multi-turn tool-calling sessions that run unattended
- **Self-hosted agent backends** - CLIs wired into cron jobs, CI pipelines, or your own orchestration
## :material-help-circle-outline: Why Autonomous Agent CLIs + stdapi.ai?
- :material-swap-horizontal: __No Vendor Lock-In__
Point the CLI's existing OpenAI- or Anthropic-compatible provider settings at stdapi.ai—no fork, no plugin, no custom integration.
- :material-aws: __Access Amazon Bedrock Models__
Claude, Nova, DeepSeek, Qwen, and 100+ models, driven through the same agent loop your CLI already runs.
- :material-lock: __No Third-Party AI Cloud__
Tool calls and model responses are processed by the AWS services you enable, reached through your own deployment — no third-party AI vendor sits in the request path.
- :material-currency-usd-off: __Pay-Per-Use Pricing__
No per-seat or per-agent licensing. Pay only Amazon Bedrock rates for the calls the agent actually makes.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
agent[" Autonomous Agent CLI (Hermes, OpenClaw)"] --> stdapi[" stdapi.ai"]
stdapi --> bedrock[" Amazon Bedrock"]
```
## :material-connection: Connect Your Own Instance
Point Hermes or OpenClaw—wherever you run it—at your stdapi.ai gateway. Nothing below requires the AWS samples in [Part 2](#deploy-the-full-stack-on-aws).
### :material-check-circle: Prerequisites
!!! info "What You'll Need"
- ✓ **stdapi.ai deployed** - [See deployment guide](operations_getting_started.md) or [run locally with Docker](operations_getting_started_local.md); see [Part 2](#deploy-the-full-stack-on-aws) for a Terraform-deployed option
- ✓ **Your stdapi.ai URL** - e.g., `https://api.example.com` or `http://localhost:8000` for local
- ✓ **Your API key** - From Terraform output or configuration (optional for local development)
### :material-compass-outline: Bootstrapping Authentication
An agent handed a key in an environment variable is ready to go. An agent handed only a URL is not — and that is the common case for an MCP client, a marketplace agent, or anything a user points at a gateway it has never seen. stdapi.ai answers that case with the standard OAuth 2.0 discovery flow, so the agent works out the rest by itself:
```mermaid
sequenceDiagram
autonumber
participant A as Agent
participant G as stdapi.ai
participant I as Authorization server
A->>G: Request without a token
G-->>A: 401 + WWW-Authenticate: Bearer resource_metadata="…", scope="…"
A->>G: GET /.well-known/oauth-protected-resource
G-->>A: authorization_servers, scopes_supported
A->>I: Read the issuer's own metadata, then sign in
I-->>A: Access token
A->>G: Retry with Authorization: Bearer
G-->>A: 200
```
The agent never needs to be told which identity provider you use, where its endpoints are, or which scopes to request — every one of those comes out of step 2 and 4. Enable it by setting [`OAUTH_RESOURCE_IDENTIFIER`](operations_configuration.md#oauth-resource-identifier) and [`OAUTH_AUTHORIZATION_SERVERS`](operations_configuration.md#oauth-authorization-servers) on the deployment.
!!! info "The agent still needs a client identity"
Discovery tells the agent *where* to authenticate; the authorization server decides *who* may. With an Amazon Cognito user pool, register the agent as an app client in the pool and give it that client ID — Cognito supports neither dynamic client registration nor client-id metadata documents, so it cannot be skipped.
Full walkthrough and the exact document served: [Authentication Discovery for Agents](operations_authentication_security.md#authentication-discovery-for-agents).
!!! tip "An authenticated agent can be billed as itself"
Once callers arrive with their own identity, [per-user cost attribution](operations_cost_management.md#per-user-attribution) runs each one's model calls under a short-lived role session of their own, and AWS reports their spend separately in Cost Explorer and the Cost and Usage Report — per agent, per tenant or per end user, from the invoice rather than an estimate. Behind a shared API key the same split is available from the identifier the request declares (`safety_identifier`, or `metadata.user_id` on the Messages API), with the caveat that a caller chooses its own: that is cost metadata, not an authorization boundary.
### :material-database-search: Giving an Agent Your Own Documents
An agent loop is only as grounded as what it can look up. Two paths reach the same [vector stores](api_openai_vector_stores.md), and neither needs a retrieval feature in the CLI itself:
- **As a tool the model calls** — an agent that composes its own [`/v1/responses`](api_openai_responses.md#file-search) request declares `file_search` with the stores it may read, and the model runs the searches its turn needs and cites the files it answered from.
- **As an MCP tool** — with [MCP](api_overview.md#mcp-model-context-protocol) enabled, `openai_vector_store_search` is one more tool in the agent's list, usable by any MCP client whatever wire format it chats with. This is the path for a CLI that builds its own request bodies — Hermes, below, connects any MCP server this way.
Building the store itself — parsing, chunking, embedding, filters — is covered in the [RAG Pipelines guide](use_cases_rag.md#managed-retrieval); an Amazon Bedrock knowledge base you already run is [addressed as a store too](use_cases_rag.md#knowledge-base).
!!! tip "State the gateway can hold for you"
An agent that resends its whole history each turn can instead keep the thread server-side with the [Conversations API](api_openai_conversations.md) and continue it by id — useful for long-running or resumable agents, and for handing one session between processes.
### :material-robot: Hermes
[Hermes](https://github.com/NousResearch/hermes-agent) (PyPI package `hermes-agent`) is an autonomous agent CLI written in Python.
#### :material-cog: Configuration
The simplest setup points Hermes at stdapi.ai through the same environment variables an OpenAI-compatible client would use:
```bash
export OPENAI_API_KEY=YOUR_STDAPI_KEY
export OPENAI_BASE_URL=https://YOUR_STDAPI_URL/v1
```
To select a specific wire format or model, declare a provider in Hermes' `config.yaml` instead:
```yaml
providers:
stdapi:
name: stdapi.ai
api: https://YOUR_STDAPI_URL/v1
key_env: STDAPI_API_KEY
transport: chat_completions
default_model: anthropic.claude-fable-5
model:
provider: stdapi
default: anthropic.claude-fable-5
```
`key_env` names the environment variable Hermes reads the API key from—set `STDAPI_API_KEY` (or whatever name you choose) to your stdapi.ai key.
#### :material-swap-horizontal: Transport Selection
`transport` is the standout setting: it picks which of stdapi.ai's three chat dialects the provider speaks, and `api` has to match the route serving it:
| `transport` | `api` base URL | API |
|---|---|---|
| `chat_completions` | `https://YOUR_STDAPI_URL/v1` | [Chat Completions](api_openai_chat_completions.md) |
| `codex_responses` | `https://YOUR_STDAPI_URL/v1` | [Responses](api_openai_responses.md) |
| `anthropic_messages` | `https://YOUR_STDAPI_URL/anthropic` | [Anthropic Messages](api_anthropic_messages.md) |
Declare more than one entry under `providers` to reach more than one route side by side.
#### :material-cached: Anthropic Prompt-Caching Breakpoints
On the `anthropic_messages` transport, Hermes automatically places [prompt-caching](api_anthropic_messages.md#prompt-caching) breakpoints on the system prompt and recent messages when the target model is Claude-named. Choose the cache lifetime with `prompt_caching.cache_ttl`:
```yaml
prompt_caching:
cache_ttl: 1h # or "5m" (the default)
```
Only `5m` and `1h` are accepted—any other value is ignored. This pairs directly with stdapi.ai's own Anthropic Messages prompt-caching support: Hermes' breakpoints arrive as standard `cache_control` markers, which stdapi.ai translates into Bedrock cache points, up to the four the Converse API allows per request.
### :material-account-cog: OpenClaw
[OpenClaw](https://github.com/openclaw/openclaw) is a personal-assistant CLI (npm package `openclaw`) that also writes code as part of a task. Custom endpoints are registered through its onboarding wizard, not through the `agent` command itself.
```bash
openclaw onboard \
--custom-provider-id stdapi \
--custom-base-url https://YOUR_STDAPI_URL/v1 \
--custom-model-id anthropic.claude-fable-5 \
--custom-compatibility openai \
--custom-api-key YOUR_STDAPI_API_KEY
```
Omit `--custom-api-key` to read the key from `CUSTOM_API_KEY` in the environment instead. Then run the agent with the model qualified by the provider id:
```bash
openclaw agent --model stdapi/anthropic.claude-fable-5
```
`--custom-compatibility` is the standout setting here: one flag picks which of stdapi.ai's three chat dialects OpenClaw speaks, and `--custom-base-url` has to match the route serving it:
| `--custom-compatibility` | `--custom-base-url` | API |
|---|---|---|
| `openai` | `https://YOUR_STDAPI_URL/v1` | [Chat Completions](api_openai_chat_completions.md) |
| `openai-responses` | `https://YOUR_STDAPI_URL/v1` | [Responses](api_openai_responses.md) |
| `anthropic` | `https://YOUR_STDAPI_URL/anthropic` | [Anthropic Messages](api_anthropic_messages.md) |
Re-run `openclaw onboard` with a different `--custom-provider-id` to register more than one route side by side.
## :material-rocket-launch: Deploy the Full Stack on AWS
The samples below are one worked example of a credible AWS deployment, not the only architecture that works. The gateway is a normal HTTPS service, and either agent can run wherever you already operate it.
### :material-sitemap: Architecture
The diagram below is the shape the [Hermes](https://github.com/stdapi-ai/samples/tree/main/getting_started_hermes) and [OpenClaw](https://github.com/stdapi-ai/samples/tree/main/getting_started_openclaw) Terraform samples share: the agent CLI and the stdapi.ai gateway run as separate ECS Fargate tasks in the same private app subnets, with only the agent's own web surface reachable — through an Application Load Balancer — from outside the VPC. Hermes exposes that ALB as two listeners (gateway API and dashboard); OpenClaw multiplexes both onto one. The diagram collapses either shape into a single listener box.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true, 'nodeSpacing': 20, 'rankSpacing': 40, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart TB
user["👤 You (browser · operator)"]
internet["Internet image registry · tool destinations"]
subgraph public["Your VPC · public subnets"]
alb[" Application Load Balancer optional HTTPS · ACM cert + Route 53"]
end
subgraph private["Your VPC · private app subnets — no inbound route from the internet"]
agent[" Agent CLI (Hermes or OpenClaw) · ECS Fargate"]
stdapi[" stdapi.ai ECS Fargate"]
egress[" NAT gateways one per Availability Zone"]
end
subgraph regional["AWS service endpoints · your account, the regions you configure"]
bedrock[" Amazon Bedrock"]
s3[" Amazon S3 SSE-KMS"]
cw[" Amazon CloudWatch logs · metrics · alarms"]
end
user -->|"HTTPS (custom domain) or HTTP from the deploying IP only"| alb
alb -->|"HTTP · private subnet"| agent
agent -->|"OpenAI/Anthropic-compatible API · API key Cloud Map private DNS, no public endpoint"| stdapi
agent -->|"HTTPS · image pull, plus the tool destinations you configure"| egress
egress --> internet
stdapi -->|"HTTPS · SigV4"| egress
egress -->|"HTTPS · SigV4"| bedrock
egress -->|"HTTPS · SigV4"| s3
egress --> cw
```
Two things are worth reading off the picture. stdapi.ai has no listener of its own: the agent reaches it only through AWS Cloud Map private DNS inside the private subnets, so the ALB never forwards to it — only to the agent's own ports. And the agent's egress is a separate path from the gateway's: image pulls and any tool destinations you configure leave as ordinary internet-bound HTTPS, while the gateway's own calls are SigV4-signed and reach AWS service endpoints only. That second path is one of the compensating controls for the sandboxing warning documented under OpenClaw — alongside a task dedicated to nothing but this agent, a task role scoped to what it declares, and a private subnet with no inbound route from the internet.
#### What Each AWS Service Does Here
| AWS service | Role in this integration | Where it is configured |
| --- | --- | --- |
| **Amazon ECS on AWS Fargate** | Runs the agent CLI and the stdapi.ai gateway as separate tasks, each with its own IAM role | Terraform sample (`module "hermes"`/`module "openclaw"`, `module "stdapi_ai"`) |
| **Elastic Load Balancing** | Public entry point for the agent's own gateway API and dashboard/Control UI; never forwards to stdapi.ai | Terraform sample (`alb.tf`) |
| **AWS Certificate Manager & Amazon Route 53** | Optional TLS certificate and DNS record for the ALB, created only when a custom domain is supplied | Terraform sample (`alb_domain_name`, `alb_route53_zone_name`) |
| **AWS Cloud Map** | Private DNS namespace (`internal`) the agent uses to resolve stdapi.ai without a public endpoint | Terraform sample (`aws_service_discovery_private_dns_namespace "internal"`, `service_discovery_dns_name`) |
| **Amazon Bedrock** | Serves the model calls the agent's chosen wire dialect sends through stdapi.ai | [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) |
| **Amazon S3** | The gateway's own bucket for generated and temporary files; KMS-encrypted, versioned, lifecycle-managed | Module baseline (`storage.tf`) |
| **AWS KMS** | Customer-managed key encrypting the gateway's S3 bucket; a separate key, created by the ECS module, encrypts the agent's EFS volumes and Fargate ephemeral storage | Terraform module baseline |
| **Amazon EFS** | Persists the agent's own state (config, sessions, workspace) across redeployments. For Hermes it holds *all* state, including every SQLite database, which is why that service runs a single task | Terraform sample (`mount_points` in `hermes.tf`/`openclaw.tf`) |
| **Amazon CloudWatch** | Container logs, gateway request logs, and, when enabled, EMF usage metrics | [Logging & monitoring](operations_logging_monitoring.md) |
| **AWS IAM** | Separate least-privilege task roles per ECS task; the gateway's role grants only the Bedrock/AI-service actions it calls | [IAM permissions](operations_iam_permissions.md) |
#### Security Measures in This Flow
- **Credential the agent presents** — in these samples the agent authenticates to the gateway with a generated API key baked into its seeded config file (`config.yaml` for Hermes, `openclaw.json` for OpenClaw); an agent that starts with only a URL instead works out where and how to authenticate through the [OAuth 2.0 protected-resource discovery flow](#bootstrapping-authentication) documented above.
- **Encryption in transit** — HTTPS from the operator's browser to the ALB when a custom domain and ACM certificate are configured, plain HTTP otherwise; plain HTTP from the ALB to the agent container and from the agent to stdapi.ai, both confined to the private subnet; HTTPS with SigV4 from the gateway to Amazon Bedrock.
- **Encryption at rest** — SSE-KMS on the gateway's S3 bucket, and a separate customer-managed key encrypting the EFS volumes that hold the agent's persistent state.
- **Least privilege / task-role scoping** — the agent and the gateway each run under their own ECS task role; the gateway's role carries only the Bedrock/AI-service actions it calls, and the agent's task carries none of it — every model call still goes through the gateway. See [IAM permissions](operations_iam_permissions.md).
- **Content policy** — a [Bedrock guardrail](operations_configuration.md#bedrock-guardrails) configured on the gateway applies to model calls regardless of which wire dialect the agent's provider settings select.
- **Cost / identity attribution** — [per-user cost attribution](operations_cost_management.md#per-user-attribution) turns a caller's declared identity into a billing boundary only under `AUTHENTICATION_MODE=cognito`; the API-key mode these samples use makes that identifier client-declared, so every call from a shared deployment is billed to the gateway's own identity.
### :material-robot: Hermes
Deploy Hermes + stdapi.ai together on ECS Fargate, with `config.yaml` pre-seeded and ready to run:
**📦 [stdapi-ai/samples/getting_started_hermes](https://github.com/stdapi-ai/samples/tree/main/getting_started_hermes)**
**What's included:**
- Hermes gateway API and web dashboard on ECS Fargate, each behind its own generated credential
- stdapi.ai gateway connected to Amazon Bedrock, registered as a custom OpenAI-compatible provider
- `config.yaml` seeded with the stdapi.ai URL and API key already filled in, and re-seeded whenever the rendered template changes — so a model or key changed in Terraform reaches the deployment, at the cost of overwriting edits made in the dashboard
- Persistent state (config, sessions, memories, skills) on EFS, so it survives redeployments
- **Exactly one task**, because every one of those stores is SQLite with no external backend ([upstream #38185](https://github.com/NousResearch/hermes-agent/issues/38185)). Recovery is a Fargate reschedule, with no failover — see *Single instance by design* in the sample's README
- No local image build — the public `nousresearch/hermes-agent` image is pulled anonymously by Fargate
- ECS Exec enabled for shelling into the container or driving Hermes' interactive CLI directly
**Deploy:**
```bash
git clone https://github.com/stdapi-ai/samples.git
cd samples/getting_started_hermes/terraform
tofu init
tofu apply
```
#### :material-gauge: What It Costs to Run
| Charge | Driver |
| --- | --- |
| stdapi.ai licence | $0.10 per gateway container-hour, metered through AWS Marketplace, with a 14-day free trial on the licence |
| ECS Fargate | The Hermes task — pinned to exactly one, see *Single instance by design* in the sample's README — and the gateway task, each sized independently; a short-lived init container re-seeds `config.yaml` on every deployment |
| Load balancing and networking | One ALB fronting the gateway API and dashboard, plus the NAT gateways the private subnets egress through |
| Amazon EFS | Standing storage and throughput cost for `/opt/data` — config, SQLite state, sessions, memories, skills |
| Amazon Bedrock usage | AWS rates, billed to your account with no markup — for a long agent loop, how much of each turn's context is served from a cache read rather than fresh input is the dominant lever; see [Anthropic Prompt-Caching Breakpoints](#anthropic-prompt-caching-breakpoints), above |
Read a model's price before sending anything to it with [`GET /model_pricing`](api_model_pricing.md). Setting [`COST_TRACKING=true`](operations_cost_management.md#cost-tracking-real-time-aws-pricing) additionally puts a per-request cost on each usage entry — estimated from published AWS prices, not read back from your invoice.
#### :material-eye-outline: What to Watch
Hermes and the gateway each log to CloudWatch: Hermes writes its own container logs, and the gateway writes one structured `request` event (or `request_stream` for streamed replies) per call, carrying the request id, path, status code, `execution_time_ms`, and a nested `usage` list with the token counts AWS billed — including `cached_tokens` (cache reads) and `cache_write_tokens` on each entry. Turning on [`CLOUDWATCH_METRICS`](operations_logging_monitoring.md#cloudwatch-metrics-emf) republishes those counts as EMF metrics in the `stdapi` namespace, dimensioned by `Model`.
An agent loop that resends its history every turn should, once the session warms up, read far more input tokens from cache than it pays for fresh:
```sql
fields Model, InputTokens, CachedTokens, CacheWriteTokens
| filter _aws.CloudWatchMetrics is not null
| stats sum(CachedTokens) as cache_reads, sum(InputTokens) as fresh_input, sum(CacheWriteTokens) as cache_writes by Model
| sort cache_reads desc
```
A model whose `cache_reads` stay near zero mid-session points at breakpoints that are not landing — check that Hermes is actually sending `cache_control` markers before assuming the model itself is at fault.
### :material-account-cog: OpenClaw
Deploy OpenClaw + stdapi.ai together on ECS Fargate, with the provider and default model preconfigured:
**📦 [stdapi-ai/samples/getting_started_openclaw](https://github.com/stdapi-ai/samples/tree/main/getting_started_openclaw)**
**What's included:**
- OpenClaw gateway and Control UI on ECS Fargate, authenticated with a generated token
- stdapi.ai gateway connected to Amazon Bedrock, registered as a custom OpenAI-compatible provider (`api: "openai-completions"`)
- `openclaw.json` seeded on first boot with the provider and default model already filled in
- Persistent config, auth material, and workspace on EFS
- No local image build — the public `ghcr.io/openclaw/openclaw` image is pulled anonymously by Fargate
!!! warning "Agent sandboxing is off"
OpenClaw can run agent tool calls inside a nested sandbox, but Fargate exposes none of the backends that requires (most commonly a host Docker socket), so this sample sets `agents.defaults.sandbox.mode` to `"off"`. Tool calls therefore run directly inside the container itself, with only the container's own boundary as isolation. Do not point this deployment at an agent workload you would not trust to run arbitrary commands there.
!!! warning "Device pairing stays manual"
OpenClaw's Control UI requires pairing a browser device before use, done through a few `aws ecs execute-command` calls after deployment — see the sample's README for the exact commands. This is not automated by Terraform.
**Deploy:**
```bash
git clone https://github.com/stdapi-ai/samples.git
cd samples/getting_started_openclaw/terraform
tofu init
tofu apply
```
#### :material-gauge: What It Costs to Run
| Charge | Driver |
| --- | --- |
| stdapi.ai licence | $0.10 per gateway container-hour, metered through AWS Marketplace, with a 14-day free trial on the licence |
| ECS Fargate | The OpenClaw task — pinned to exactly one, see *Single Instance, by Design* in the sample's README — and the gateway task, each sized independently; a short-lived init container seeds `openclaw.json` on first boot only, and runs harmlessly on every later deployment |
| Load balancing and networking | One ALB fronting the gateway and Control UI, plus the NAT gateways the private subnets egress through |
| Amazon EFS | Standing storage and throughput cost for OpenClaw's config, auth material and workspace |
| Amazon Bedrock usage | AWS rates, billed to your account with no markup |
Read a model's price before sending anything to it with [`GET /model_pricing`](api_model_pricing.md). Setting [`COST_TRACKING=true`](operations_cost_management.md#cost-tracking-real-time-aws-pricing) additionally puts a per-request cost on each usage entry — estimated from published AWS prices, not read back from your invoice.
#### :material-eye-outline: What to Watch
OpenClaw and the gateway each write their own container logs to CloudWatch; the gateway additionally writes one structured `request` event per call, carrying the request id, path, status code, `execution_time_ms`, the model that served it, and the token counts AWS billed. Turning on [`CLOUDWATCH_METRICS`](operations_logging_monitoring.md#cloudwatch-metrics-emf) republishes those counts as EMF metrics in the `stdapi` namespace, dimensioned by `Model`.
```sql
fields model_id, execution_time_ms
| filter type = "request" and ispresent(model_id)
| stats count(*) as calls, pct(execution_time_ms, 95) as p95_ms by model_id
| sort calls desc
```
A rising p95 here is what an operator notices as a slow agent turn, before it shows up in any cost report.
## :material-arrow-right: Next Steps
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy stdapi.ai to AWS with Terraform
- :material-docker: [**Local Development**](operations_getting_started_local.md) — Run stdapi.ai locally with Docker
- :material-language-python: [**Python Client Libraries**](use_cases_python_libraries.md) — Configuring LangChain, pydantic-ai and the OpenAI Agents SDK directly against stdapi.ai
- :material-puzzle: [**More Use Cases**](use_cases.md) — Explore other integrations and tools
---
## RAG Pipelines
Source: https://stdapi.ai/md/use_cases_rag.md
# :material-magnify: RAG Pipelines Integration
Build retrieval-augmented generation and semantic search pipelines on Amazon Bedrock through stdapi.ai's OpenAI-compatible embeddings and Cohere-compatible reranking—one deployment serving every stage of the pipeline.
There are two ways to do it, and they share the same deployment:
- **Managed** — attach your files to a [vector store](api_openai_vector_stores.md) and search it, or let the model search it for itself. Chunking, embedding, indexing and retrieval are the gateway's job; you write no pipeline.
- **Assembled** — keep your own framework and vector database, and use stdapi.ai for the embedding, reranking and generation calls.
Start managed, and move to the assembled pipeline when you need a retrieval strategy of your own.
## :material-information-outline: About Retrieval-Augmented Generation
A RAG pipeline grounds a model's answer in your own documents instead of its training data: a retriever finds candidate passages by vector similarity, an optional reranker reorders them by relevance to the actual question, and a chat model answers from the reordered context.
**What a RAG pipeline needs from its backend:**
- **Embeddings** - Vectorize documents and queries into the same space
- **Reranking** - Reorder retrieved candidates by relevance before they reach the model
- **Generation** - Answer from the retrieved context with a chat-capable model
stdapi.ai serves all three from Amazon Bedrock through standard, unmodified client libraries.
## :material-help-circle-outline: Why RAG + stdapi.ai?
- :material-database-search: __Managed Vector Stores__
Attach a file to a [vector store](api_openai_vector_stores.md) and search it by meaning—no chunker, embedder or vector database to run.
- :material-database-import: __Your Existing Knowledge Base__
An [Amazon Bedrock knowledge base](api_openai_vector_stores.md#knowledge-base-stores) you already run is addressed as a vector store too—searched and extended, never recreated.
- :material-swap-horizontal: __Two Dialects, One Deployment__
Point your embedder and chat model at the OpenAI-compatible `/v1` route, and your reranker at the Cohere-compatible `/cohere` route—no separate services to run.
- :material-aws: __Bedrock Embedding and Rerank Models__
Amazon Titan, Cohere Embed, and Cohere Rerank, served through the SDKs your RAG framework already uses.
- :material-lock: __Enterprise Data Privacy__
Documents, queries, and embeddings never leave your AWS account.
- :material-currency-usd-off: __Pay-Per-Use Pricing__
No per-query fees or vector-database markup. Pay only Amazon Bedrock rates for the embed, rerank, and generation calls you make.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
rag["Your RAG Framework (Haystack, LlamaIndex, ...)"] --> stdapi[" stdapi.ai"]
stdapi --> bedrock[" Amazon Bedrock"]
```
## :material-connection: Connect Your Own Instance
Both retrieval modes work against any stdapi.ai deployment, wherever it runs. Nothing below requires the AWS sample in [Part 2](#deploy-the-full-stack-on-aws).
### :material-database-search: Managed Retrieval with Vector Stores { #managed-retrieval }
Upload your files, attach them to a vector store, and search it. Nothing else runs.
!!! example "Index and search"
```python
import time
from openai import OpenAI
client = OpenAI(base_url="https://YOUR_STDAPI_URL/v1", api_key="YOUR_API_KEY")
uploaded = client.files.create(file=open("handbook.txt", "rb"), purpose="assistants")
store = client.vector_stores.create(name="handbook", file_ids=[uploaded.id])
# Indexing is asynchronous: wait until the store reports it finished.
while client.vector_stores.retrieve(store.id).status == "in_progress":
time.sleep(2)
for result in client.vector_stores.search(
store.id, query="How much parental leave do I get?"
):
print(result.score, result.filename, result.content[0].text)
```
Feed the returned passages to a chat model as context and you have a complete RAG loop in a dozen lines. Tag files with `attributes` to scope a search to a department, a product or a language.
Only **text** files can be indexed — convert PDFs and office documents first, with the [document parsing](#document-parsing) stage below. See the [Vector Stores API](api_openai_vector_stores.md) for chunking, filters, expiration and the storage it needs in your account.
!!! tip "Indexing that survives a deployment"
A file is indexed by the server that accepted it. Point [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url) at a queue and the job is handed over to it instead, so a bulk ingestion keeps going — and finishes — when that server is replaced mid-way. See [Durable indexing](api_openai_vector_stores.md#durable-indexing).
#### :material-file-search: Let the Model Do the Retrieving { #file-search }
Naming a store as a `file_search` tool on the [Responses API](api_openai_responses.md#file-search) moves the whole loop into a single request: the model decides when to search and with which query, answers from the passages it gets back, and annotates the answer with a citation per file it drew on.
!!! example "One request, retrieval included"
```python
response = client.responses.create(
model="amazon.nova-2-lite-v1:0",
input="How much parental leave do I get?",
tools=[{"type": "file_search", "vector_store_ids": [store.id]}],
)
print(response.output_text)
```
No retriever to call, no context to assemble, no citation bookkeeping: the searches the turn ran come back as `file_search_call` items when you want to show them, and `include=["file_search_call.results"]` returns the passages themselves. See [File Search](api_openai_responses.md#file-search) for attribute filters, score thresholds and the streamed event order.
#### :material-database-import: Searching a Knowledge Base You Already Run { #knowledge-base }
If your documents are already in an **Amazon Bedrock knowledge base**, keep it where it is. Allowlist it in [`AWS_BEDROCK_KNOWLEDGE_BASE_IDS`](operations_configuration.md#aws-bedrock-knowledge-base-ids) and it is addressed as the vector store `vs_kb_` — searched, listed, read and (on a custom data source) extended with new documents, through the same client code and the same `file_search` tool as above.
The knowledge base stays yours: it is never created and never deleted here, and a request that would reshape it — renaming, an expiry, a chunking strategy — is refused with the reason rather than half-applied. Its retrieval scores are reported as the backend states them, so a `score_threshold` against such a store is refused instead of meaning something else. See [Knowledge Base Stores](api_openai_vector_stores.md#knowledge-base-stores).
### :material-hammer-screwdriver: Assembling Your Own Pipeline { #assembled-pipeline }
The rest of this part wires stdapi.ai into a pipeline you build yourself, with your own vector database and retrieval strategy.
### :material-check-circle: Prerequisites
!!! info "What You'll Need"
- ✓ **stdapi.ai deployed** - [See deployment guide](operations_getting_started.md) or [run locally with Docker](operations_getting_started_local.md)
- ✓ **Your stdapi.ai URL** - e.g., `https://api.example.com`
- ✓ **Your API key** - From Terraform output or configuration
- ✓ **A vector store** - for the assembled pipeline only: stdapi.ai serves embeddings and reranking, and the vectors live in your framework's own store (in-memory, pgvector, Qdrant, and others all work). [Managed retrieval](#managed-retrieval) needs none.
### :material-cog: Configuration
Every RAG framework that speaks the OpenAI and Cohere SDKs follows the same pattern: the embedder and the generator take the OpenAI-compatible `/v1` base URL, and the reranker takes the Cohere-compatible `/cohere` base URL.
#### :material-file-document-outline: Document Parsing { #document-parsing }
Before embedding, a RAG pipeline needs plain text out of PDFs and office documents. [Docling Serve](https://github.com/docling-project/docling-serve) converts them to Markdown/JSON over an HTTP API — deploy it as the ingestion stage in front of the embedder below, not as a standalone gateway showcase.
Docling's default pipeline is classical layout/OCR/table-structure extraction and never calls an LLM. Its optional VLM pipeline additionally routes page images through stdapi.ai to a vision-capable Bedrock model, for documents that benefit from model-assisted layout understanding.
!!! example "Convert a document"
```bash
curl -s -X POST "$DOCLING_URL/v1/convert/source" \
-H 'Content-Type: application/json' \
-d '{
"options": {"to_formats": ["md"]},
"sources": [{"kind": "http", "url": "https://example.com/document.pdf"}]
}' | jq -r '.document.md_content'
```
Docling Serve's own usage documentation shows `http_sources` here, but the server's OpenAPI schema requires the `sources` array with a `kind` discriminator shown above.
Feed the returned Markdown into the embedder below to complete the ingestion stage.
See [Part 2](#deploy-the-full-stack-on-aws) for a Terraform-deployed Docling Serve on AWS.
#### :material-vector-polyline: Embeddings
Point your framework's OpenAI-compatible embedder at `/v1` with an embeddings-capable model.
!!! example "Haystack"
```python
from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
from haystack.utils import Secret
document_embedder = OpenAIDocumentEmbedder(
api_key=Secret.from_env_var("STDAPI_API_KEY"),
model="amazon.titan-embed-text-v2:0",
api_base_url="https://YOUR_STDAPI_URL/v1",
)
text_embedder = OpenAITextEmbedder(
api_key=Secret.from_env_var("STDAPI_API_KEY"),
model="amazon.titan-embed-text-v2:0",
api_base_url="https://YOUR_STDAPI_URL/v1",
)
```
Embed your corpus with `OpenAIDocumentEmbedder` and each incoming query with `OpenAITextEmbedder`, using the same model for both so the vectors share a space. See [Embeddings API](api_openai_embeddings.md) for supported models.
!!! tip "A first ingestion does not have to run synchronously"
Embedding a whole corpus is exactly the shape the [Batch API](api_openai_batches.md) is for: write one `/v1/embeddings` request per passage into a JSONL file, submit it, and collect the vectors when it finishes — at the Amazon Bedrock batch price rather than the synchronous one. Queries stay on the synchronous route, where latency matters.
#### :material-sort-variant: Reranking
Point your framework's Cohere-compatible reranker at `/cohere`, not the full rerank path—the Cohere client appends the operation itself.
!!! example "Haystack"
```python
from haystack_integrations.components.rankers.cohere import CohereRanker
from haystack.utils import Secret
ranker = CohereRanker(
api_key=Secret.from_env_var("STDAPI_API_KEY"),
model="cohere.rerank-v3-5:0",
api_base_url="https://YOUR_STDAPI_URL/cohere",
top_k=3,
)
```
Requires the `cohere-haystack` integration package alongside `haystack-ai`. Give the ranker the retriever's full candidate set—every retrieved document, not just the top few—so it has something to reorder rather than merely confirm. See [Cohere Rerank API](api_cohere_rerank.md) for supported models.
!!! tip "Regional availability"
Amazon Bedrock serves reranking from a subset of regions only. Keep at least one of them in [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions); stdapi.ai fails over to it automatically.
#### :material-chat: Generation
Point your framework's OpenAI-compatible chat generator at `/v1`, answering from the reranked context.
!!! example "Haystack"
```python
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.utils import Secret
generator = OpenAIChatGenerator(
api_key=Secret.from_env_var("STDAPI_API_KEY"),
model="anthropic.claude-haiku-4-5-20251001-v1:0",
api_base_url="https://YOUR_STDAPI_URL/v1",
)
```
Any text/chat-capable model works here; it never needs to match the embedding or reranking model. See [Chat Completions API](api_openai_chat_completions.md) for supported models.
#### :material-toolbox: Other Frameworks
The same two-route pattern—OpenAI-compatible `/v1` for embedding and generation, Cohere-compatible `/cohere` for reranking—applies to any framework built on those SDKs, including LlamaIndex, RAGFlow, and LightRAG. Set the base URL and API key on the framework's OpenAI and Cohere client configuration; the vector store itself (pgvector, Qdrant, or an in-memory store) is unaffected and stores whatever the embedder returns.
Frameworks that drive the Responses API can take the [managed](#managed-retrieval) half instead of a store of their own: given a file and a `file_search` tool, **Agno** uploads it, creates the vector store, waits for the indexing and names the store on the turn — all against the base URL it already chats with.
!!! warning "n8n cannot rerank through stdapi.ai"
n8n's Cohere Reranker node has no base URL field—see [n8n Integration: Known Limitations](use_cases_n8n.md#known-limitations) for a workaround.
## :material-rocket-launch: Deploy the Full Stack on AWS
The architecture below is one worked example of a credible AWS deployment for this pipeline, not the only one that works: the gateway is a normal HTTPS service, and the document-parsing and vector-store pieces it depicts can run wherever you already operate workloads.
### :material-sitemap: Architecture
The two modes differ in who runs the pipeline and where its state lives. Managed retrieval puts chunking, embedding, indexing and search inside the gateway, backed by a vector bucket and a durable indexing queue in your account; the assembled pipeline keeps your framework in charge of chunking, indexing and search, and calls stdapi.ai only for the embedding, reranking and generation steps in between.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true, 'nodeSpacing': 20, 'rankSpacing': 40, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart TB
app["Your application or RAG framework (Haystack, LlamaIndex, ...)"]
ownvdb["Your vector database (pgvector, Qdrant, ...) assembled pipeline only"]
subgraph public["Your VPC · public subnets"]
alb[" Application Load Balancer document parsing only"]
end
subgraph private["Your VPC · private app subnets — no inbound route from the internet"]
docling[" Docling Serve ECS Fargate · document parsing"]
stdapi[" stdapi.ai ECS Fargate"]
egress[" NAT gateways one per Availability Zone"]
end
subgraph endpoints["AWS service endpoints · your account, the regions you configure"]
bedrock[" Amazon Bedrock"]
s3[" Amazon S3 files and vector store records"]
s3vectors[" Amazon S3 Vectors vector indexes managed retrieval only"]
sqs[" Amazon SQS durable indexing queue + DLQ managed retrieval only"]
bedrock ~~~ s3vectors
s3 ~~~ sqs
end
app -->|"HTTPS · API key parse PDFs/office documents"| alb
alb -->|"HTTP · private subnet"| docling
docling -->|"OpenAI API · API key optional VLM pipeline"| stdapi
app -->|"managed: upload, attach, search a store, then generate the answer"| stdapi
app -.->|"assembled: embed passages, embed the query, rerank, then generate the answer"| stdapi
app -.->|"assembled: index and search vectors yourself"| ownvdb
stdapi --> egress
egress -->|"HTTPS · SigV4 embed · rerank · generate"| bedrock
egress -->|"HTTPS · SigV4 vector store records"| s3
egress -->|"HTTPS · SigV4 index and search vectors"| s3vectors
egress -.->|"SendMessage / ReceiveMessage any task may resume a job"| sqs
```
Two things are worth reading off the picture. In managed retrieval, your documents and their vectors come to rest inside your account — the general-purpose bucket holds the vector store's records, the S3 Vectors bucket holds the indexed embeddings, and the SQS queue makes an indexing job outlive the task that accepted it: this is the one deployment shape on this page where the gateway is not stateless. In the assembled pipeline the gateway stays exactly as stateless as everywhere else on this site — it holds no vectors at all, which land wherever your own framework's vector database puts them, never in this picture.
#### What Each AWS Service Does Here
| AWS service | Role in this integration | Where it is configured |
| --- | --- | --- |
| **Amazon ECS on AWS Fargate** | Runs the stdapi.ai gateway and, for the document-parsing stage, Docling Serve, as independent services | Terraform sample |
| **Elastic Load Balancing** | Public entry point for Docling Serve's document-parsing API; the gateway itself has no listener of its own | Terraform sample |
| **Amazon Bedrock** | Embedding, reranking and generation models for both modes, plus the vision model behind Docling's optional VLM pipeline | [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) |
| **Amazon S3** | Holds the Vector Stores API's own records — the stores, their attached files and file batches | [`AWS_S3_BUCKET`](operations_configuration.md#aws-s3-bucket) |
| **Amazon S3 Vectors** | Holds the indexed embeddings of every managed vector store, one index per store | [`AWS_S3_VECTORS_BUCKET`](operations_configuration.md#aws-s3-vectors-bucket) |
| **Amazon SQS** | Carries indexing jobs so an attach survives the task that accepted it, redriving a job that keeps failing to its dead-letter queue | [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url) |
| **AWS KMS** | Customer-managed keys encrypting the S3 bucket, the S3 Vectors bucket and the SQS queue, each independently | Terraform sample |
| **AWS IAM** | Least-privilege task role for the gateway; the Vector Stores, durable indexing and knowledge base permissions are granted only when their feature is enabled | [IAM permissions](operations_iam_permissions.md#vector-stores-optional) |
#### Security Measures in This Flow
- **Authentication** — every call your application or framework makes to the gateway carries a stdapi.ai [API key](operations_authentication_security.md#api-key-authentication); Docling's optional VLM pipeline authenticates the same way when it calls the gateway for vision inference.
- **Encryption in transit** — HTTPS from wherever your application or framework runs to the gateway, and HTTPS with SigV4 from the gateway to Amazon Bedrock, Amazon S3, Amazon S3 Vectors and Amazon SQS.
- **Encryption at rest** — SSE-KMS on the S3 bucket holding vector store records, and on the S3 Vectors bucket and the SQS queue, each behind its own key: `aws_s3_vectors_kms_key_arn` and `aws_sqs_vector_store_queue_kms_key_arn` when you bring your own bucket or queue, a dedicated key created for you otherwise.
- **Least privilege** — the gateway's task role is granted the Vector Stores, durable indexing and knowledge base actions only when the corresponding feature is configured, scoped to the ARNs of the bucket, the queue and the knowledge bases you name.
- **Content policy** — a [Bedrock guardrail](operations_configuration.md#bedrock-guardrails) applies to embeddings and reranking through the ApplyGuardrail API rather than Bedrock's native integration, checking each text input before it is embedded or reranked, and its consumed units are reported — unlike the native integration chat routes use.
- **Data handling** — uploaded files and their [S3 storage](operations_compliance.md#s3-data-storage) stay in your account, and an Amazon Bedrock knowledge base you already operate is [addressed as a vector store](api_openai_vector_stores.md#knowledge-base-stores) — searched and extended, never recreated.
### :material-cube-outline: What's Included
**📦 [stdapi-ai/samples/getting_started_docling](https://github.com/stdapi-ai/samples/tree/main/getting_started_docling)** is one worked example of the document-parsing stage on AWS — Docling Serve on ECS Fargate, CPU-only, with the VLM pipeline pre-wired to a Bedrock vision model through stdapi.ai. See the sample's [README](https://github.com/stdapi-ai/samples/tree/main/getting_started_docling#readme) for deployment steps.
Want the platform instead of the pipeline? This page covers wiring stdapi.ai into a pipeline you assemble yourself. If you would rather run a finished product—a web UI, document parsing, knowledge bases, and grounded chat, with no pipeline code to write—see the **[RAGFlow Integration](use_cases_ragflow.md)** guide, which deploys RAGFlow with all three stages already bound to Amazon Bedrock.
### :material-gauge: What It Costs to Run
| Charge | Driver |
| --- | --- |
| stdapi.ai licence | $0.10 per gateway container-hour, metered through AWS Marketplace, with a 14-day free trial on the licence |
| Amazon ECS on AWS Fargate | The gateway, and — for the document-parsing stage — Docling Serve, each sized independently |
| Amazon Bedrock — ingestion | One embedding call per indexed passage, paid once when a file is attached |
| Amazon Bedrock — query | One embedding call per search query; the assembled pipeline adds a reranking call, and either mode adds a generation call whenever a chat model answers from the retrieved passages |
| Amazon S3 Vectors | Storage and request charges for the vector indexes themselves — see [Vector Stores pricing](operations_cost_management.md#vector-stores) for what is and is not in stdapi.ai's usage log |
| Amazon S3 | Standard storage for the vector store's own records and any uploaded files |
| Amazon SQS | No standing charge — billed per request, so an idle queue between indexing jobs costs nothing |
Read a model's price before you send anything to it with [`GET /model_pricing`](api_model_pricing.md). Setting [`COST_TRACKING=true`](operations_cost_management.md#cost-tracking-real-time-aws-pricing) additionally puts a per-request cost on each usage entry — estimated from published AWS prices, not read back from your invoice — though the vector bucket's own storage and request charges never appear in it; read those from AWS Cost Explorer instead.
### :material-eye-outline: What to Watch
The gateway logs one `request` event per call and, for managed indexing, a separate `background` event named `vector_store_indexing` correlated to it by `id` — group on that field to see how long a file's indexing actually took after the attach call returned. The dead-letter queue behind [`AWS_SQS_VECTOR_STORE_QUEUE_URL`](operations_configuration.md#aws-sqs-vector-store-queue-url) is the signal that a document failed to index: a file still failing after its queue's redrive policy exhausts its deliveries lands there instead of being silently dropped, so a non-empty dead-letter queue means a document needs attention. Turning on [`CLOUDWATCH_METRICS`](operations_logging_monitoring.md#cloudwatch-metrics-emf) republishes the same billed quantities as EMF metrics in the `stdapi` namespace, dimensioned by `Model`.
```sql
fields @timestamp, event, execution_time_ms, id
| filter type = "background" and event = "vector_store_indexing"
| sort @timestamp desc
| limit 100
```
Watch the dead-letter queue depth alongside this query — a job that never appears here again after being sent once, with the primary queue's `ApproximateNumberOfMessagesVisible` back at zero, finished; one that keeps reappearing is heading for the dead-letter queue instead.
## :material-arrow-right: Next Steps
- :material-database-search: [**Vector Stores API**](api_openai_vector_stores.md) — The managed retrieval half of this page
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy stdapi.ai to AWS with Terraform
- :material-docker: [**Local Development**](operations_getting_started_local.md) — Run stdapi.ai locally with Docker
- :material-puzzle: [**More Use Cases**](use_cases.md) — Explore other integrations and tools
- :material-api: [**API Overview**](api_overview.md) — Explore supported endpoints
---
## RAGFlow Integration
Source: https://stdapi.ai/md/use_cases_ragflow.md
# :material-file-document-multiple-outline: RAGFlow Integration
Run RAGFlow as a complete, self-hosted retrieval-augmented generation platform with Amazon Bedrock behind it. Document parsing, embeddings, reranking, and answer synthesis all reach Bedrock through stdapi.ai — and the models are already bound to the tenant before the first login.
## :material-information-outline: About RAGFlow
**🔗 Links:** [Website](https://ragflow.io/) | [GitHub](https://github.com/infiniflow/ragflow) | [Documentation](https://ragflow.io/docs/dev/)
RAGFlow is an open-source (Apache-2.0) RAG engine built around deep document understanding. Rather than a library you assemble a pipeline from, it is a finished product: a web UI for knowledge bases, a document parser, a retrieval stack, a chat assistant with citations, and an agent builder.
**Key Features:**
- **Deep document parsing** - Layout-aware extraction from PDFs, office documents, and images, with chunk-level visual grounding
- **Knowledge bases** - Upload, parse, chunk, and index documents through the UI, with per-base parsing and embedding settings
- **Hybrid retrieval** - BM25 keyword scoring combined with vector similarity, then reranked before generation
- **Grounded chat** - Answers cite the exact chunks they came from, so every claim is traceable to a source page
- **Agents** - A visual builder for multi-step retrieval and tool workflows on top of the knowledge bases
## :material-help-circle-outline: Why RAGFlow + stdapi.ai?
- :material-layers-triple: __Every RAG Stage, One Gateway__
Chat, embeddings, and reranking are three different API dialects. stdapi.ai serves all three from one deployment, so RAGFlow needs a single endpoint and a single key.
- :material-aws: __Access Amazon Bedrock Models__
Nova and Claude for synthesis, Cohere Embed and Amazon Titan for indexing, Cohere Rerank for two-stage retrieval — all through Bedrock.
- :material-sort-variant: __Reranking Without a Second Vendor__
RAGFlow's reranking stage speaks the Cohere API. It reaches Bedrock's `Rerank` through stdapi.ai's [Cohere-compatible route](api_cohere_rerank.md), with no Cohere account involved.
- :material-lock: __Enterprise Data Privacy__
Your documents, their chunks, and their vectors stay in your AWS account, in services you own.
- :material-currency-usd-off: __Pay-Per-Use Pricing__
No RAGFlow Cloud subscription and no per-seat fees. Pay Amazon Bedrock rates for the parse, embed, rerank, and chat calls you actually make.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
ragflow[" RAGFlow"] --> stdapi[" stdapi.ai"]
stdapi --> bedrock[" Amazon Bedrock"]
```
## :material-connection: Connect Your Own Instance
Point any RAGFlow instance's model providers at stdapi.ai — the deployment underneath doesn't matter to RAGFlow.
### :material-check-circle: Prerequisites
!!! info "What You'll Need"
- ✓ **stdapi.ai deployed** - [See deployment guide](operations_getting_started.md)
- ✓ **Your stdapi.ai URL** - e.g., `https://api.example.com`
- ✓ **Your API key** - From Terraform output or configuration
- ✓ **RAGFlow instance** - Running or ready to deploy (see Deployment section below), on **x86_64** — RAGFlow publishes no arm64 image
- ✓ **A reranking region** - Bedrock serves `Rerank` from a subset of regions; keep one in [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) and stdapi.ai fails over to it automatically
---
### :material-cog: Configuration
RAGFlow stores model credentials in its own database, added through **Settings → Model providers** in the web UI. Each entry pairs a *provider* (which decides the HTTP client and the URL layout) with an *instance* (a base URL and an API key). Three entries cover the whole pipeline:
| Stage | RAGFlow provider | Base URL | stdapi.ai route |
| --- | --- | --- | --- |
| Chat | `OpenAI-API-Compatible` | `https://YOUR_STDAPI_URL/v1` | [`/v1/chat/completions`](api_openai_chat_completions.md) |
| Rerank | `OpenAI-API-Compatible` | `https://YOUR_STDAPI_URL/cohere/v2` | [`/cohere/v2/rerank`](api_cohere_rerank.md) |
| Embedding | `OpenAI-API-Compatible` | `https://YOUR_STDAPI_URL/v1` | [`/v1/embeddings`](api_openai_embeddings.md) |
All three use the same stdapi.ai API key. Reranking needs its own entry because RAGFlow derives the request path from the instance's base URL, and reranking is served on the Cohere-compatible routes rather than the OpenAI ones.
Once the entries exist, set them as the tenant defaults under **Settings → Model providers → System model settings**, and RAGFlow uses them for every knowledge base and assistant.
#### :material-database-search: Document Engine
RAGFlow defaults to a self-hosted Elasticsearch as its document and vector store, which **cannot run on AWS Fargate**: Elasticsearch requires the host sysctl `vm.max_map_count=262144`, and Fargate exposes no way to set it. Setting `DOC_ENGINE=opensearch` and pointing RAGFlow at an **Amazon OpenSearch Service** domain removes that constraint entirely — and replaces a container you would have to operate with a managed service.
!!! example "Configuration"
Select the engine with an environment variable, and give it the domain in `service_conf.yaml`:
```bash
DOC_ENGINE=opensearch
```
```yaml
os:
hosts: 'https://YOUR_DOMAIN_ENDPOINT:443'
username: 'YOUR_MASTER_USER'
password: 'YOUR_MASTER_USER_PASSWORD'
```
The scheme and port are literals in RAGFlow's shipped template, so a domain endpoint alone is not enough — the file has to be replaced, which is what the sample below does.
Hybrid BM25 + vector retrieval requires OpenSearch 2.10 or later and the `cluster:admin/search/pipeline/put` privilege. RAGFlow creates the search pipeline itself on start-up and silently falls back to vector-only search if the call is refused, so it is worth confirming in the application log which mode you got.
!!! danger "The OpenSearch backend has no upstream CI coverage"
RAGFlow's own GitHub workflows exercise the `elasticsearch` and `infinity` engines only. Nothing upstream tests `opensearch`, so a working combination is a property of a specific RAGFlow version rather than a supported contract. Pin an exact image tag and re-validate before upgrading — the sample below pins `v0.26.4`.
---
### :material-alert-outline: Known Issues
RAGFlow `v0.26.4`'s behavior with the `opensearch` document engine carries a few gaps that are properties of the application, not of any particular deployment: **Agent Memory** is unavailable (RAGFlow only wires a message store for the Elasticsearch and Infinity engines), the Agent **"Code"** component is unavailable (it needs a Docker socket and gVisor that `DOC_ENGINE=opensearch` deployments typically don't have), and **Pagerank and the resume parser** branch on Elasticsearch upstream and do not apply here. See [Known Limitations](https://github.com/stdapi-ai/samples/blob/main/getting_started_ragflow/README.md#known-limitations) in the sample's README for the full list, with the code references that back each one.
## :material-rocket-launch: Deploy the Full Stack on AWS
The sample below is one worked example of a credible AWS deployment, not the only architecture that works. The gateway is a normal HTTP service, and RAGFlow and stdapi.ai can run anywhere you like — your own ECS or EKS cluster, EC2, another cloud, or a laptop.
### :material-sitemap: Architecture
The diagram below is the topology the [Terraform sample](#whats-included) builds: RAGFlow and the stdapi.ai gateway as separate ECS Fargate services in one VPC, with OpenSearch, Aurora and Valkey as the managed stores behind RAGFlow.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true, 'nodeSpacing': 20, 'rankSpacing': 40, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart TB
user["👤 Your users (browser)"]
subgraph public["Your VPC · public subnets"]
alb[" Application Load Balancer fronts RAGFlow only"]
end
subgraph private["Your VPC · private app subnets — no inbound route from the internet"]
ragflow[" RAGFlow ECS Fargate"]
stdapi[" stdapi.ai ECS Fargate"]
opensearch[" Amazon OpenSearch Service document + vector index"]
aurora[" Aurora PostgreSQL Serverless v2 · metadata"]
valkey[" ElastiCache Valkey TLS via loopback sidecar"]
egress[" NAT gateways one per Availability Zone"]
opensearch ~~~ valkey
end
subgraph regional["AWS service endpoints · your account, the regions you configure"]
bedrock[" Amazon Bedrock"]
s3[" Amazon S3 SSE-KMS"]
cw[" Amazon CloudWatch logs"]
end
user -->|"HTTPS · TLS 1.2+ (ACM certificate, when a custom domain is configured)"| alb
alb -->|"HTTP · private subnet"| ragflow
ragflow -->|"OpenAI + Cohere API · API key Cloud Map private DNS no public endpoint"| stdapi
ragflow -->|"HTTPS · basic auth certificate not verified"| opensearch
ragflow -->|"PostgreSQL · username + password"| aurora
ragflow -->|"TLS via loopback sidecar · auth token"| valkey
ragflow --> egress
stdapi --> egress
egress -->|"HTTPS · SigV4"| bedrock
egress -->|"HTTPS · SigV4"| s3
egress --> cw
```
Two things are worth reading off the picture. RAGFlow's documents and their vectors come to rest inside the account: source files land in the shared S3 bucket, and their parsed chunks and embeddings land in the OpenSearch domain — neither is reachable outside the VPC. And the gateway holds neither: it serves the `Embed` and `Rerank` calls RAGFlow sends it and returns the result, with no database of its own to persist anything in.
#### What Each AWS Service Does Here
| AWS service | Role in this integration | Where it is configured |
| --- | --- | --- |
| **Amazon ECS on AWS Fargate** | Runs RAGFlow (three containers: `main`, a TLS sidecar for Valkey, and a one-shot bootstrap) and the stdapi.ai gateway as separate services | `ragflow.tf`, Terraform sample |
| **Elastic Load Balancing** | The only public entry point; forwards only to RAGFlow; TLS with an ACM certificate and a Route 53 alias record when a custom domain is configured, otherwise plain HTTP on the load balancer's own DNS name | `alb.tf` |
| **AWS Cloud Map** | Private DNS name that lets RAGFlow reach the gateway without exposing it publicly | `main.tf` (`service_discovery_dns_name`) |
| **Amazon Bedrock** | Chat completions for synthesis, embeddings for indexing, reranking for retrieval — RAGFlow never calls it directly | [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) |
| **Amazon OpenSearch Service** | RAGFlow's document and vector index; a single-node VPC domain with fine-grained access control and enforced HTTPS | `opensearch.tf` |
| **Amazon Aurora PostgreSQL** | RAGFlow's metadata database (knowledge bases, chat sessions, users); Serverless v2, initialized over the RDS Data API | `postgres.tf` |
| **Amazon ElastiCache (Valkey)** | RAGFlow's cache and task queue (Redis Streams on database 1), reached through a `socat` sidecar that terminates TLS on loopback | `valkey.tf` |
| **Amazon S3** | Shared with the gateway's own bucket; RAGFlow stores documents and generated files under its own prefix, reached through its own ECS task role | [S3 storage](operations_compliance.md#s3-data-storage), `ragflow.tf` |
| **AWS Secrets Manager** | Holds the Aurora master username and password, read by the one-time provisioners that create RAGFlow's login role and grant it schema privileges | `postgres.tf` |
| **AWS KMS** | One customer-managed key (from the VPC module) encrypts OpenSearch, Aurora and Valkey storage; the gateway module's own key encrypts the shared S3 bucket | `network.tf`, `ragflow.tf` |
| **AWS IAM** | Separate task roles per service; RAGFlow's (`aws_iam_policy.ragflow`) is scoped to its own S3 prefix and its own KMS key actions — it carries no permission for Amazon Bedrock | `ragflow.tf`, [IAM permissions](operations_iam_permissions.md) |
| **Amazon CloudWatch** | Container logs for every task through the ECS `awslogs` driver, plus the gateway's structured request events | [Logging & Monitoring](operations_logging_monitoring.md) |
#### Security Measures in This Flow
- **Authentication** — RAGFlow calls the gateway with a stdapi.ai [API key](operations_authentication_security.md#api-key-authentication) that Terraform generates and injects into the container environment; the ALB's security group restricts inbound traffic to the deploying operator's current IP address.
- **Encryption in transit** — HTTPS from the browser to the ALB when a custom domain is configured (otherwise plain HTTP), private-VPC HTTP from the ALB to RAGFlow, and HTTPS with SigV4 from the gateway to Amazon Bedrock. The OpenSearch and Valkey hops carry their own caveats, covered in [Security Notes](#security-notes) below rather than repeated here.
- **Encryption at rest** — SSE-KMS on the shared S3 bucket, and KMS-encrypted storage for OpenSearch, Aurora and Valkey.
- **Least privilege** — RAGFlow's task role is scoped to its own S3 prefix and its own KMS key actions and carries no Bedrock permission; only the gateway's task role can call Amazon Bedrock.
- **Content policy** — a [Bedrock guardrail](operations_configuration.md#bedrock-guardrails) configured on the gateway applies to chat, embeddings and reranking alike, since all three reach Bedrock through the same deployment.
- **Data handling** — the gateway is stateless and holds request bodies in memory only for the duration of a call; the documents themselves persist in S3 and OpenSearch, inside the account, and no third party sits between RAGFlow's users and the models it calls.
---
### :material-cube-outline: What's Included
Deploy RAGFlow + stdapi.ai together, with the model providers already configured:
**📦 [stdapi-ai/samples/getting_started_ragflow](https://github.com/stdapi-ai/samples/tree/main/getting_started_ragflow)**
**What's included:**
- RAGFlow `v0.26.4` on ECS Fargate (4 vCPU / 16 GB), from the official `infiniflow/ragflow` image
- stdapi.ai gateway connected to Amazon Bedrock, registered as RAGFlow's chat, embedding, and rerank provider
- Amazon OpenSearch Service 2.19 as the document engine — a managed VPC domain with fine-grained access control, enforced HTTPS, and node-to-node and at-rest KMS encryption
- Aurora PostgreSQL Serverless v2 for metadata, initialized over the RDS Data API
- ElastiCache for Valkey with in-transit encryption and an AUTH token, reached through a `socat` TLS sidecar
- Amazon S3 for documents and generated files, accessed with the **ECS task role** — no IAM user and no static access keys
- A superuser account provisioned from a generated password, with self-registration closed
Every backing service is a managed AWS one; the only container in the task besides RAGFlow itself is the TLS sidecar and a short-lived bootstrap container.
**Deploy:**
```bash
git clone https://github.com/stdapi-ai/samples.git
cd samples/getting_started_ragflow/terraform
tofu init
tofu apply
```
Then read the URL and the generated superuser credentials from the Terraform outputs and sign in. There is no signup screen and no model-provider dialog to work through.
#### :material-account-check: Zero-Touch Model Configuration
Out of the box, RAGFlow is not usable until an administrator opens the UI and adds a model provider by hand — a knowledge base cannot even be created without an embedding model bound to the tenant. Since RAGFlow v0.26 those credentials live in the database rather than in `service_conf.yaml`, so there is no configuration file to preseed them from either.
The sample closes that gap with a bootstrap container that logs in as the generated superuser, creates the three provider instances from the table above, and binds them as the tenant's chat, embedding, and rerank defaults before the first login — see [How the Model Provider Gets There](https://github.com/stdapi-ai/samples/blob/main/getting_started_ragflow/README.md#how-the-model-provider-gets-there) in the sample's README for exactly how it's wired. The result is the point of the sample: **the first login lands on a working product**, with a knowledge base ready to accept its first document.
#### :material-shield-alert-outline: Security Notes
The sample encrypts every hop and keeps every secret out of plain environment variables, but two RAGFlow application behaviors are worth knowing before you put real documents in it: it does not validate the OpenSearch server certificate, and its Redis client cannot speak TLS on its own — reached instead through a `socat` TLS sidecar that terminates encryption on loopback. Both are covered in detail, with the exact mitigations, under [Security](https://github.com/stdapi-ai/samples/blob/main/getting_started_ragflow/README.md#security) in the sample's README.
---
### :material-gauge: What It Costs to Run
| Charge | Driver |
| --- | --- |
| stdapi.ai licence | $0.10 per gateway container-hour, metered through AWS Marketplace, with a 14-day free trial on the licence |
| ECS Fargate | The RAGFlow task (4 vCPU / 16 GB, three containers, one of which — the bootstrap — runs once per task start) and the gateway task, each billed for the vCPU and memory reserved while running |
| Load balancing and networking | One ALB fronting RAGFlow, plus the NAT gateways the private subnets egress through |
| Amazon OpenSearch Service | A standing charge: one `t3.small.search` data node plus 20 GiB of `gp3` storage, independent of query volume |
| Aurora PostgreSQL | A standing charge: the Serverless v2 instance stays provisioned even when its capacity scales down to the configured 0 ACU floor |
| ElastiCache Valkey | A standing per-node charge: one `cache.t4g.micro` node, no replicas |
| Model usage | Amazon Bedrock chat, embedding and rerank calls at AWS rates, billed to your account with no markup |
Read a model's price before sending it anything with [`GET /model_pricing`](api_model_pricing.md). Setting [`COST_TRACKING=true`](operations_cost_management.md#cost-tracking-real-time-aws-pricing) adds a per-request cost estimate to each usage entry — estimated from published AWS prices, not read back from your invoice. See [Sizing and cost](https://github.com/stdapi-ai/samples/blob/main/getting_started_ragflow/README.md#sizing-and-cost) in the sample's README for the knobs that move this bill and actual dollar figures at on-demand prices.
---
### :material-eye-outline: What to Watch
Every task's containers write their own logs to CloudWatch through the ECS `awslogs` driver; the gateway additionally writes one structured `request` event per call, carrying the request id, path, status code, `execution_time_ms`, the model that served it, and the token or search-unit counts AWS billed. Turning on [`CLOUDWATCH_METRICS`](operations_logging_monitoring.md#cloudwatch-metrics-emf) republishes those counts as EMF metrics in the `stdapi` namespace, dimensioned by `Model`.
For a retrieval pipeline the useful first question is the split between embedding, reranking and generation, since each stage in this sample resolves to a different model:
```sql
fields model_id, execution_time_ms
| filter type = "request" and ispresent(model_id)
| stats count(*) as calls, pct(execution_time_ms, 95) as p95_ms by model_id
| sort calls desc
```
Amazon Bedrock [model invocation logging](operations_compliance.md#amazon-bedrock-invocation-logging) is the AWS-side counterpart — off by default, and the record to enable when you need the prompts, chunks and completions themselves rather than metadata.
## :material-arrow-right: Next Steps
- :material-magnify: [**RAG Pipelines**](use_cases_rag.md) — Wire embeddings and reranking into your own framework
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy stdapi.ai to AWS with Terraform
- :material-puzzle: [**More Use Cases**](use_cases.md) — Explore other integrations and tools
- :material-api: [**API Overview**](api_overview.md) — Explore supported endpoints
---
## Home Assistant Voice
Source: https://stdapi.ai/md/use_cases_home_assistant.md
# :material-microphone-message: Home Assistant Voice Integration
Give Home Assistant's Assist voice pipeline speech-to-text and text-to-speech backed by Amazon Transcribe and Amazon Polly, through stdapi.ai's OpenAI-compatible audio routes.
## :material-information-outline: About Home Assistant Assist and Wyoming
**🔗 Links:** [Home Assistant Assist](https://www.home-assistant.io/voice_control/) | [wyoming-openai](https://github.com/roryeckel/wyoming_openai) | [Wyoming protocol](https://github.com/OHF-Voice/wyoming)
Home Assistant's Assist voice pipeline speaks [Wyoming](https://github.com/OHF-Voice/wyoming), a lightweight protocol for local voice satellites and speech services—not the OpenAI or Anthropic APIs directly. [wyoming-openai](https://github.com/roryeckel/wyoming_openai) is an open-source proxy that bridges Wyoming to any OpenAI-compatible speech-to-text and text-to-speech backend, which is what lets Assist reach stdapi.ai.
**What the proxy adds on top of the audio routes:**
- **Wyoming discovery** - Advertises configured speech-to-text models and text-to-speech voices to Assist
- **Streaming synthesis** - Speaks a response as it is generated, in overlapping chunks, rather than waiting for the whole sentence
- **Format translation** - Reassembles the response as raw PCM frames for Assist's audio pipeline
## :material-help-circle-outline: Why Home Assistant + stdapi.ai?
- :material-swap-horizontal: __No Cloud Voice Subscription__
Replace a cloud speech-to-text/text-to-speech subscription with Amazon Transcribe and Amazon Polly, billed at Bedrock/AWS rates.
- :material-lock: __No Third-Party Voice Vendor__
Spoken audio and transcripts are processed by Amazon Transcribe, Polly and Bedrock in the regions you enable, reached through your own deployment — no consumer voice-assistant vendor in the path.
- :material-home-automation: __Works with Your Existing Assist Setup__
Assist's speech-to-text and text-to-speech pipeline selection is unchanged—only the backend the proxy talks to is stdapi.ai.
- :material-currency-usd-off: __Pay-Per-Use Pricing__
No per-satellite or per-minute voice assistant fees. Pay only Amazon Transcribe and Amazon Polly rates for actual usage.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
ha[" Home Assistant Assist"] -->|Wyoming| proxy[" wyoming-openai"]
proxy -->|OpenAI API| stdapi[" stdapi.ai"]
stdapi --> transcribe[" Amazon Transcribe"]
stdapi --> polly[" Amazon Polly"]
```
## :material-connection: Connect Your Own Instance
Point any Home Assistant instance's Wyoming bridge at stdapi.ai — the deployment underneath doesn't matter to Assist.
### :material-check-circle: Prerequisites
!!! info "What You'll Need"
- ✓ **stdapi.ai deployed** - [See deployment guide](operations_getting_started.md) or [run locally with Docker](operations_getting_started_local.md)
- ✓ **Your stdapi.ai URL** - reachable from wherever the proxy runs, e.g. `https://api.example.com`
- ✓ **Your API key** - From Terraform output or configuration
- ✓ **Home Assistant** - With the Assist voice pipeline set up
- ✓ **A place to run wyoming-openai** - A container alongside Home Assistant, e.g. as a Home Assistant OS add-on or a standalone container
---
### :material-cog: Configuration
wyoming-openai is configured through environment variables, split into a speech-to-text half and a text-to-speech half. Point both at your stdapi.ai deployment.
!!! example "Environment Variables"
```bash
# Speech to text
STT_OPENAI_URL=https://YOUR_STDAPI_URL/v1
STT_OPENAI_KEY=YOUR_STDAPI_KEY
STT_MODELS=amazon.transcribe
# Text to speech
TTS_OPENAI_URL=https://YOUR_STDAPI_URL/v1
TTS_OPENAI_KEY=YOUR_STDAPI_KEY
TTS_MODELS=amazon.polly-neural
TTS_VOICES=alloy
# Backend selection
STT_BACKEND=OPENAI
TTS_BACKEND=OPENAI
```
The proxy calls `POST /v1/audio/transcriptions` (see [Audio Transcriptions API](api_openai_audio_transcriptions.md)) for speech to text and `POST /v1/audio/speech` (see [Audio Speech API](api_openai_audio_speech.md)) for text to speech, so `STT_MODELS` must be a speech-to-text-capable model and `TTS_MODELS` a text-to-speech-capable model from the correct family.
!!! tip "Pin the backend"
Left unset, wyoming-openai probes a few well-known self-hosted backends before falling back to a generic OpenAI-compatible one. Setting `STT_BACKEND=OPENAI` and `TTS_BACKEND=OPENAI` skips that probing and connects directly.
!!! tip "A cheaper speech-to-text model"
`STT_MODELS=amazon.nova-2-sonic-v1:0` transcribes through [Amazon Nova Sonic](api_openai_audio_transcriptions.md#amazon-nova-sonic) instead of Amazon Transcribe: the lowest-cost transcription available here, punctuated and in the language spoken. It answers `json` and `text` only — which is all the proxy asks for — and caps a recording at 10 minutes, well beyond any voice command. Keep `amazon.transcribe` if the same deployment also serves subtitles, timestamps or speaker labels.
#### :material-waveform: Streaming Speech to Text
Enables: recognizing a spoken command phrase by phrase, instead of after the whole utterance has been recorded.
!!! example "Environment Variables"
```bash
STT_STREAMING_MODELS=amazon.transcribe
```
Only the models listed there are called in streaming mode, which is what makes the proxy ask stdapi.ai for a [streamed transcription](api_openai_audio_transcriptions.md#streaming). The gateway returns each phrase as it is recognized whenever the request names the language to expect; if the proxy sends none, set [`AWS_TRANSCRIBE_STREAM_LANGUAGES`](operations_configuration.md#aws-transcribe-stream-languages) on stdapi.ai to the languages your satellites actually speak and those requests take the same fast path. Streamed transcription stages nothing, so it works on a deployment with no S3 bucket configured.
!!! note "This is the streaming option to use, not the realtime one"
stdapi.ai's [Realtime API](api_openai_realtime.md) serves speech-to-speech sessions, and a transcription-only session is requested through an [ephemeral client secret](api_openai_realtime.md#ephemeral-client-secrets) rather than on the socket — so a client that expects OpenAI's realtime *transcription* socket gets no transcript from it. Assist's pipeline is turn-based anyway: speech to text, then a conversation agent, then text to speech.
#### :material-volume-high: Streaming Text to Speech
Enables: speaking a response as it is generated, instead of waiting for the whole sentence to synthesize.
!!! example "Environment Variables"
```bash
TTS_STREAMING_MODELS=amazon.polly-neural
```
Naming the same model in both `TTS_MODELS` and `TTS_STREAMING_MODELS` puts its voice in the proxy's streaming program, so Assist can use it for both a plain synthesis request and a streamed one. The proxy splits a streamed reply into sentences and synthesizes several `/v1/audio/speech` calls concurrently, then replays the audio in the original order.
#### :material-tune-vertical: Voice Mapping
`TTS_VOICES` lists OpenAI-style voice names (`alloy`, `echo`, `fable`, and so on); stdapi.ai maps each one to an Amazon Polly voice of matching gender and language. List one entry per voice you want Assist to offer.
---
### :material-alert-outline: Known Issues
The proxy speaks the Wyoming protocol over its own TCP port, not HTTP—there is no `/health` endpoint to check readiness with a plain web request. Wait for a successful Wyoming `describe` exchange (or check the container logs) rather than polling an HTTP path.
## :material-rocket-launch: Deploy the Full Stack on AWS
The sample below is one worked example of a credible AWS deployment, not the only architecture that works. The gateway is a normal HTTP service, and Home Assistant, wyoming-openai and stdapi.ai itself can run anywhere you like — your own ECS or EKS cluster, EC2, another cloud, or a laptop.
### :material-sitemap: Architecture
The diagram below is the topology the [Terraform sample](#whats-included) builds: a public-facing Home Assistant behind an ALB, wyoming-openai as a sidecar in the same ECS task, and the stdapi.ai gateway as a separate, internally-reachable service in the same VPC.
```mermaid
%%{init: {'flowchart': {'htmlLabels': true, 'nodeSpacing': 20, 'rankSpacing': 40, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart TB
user["👤 Household members (browser · Assist microphone)"]
subgraph public["Your VPC · public subnets"]
alb[" Application Load Balancer HTTPS · ACM cert, when a custom domain is set"]
end
subgraph private["Your VPC · private app subnets — no inbound route from the internet"]
ha[" Home Assistant ECS Fargate task"]
wyoming[" wyoming-openai sidecar in the same task"]
efs[" Amazon EFS recorder DB · .storage · configuration.yaml encrypted · one task only"]
stdapi[" stdapi.ai ECS Fargate"]
egress[" NAT gateways one per Availability Zone"]
end
subgraph regional["AWS service endpoints · your account, the regions you configure"]
transcribe[" Amazon Transcribe"]
polly[" Amazon Polly"]
s3[" Amazon S3 SSE-KMS"]
cw[" Amazon CloudWatch container logs"]
transcribe ~~~ s3
polly ~~~ cw
end
user -->|"HTTPS · TLS 1.2+"| alb
alb -->|"HTTP · private subnet"| ha
ha -->|"Wyoming · TCP localhost, same task"| wyoming
wyoming -->|"OpenAI API · API key private DNS, no public endpoint"| stdapi
ha --> efs
ha -->|"HTTPS · SigV4 seeds configuration.yaml, first boot only"| s3
stdapi --> egress
egress -->|"HTTPS · SigV4"| transcribe
egress -->|"HTTPS · SigV4"| polly
egress -->|"HTTPS · SigV4 stages audio for non-streaming transcription"| s3
egress --> cw
```
The ALB is the only public address in the picture, and it forwards only to Home Assistant — stdapi.ai has no listener of its own and is reached exclusively through AWS Cloud Map private DNS from the wyoming-openai sidecar. A household's state (recorder database, `.storage`, `configuration.yaml`) comes to rest on the single EFS volume mounted into the Home Assistant task, never on the gateway; the gateway itself is stateless and only its egress path crosses the VPC boundary, over HTTPS with SigV4, to Amazon Transcribe and Amazon Polly.
#### What Each AWS Service Does Here
| AWS service | Role in this integration | Where it is configured |
| --- | --- | --- |
| **Amazon ECS on AWS Fargate** | Runs Home Assistant and wyoming-openai as containers in one task, and the stdapi.ai gateway as a separate service | Terraform sample (`home_assistant.tf`) |
| **Elastic Load Balancing** | Public entry point for Home Assistant; terminates TLS when a custom domain and certificate are configured | Terraform sample (`alb.tf`) |
| **AWS Cloud Map** | Private DNS name wyoming-openai uses to reach the gateway, with no public endpoint | Terraform sample (`service_discovery_dns_name`) |
| **Amazon Transcribe** | Speech-to-text behind `POST /v1/audio/transcriptions` | `STT_MODELS` (wyoming-openai) |
| **Amazon Polly** | Text-to-speech behind `POST /v1/audio/speech`, streamed as concurrent per-sentence calls | `TTS_MODELS` / `TTS_STREAMING_MODELS` (wyoming-openai) |
| **Amazon EFS** | Home Assistant's recorder database, `.storage`, and `configuration.yaml`; a second concurrent writer would corrupt it, so the task is pinned to exactly one | Terraform sample (`home_assistant.tf`, EFS mount point) |
| **Amazon S3** | Seeds `configuration.yaml` on first boot through a read-only S3 Files mount, and on the gateway side stages audio for non-streaming transcription | Terraform sample (config seed) / gateway module default bucket |
| **AWS KMS** | Customer-managed keys encrypting the EFS volume and the S3 buckets | ECS module and gateway module defaults |
| **Amazon CloudWatch** | Container logs for both ECS services | ECS module and gateway module defaults |
| **AWS IAM** | Separate task roles; the gateway's role grants only the Transcribe and Polly actions it invokes | [IAM permissions](operations_iam_permissions.md) |
#### Security Measures in This Flow
- **Authentication** — wyoming-openai calls the gateway with a stdapi.ai [API key](operations_authentication_security.md#api-key-authentication) that Terraform generates (`api_key_create = true`) and injects as `STT_OPENAI_KEY`/`TTS_OPENAI_KEY` container secrets; the sample's ALB security group additionally restricts inbound traffic to the deploying operator's own IP address.
- **Encryption in transit** — HTTPS from the browser to the ALB when a custom domain and certificate are configured; Wyoming stays inside the ECS task over localhost; HTTPS with SigV4 from the gateway to Amazon Transcribe and Amazon Polly.
- **Encryption at rest** — the EFS volume backing Home Assistant's state and both S3 buckets (config seed, gateway staging) use customer-managed KMS keys.
- **Least privilege** — the gateway's task role grants only the Transcribe and Polly actions it invokes; Home Assistant's task role carries none of them.
- **Content policy** — a [Bedrock guardrail](operations_configuration.md#bedrock-guardrails), if configured on the gateway, checks the text to synthesize as `INPUT` on `/v1/audio/speech` and the produced transcript as `OUTPUT` on `/v1/audio/transcriptions`, through the ApplyGuardrail API rather than a native chat-style integration.
- **Data handling** — the gateway holds request audio in memory, or briefly in its own S3 bucket when staging a non-streaming transcription job, and does not persist it; Home Assistant's own recordings and conversation history stay on the EFS volume in your account.
---
### :material-cube-outline: What's Included
Deploy Home Assistant, wyoming-openai, and stdapi.ai together on ECS Fargate:
**📦 [stdapi-ai/samples/getting_started_home_assistant](https://github.com/stdapi-ai/samples/tree/main/getting_started_home_assistant)**
**What's included:**
- Home Assistant and wyoming-openai in the same ECS Fargate task, talking over `localhost`
- stdapi.ai gateway connected to Amazon Bedrock, Amazon Transcribe, and Amazon Polly
- `configuration.yaml` seeded on first boot with the reverse-proxy trust settings Home Assistant needs behind an ALB
- Both container images pulled directly and anonymously from ghcr.io — no local build, no registry credential
- HTTPS-capable ALB on your own domain (needed for microphone access in the browser)
!!! warning "Demonstration sample, not a production Home Assistant deployment"
AWS Fargate has no route to your home network, so Zigbee/Z-Wave USB dongles, mDNS device discovery, and other LAN-only integrations do not work here. Use it to try Assist voice through Amazon Transcribe/Polly, or as a starting point for a self-hosted, cloud-reachable instance you administer through the web UI. If you already run Home Assistant at home, the sample's README covers deploying only the cloud-side pieces instead of moving Home Assistant itself.
**Deploy:**
```bash
git clone https://github.com/stdapi-ai/samples.git
cd samples/getting_started_home_assistant/terraform
tofu init
tofu apply
```
Three steps stay manual after `tofu apply`, for reasons specific to Home Assistant: creating the owner account through the onboarding wizard, adding the Wyoming integration (**Settings → Devices & Services**), and pointing an Assist pipeline at it. See the sample's README for the exact steps.
---
### :material-gauge: What It Costs to Run
| Charge | Driver |
| --- | --- |
| stdapi.ai licence | $0.10 per gateway container-hour, metered through AWS Marketplace, with a 14-day free trial on the licence |
| ECS Fargate | Two services — the Home Assistant + wyoming-openai task, pinned to exactly one, and the gateway, sized independently |
| Load balancing and networking | One ALB, plus the NAT gateways — one per Availability Zone — the private subnets egress through |
| Amazon EFS | Standing storage and throughput for the recorder database, `.storage`, and `configuration.yaml` |
| Amazon Polly | Billed per character of text synthesized, not per token |
| Amazon Transcribe | Billed per second of audio transcribed, not per token |
Read a model's price before you send anything to it with [`GET /model_pricing`](api_model_pricing.md). Setting [`COST_TRACKING=true`](operations_cost_management.md#cost-tracking-real-time-aws-pricing) additionally puts a per-request cost on each usage entry — estimated from published AWS prices, not read back from your invoice.
---
### :material-eye-outline: What to Watch
The gateway logs Polly usage — `input_characters`, always on the `request` event — and Transcribe usage — `input_seconds`, on `request` normally or on `request_stream` if you turn on `STT_STREAMING_MODELS` — with `execution_time_ms` on every entry. Turning on [`CLOUDWATCH_METRICS`](operations_logging_monitoring.md#cloudwatch-metrics-emf) republishes those counts as EMF metrics in the `stdapi` namespace, dimensioned by `Model`: `Count` for characters, `Seconds` for audio duration.
```sql
fields path, execution_time_ms
| filter type = "request" and (path = "/v1/audio/transcriptions" or path = "/v1/audio/speech")
| stats count(*) as calls, avg(execution_time_ms) as avg_ms, pct(execution_time_ms, 95) as p95_ms by path
| sort path
```
A rising p95 on either path is what a household notices as a slow turn, before it shows up in any cost report.
## :material-arrow-right: Next Steps
- :material-rocket-launch: [**Getting Started**](operations_getting_started.md) — Deploy stdapi.ai to AWS with Terraform
- :material-docker: [**Local Development**](operations_getting_started_local.md) — Run stdapi.ai locally with Docker
- :material-puzzle: [**More Use Cases**](use_cases.md) — Explore other integrations and tools
- :material-api: [**API Overview**](api_overview.md) — Explore supported endpoints
+ many more
Open WebUI"] --> stdapi["
stdapi.ai"]
n8n["
n8n"] --> stdapi
ide["
IDE + AI Assistant"] --> stdapi
openai_app["
Any OpenAI App"] --> stdapi
anthropic_app["
Any Anthropic App"] --> stdapi
stdapi --> bedrock["
Amazon Bedrock"]
bedrock --> claude["
Claude"]
bedrock --> qwen["
Qwen"]
bedrock --> mistral["
Mistral"]
bedrock --> stability["
Stability AI"]
bedrock --> more["✨ and more..."]
stdapi --> transcribe["
Amazon Transcribe"]
stdapi --> polly["
Amazon Polly"]
stdapi --> s3["
Amazon S3"]
```
---
## :material-api: API Compatibility
### 80+ endpoints, all three protocols, every one on an AWS service
Not a chat proxy with a few extras: the OpenAI, Anthropic and Cohere surfaces are served in full, each endpoint backed by an AWS service running in your account.
| What your application calls it for | AWS service behind it |
|-----------------------------------------------------------------------------------------|------------------------------------------------------------------|
| Chat completions, Responses, Messages, legacy completions, token counting | Amazon Bedrock Converse API · Bedrock Mantle |
| [Server-side conversations](api_openai_conversations.md) and stored responses | Amazon Bedrock Sessions · Bedrock Mantle |
| [Embeddings](api_cohere_embed.md) and [reranking](api_cohere_rerank.md) | Amazon Bedrock embedding and rerank models |
| [Vector stores and file search](api_openai_vector_stores.md) | Amazon S3 Vectors · Amazon Bedrock Knowledge Bases |
| [Batch inference](api_openai_batches.md) | Amazon Bedrock batch inference |
| Image generation, editing and variations | Amazon Bedrock image models |
| [Video generation](api_openai_videos.md) | Amazon Bedrock video models |
| Text-to-speech | Amazon Polly |
| Transcription and speech translation | Amazon Transcribe · Amazon Translate · Amazon Nova Sonic |
| [Live speech-to-speech](api_openai_realtime.md) | Amazon Bedrock |
| [Content moderation](api_openai_moderations.md) | Amazon Bedrock Guardrails · Amazon Comprehend |
| Files and multipart uploads | Amazon S3 |
| [Model discovery](api_search_models.md) and [pricing](api_model_pricing.md) | Amazon Bedrock · AWS Price List |
Anthropic and Cohere routes live under `/anthropic` and `/cohere`, so all three protocols are served side by side without colliding on `/v1` — and either prefix can become the path your clients already send ([Anthropic](operations_configuration.md#anthropic-routes-prefix), [Cohere](operations_configuration.md#cohere-routes-prefix)).
[:octicons-arrow-right-24: Every endpoint, with its parameters](api_overview.md)
### Parameter Coverage
stdapi.ai maps as many parameters as possible to Bedrock equivalents — across all routes, not just chat:
- **Generation controls** — `temperature`, `max_tokens`, `top_p`, `top_k`, `stop`, `seed`, `frequency_penalty`, `presence_penalty`, `logit_bias`, `top_logprobs`, streaming via SSE, token usage reporting
- **Reasoning** — `reasoning_effort` (none/minimal/low/medium/high/xhigh), `enable_thinking`, `thinking_budget`
- **Tool / function calling** — Full OpenAI and Anthropic schemas, parallel tool calls, tool choice modes
- **All content types** — System, developer, user, assistant, and tool roles; text, image, audio, video, and document content
- **Response formats** — JSON object, JSON schema, streaming chunks, `reasoning_content`, `annotations`
- **Model-specific extras** — Any parameter beyond the standard API via `extra_body` or top-level request fields
!!! note "Bedrock & model differences"
Not every parameter maps identically across all models. Check the [API documentation](api_openai_chat_completions.md) for details.
---
## :material-brain: 100+ Models Across 20+ Providers
Access every model available on Amazon Bedrock through a single, consistent API — including OpenAI GPT, xAI Grok, and [other frontier models](#bedrock-mantle-models). [Browse the full list](models.md).
ALB"]
anthropic["
CloudWatch"]
```
### Get Your Credentials
```bash
terraform output -raw api_key
terraform output api_endpoint
terraform output docs_url
```
!!! tip "Ready-to-use Terraform example on GitHub"
:material-map-marker: **Single region** — [getting_started_production](https://github.com/stdapi-ai/samples/tree/main/getting_started_production)
!!! tip "Optional: expose the API as MCP tools"
The [MCP server](features.md#mcp-model-context-protocol) is off by default. Set `enable_mcp_streamable_http = true` on the Terraform module and every endpoint becomes a named MCP tool at `
AWS Marketplace ECR"] --> ecs
ecs --> bedrock["
**Alibaba Cloud** (China 🇨🇳) — Qwen models
- 
**Amazon** (United States 🇺🇸) — Nova and Titan models
- 
**Cohere** (Canada 🇨🇦) — Command and Embed models
- 
**DeepSeek** (China 🇨🇳) — DeepSeek models
- 
**Meta** (United States 🇺🇸) — Llama models
- 
**MiniMax** (China 🇨🇳) — MiniMax models
- 
**Moonshot AI** (China 🇨🇳) — Kimi models
- 
**Writer** (United States 🇺🇸) — Palmyra models
---
## :material-microphone: AWS AI Services
Amazon Polly, Transcribe, Comprehend, and Translate each run in an independently configurable region. By default all four treat every `AWS_BEDROCK_REGIONS` entry as a candidate and fail over between them, so pointing `AWS_BEDROCK_REGIONS` to your target geography is usually sufficient.
### AWS AI Service Improvement Opt-Out
!!! warning "Action required before processing sensitive data with Polly, Transcribe, Comprehend, or Translate"
Unlike Amazon Bedrock, AWS may use content processed by these four AI services to improve service quality **by default**. This means audio recordings, transcription text, translated content, and language detection inputs could be used for model training unless you opt out.
Opt out by configuring an [AI services opt-out policy](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_ai-opt-out.html) at the AWS Organizations level. This is a **one-time action** in the AWS Console. It applies to your entire account immediately and AWS also deletes previously stored content:
> *"When you opt out of content use by an AWS AI service, that service deletes all of the associated historical content that was shared with AWS before you set the option."*
You do not need to opt out for Amazon Bedrock — Bedrock does not use prompts or completions to improve models, and does not share them with model providers unless you enable `provider_data_share`. Retention itself is governed separately by your [data retention mode](#data-retention-modes), and by the model-specific [abuse-detection](#abuse-detection) rules that apply to a small number of models.
---
## :material-shield-check: Content Safety
Content filtering is a control you configure, not one the gateway supplies of its own. It enforces an [Amazon Bedrock guardrail](operations_configuration.md#bedrock-guardrails) that you create and own, evaluated by Bedrock in your account and in your configured regions. With no guardrail configured, nothing is filtered or refused on inference — the [Moderations API](api_openai_moderations.md) still classifies content on demand, but classification is a report, not a block.
There is no separate stdapi.ai content policy, model evaluation, or review stage layered on top. Every behavior below is the enforcement of the policy **you** define on the guardrail, plus the classification backends the Moderations API exposes.
### Guardrail Coverage
A guardrail set through [`AWS_BEDROCK_GUARDRAIL_IDENTIFIER` and `AWS_BEDROCK_GUARDRAIL_VERSION`](operations_configuration.md#aws-bedrock-guardrail-identifier) — or carried by a [model alias](operations_configuration.md#model-aliases-configuration), which overrides the global one for the requests naming that alias — applies to **every route**:
| Routes | Direction checked |
|--------------------------------------------------------------|--------------------------------------------------------------------------------------|
| Chat Completions, Responses, Completions, Anthropic Messages | **Input and output**, natively inside the Bedrock invocation |
| Embeddings, Rerank, Images Generations / Edits, Videos, Audio Speech | **Input** — the client-supplied text, checked before it reaches the backend |
| Audio Transcriptions, Audio Translations | **Output** — the transcript or translated text, checked before it is returned |
| Realtime | **Input and output**, per turn — written items before the model sees them, transcribed caller speech, and each completed answer |
Routes whose AWS backend has no native guardrail mechanism call the ApplyGuardrail API explicitly; see [Route Coverage](operations_configuration.md#route-coverage) for the mechanism used per route, and [Bedrock Guardrails](operations_iam_permissions.md#bedrock-guardrails-optional) for the IAM permission it requires.
Clients cannot weaken the policy: [`AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE`](operations_configuration.md#aws-bedrock-allow-guardrail-override) is `false` by default, so guardrail request headers and request-body guardrail configuration are ignored while a guardrail is configured. It is auto-enabled at startup only when no guardrail is configured anywhere — there is then no policy to bypass.
!!! warning "Two places a configured guardrail does not reach"
- **Bedrock Mantle-served models** — guardrails are a `bedrock-runtime` feature and do not apply to requests served by the Mantle endpoint. Startup emits a warning counting the affected models; set [`AWS_BEDROCK_MANTLE_ENABLED=false`](operations_configuration.md#bedrock-mantle-enabled) to close the gap. A **model alias** that carries a guardrail while pointing at a Mantle-served model is fatal at startup rather than silently unguarded.
- **Batch inference** — the Bedrock batch API cannot carry a guardrail. A batched request that a configured guardrail would apply to is **refused**, not run unguarded; send those requests without batching.
Both are fail-closed by design: the failure mode is a refused request or a refused startup, never content served past a policy the operator believes is active.
### Intervention Behavior
What a client sees when the guardrail intervenes depends on the mechanism the route uses:
| Route family | Result of a blocking intervention |
|--------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
| ApplyGuardrail-enforced routes (embeddings, rerank, images, videos, speech, transcription, translation, moderations) | HTTP **400** with error code `content_filter`, carrying the guardrail's own configured blocked messaging |
| Chat Completions, Responses, Completions | The response is returned and reports `content_filter` as its finish (or incomplete) reason |
| Anthropic Messages | The response is returned with stop reason `refusal` |
| Realtime | A terminal `error` event and close code `3000` — the session ends |
A **masking-only** intervention (the guardrail's sensitive-information policy anonymizing rather than blocking) substitutes the masked text instead of failing: masked input reaches the backend model already masked, and a masked transcript or translation is returned on the plain `json` and `text` formats. Response formats that cannot represent masked text — `srt`, `vtt`, `verbose_json`, `diarized_json` — fail with the same `content_filter` error rather than return the unmasked content.
On the Realtime API the check cannot precede delivery in one direction: the model's speech is streamed while it is generated and its transcript is only complete once the answer is over, so a blocked answer may already have been partly heard when the session ends.
!!! info "Guardrail evaluation is a billed AWS operation"
AWS charges for the guardrail on every route it applies to. Only the ApplyGuardrail-enforced routes report the units consumed, which appear per guardrail policy in [usage logs and cost tracking](operations_logging_monitoring.md). Native chat routes are billed by AWS but return no unit counts, so tracked cost on them is lower than the AWS bill by the guardrail's share.
### Content Classification
The [Moderations API](api_openai_moderations.md) (`POST /v1/moderations`) classifies text and images against three backends, all inside your AWS account:
| Backend | Requires | Inputs |
|--------------------------------------------------------------------------|-------------------------------------------------------|------------------|
| Amazon Bedrock Guardrails | A configured guardrail resource | Text and images |
| Inline guardrail checks (`InvokeGuardrailChecks`) | **No guardrail resource** — only a Bedrock region offering the operation | Text only |
| Amazon Comprehend toxicity detection | Nothing — always available | Text only |
Because the last two need no setup, `/v1/moderations` works on any deployment: with no guardrail configured it resolves to inline guardrail checks where a configured region offers them, then falls back to Comprehend. See [Model Support](api_openai_moderations.md#model-support) for the selection rules and category mapping, and [Comprehend Moderation](operations_iam_permissions.md#comprehend-moderation) for its permission.
The same classification is available inline on generation: the `moderation` request parameter of the Chat Completions and Responses APIs reports how the guardrail assessed the input and the output of that request. It requires a guardrail resource, and is rejected on Mantle-served models.
### Personal Data in Content
Personal data is handled in exactly two places, both of which you switch on deliberately:
- **The guardrail's sensitive-information policy** — the PII entity types and regular expressions you configure on the guardrail are masked or blocked wherever that guardrail is checked, in both directions, following the intervention behavior above.
- **Amazon Transcribe PII redaction** — a client may request `ContentRedaction` on a transcription; only the single-output `redacted` mode is accepted, so no unredacted copy is produced. See [Audio Transcriptions](api_openai_audio_transcriptions.md).
Just as importantly, there is no personal-data handling anywhere else, and none should be assumed:
- The gateway performs **no detection, classification or redaction of its own** — everything above is AWS-side policy evaluation.
- Only **text** is submitted for checking on inference routes. Images, audio and documents passed to a model are not scanned for personal data; the Moderations API is the one surface that submits images to a guardrail.
- Objects held temporarily in your S3 buckets are **not scanned or redacted** — see [S3 Data Storage](#s3-data-storage) for their lifecycle.
- Application logs are not filtered for personal data; prompt and response content simply never reaches them unless you enable it — see [Logging](#logging).
---
## :material-bucket-outline: S3 Data Storage
S3 is used as temporary storage for multimodal content (images, PDFs, audio files) passed to or returned from Bedrock and the AI services. Data is stored only in buckets you own and configure.
- **Primary bucket** (`AWS_S3_BUCKET`) — must reside in the same AWS region as the first entry in `AWS_BEDROCK_REGIONS`.
- **Regional buckets** (`AWS_S3_REGIONAL_BUCKETS`) — for multi-region deployments, one bucket per Bedrock region ensures each region reads and writes data locally. When using the Terraform module, these are created automatically.
- **Lifecycle policies** — the Terraform module applies a 1-day lifecycle policy to the temporary prefix as a failsafe. The application itself removes temporary files as soon as the operation completes, so files rarely remain beyond the duration of a single request.
S3 stores data within the AWS region where each bucket is created. Data does not leave that region unless you explicitly configure replication.
---
## :material-text-box-check-outline: Logging
### Application Logging
By default, stdapi.ai logs only request metadata — HTTP method, path, status code, execution time, and model identifier. **Prompt and response content are never written to logs** unless explicitly enabled.
Setting `LOG_REQUEST_PARAMS=true` enables full request/response payload logging. This is **disabled by default** and should remain disabled in production environments handling sensitive data. See [Logging & Monitoring](operations_logging_monitoring.md) for details.
### Amazon Bedrock Invocation Logging
Bedrock optionally supports invocation logging — recording model inputs and outputs to S3 or CloudWatch Logs. This feature is **disabled by default** on AWS. When enabled:
- **S3 destination**: objects are encrypted using SSE-KMS (CMK supported via key policy).
- **CloudWatch destination**: log group can be encrypted with a KMS CMK.
- Logging scope is configurable: metadata only, or including full prompt and completion content.
See [Amazon Bedrock invocation logging](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html) for configuration details.
---
## :material-key-outline: KMS Encryption
All data stored by stdapi.ai is encrypted at rest. The Terraform module automatically creates a **Customer Managed Key (CMK)** with a restrictive key policy and automatic annual rotation enabled.
For higher compliance needs, AWS KMS supports additional controls: custom key policies, crypto-shredding, multi-region keys, and [CloudHSM-backed custom key stores](https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) for FIPS 140-3 Level 3 hardware-validated key storage.
!!! tip "Bring your own CMK"
To use an existing CMK, create the S3 bucket and CloudWatch log groups outside the Terraform module and pass them via `aws_s3_bucket` and related parameters. See [Advanced Deployment](operations_deploy_advanced.md#integration-with-existing-infrastructure).
Two stores are encrypted under a key you name rather than the bucket's own, so their key policy can be scoped to this workload instead of to everything the bucket holds:
| Setting | Encrypts | Default when unset |
|-------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------|---------------------------------------------|
| [`AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-transcribe-output-encryption-key-arn) | Each transcription job's output, with the job's request identifiers as the KMS encryption context | The bucket's own default encryption |
| [`AWS_BEDROCK_SESSION_ENCRYPTION_KEY_ARN`](operations_configuration.md#aws-bedrock-session-encryption-key-arn) | The Amazon Bedrock session storage behind stored responses, chat completions and conversations | The AWS-managed key |
---
## :material-shield-star: AWS Security Hub, GuardDuty & DNS Firewall Integration
Security Hub Foundational Security Best Practices control mapping, GuardDuty Runtime Monitoring, and Route 53 Resolver DNS Firewall are infrastructure-level security controls, covered on the [Authentication & Security](operations_authentication_security.md) page — see [AWS Security Hub, GuardDuty & DNS Firewall Integration](operations_authentication_security.md#aws-security-hub-guardduty-dns-firewall-integration) for the full module compliance mapping and configuration details.
---
## :material-cog-outline: Compliance Configuration Reference
| Variable | Purpose | Compliance relevance |
|---------------------------------------------|---------------------------------------------------------------|-------------------------------------------------------|
| `AWS_BEDROCK_REGIONS` | Ordered list of Bedrock regions | Restrict model inference to a specific geography |
| `AWS_BEDROCK_CROSS_REGION_INFERENCE` | Enable Bedrock cross-region routing within configured regions | Set `false` to restrict inference to a single region |
| `AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL` | Allow Bedrock to route globally outside configured regions | Set `false` to enforce geographic boundaries |
| `AWS_S3_BUCKET` | Primary S3 bucket | Must be in your target region |
| `AWS_S3_REGIONAL_BUCKETS` | Per-region S3 buckets for multi-region setups | Prevent cross-region data transfer |
| `AWS_POLLY_REGION` | Polly service region | Pin to your target geography |
| `AWS_TRANSCRIBE_REGION` | Transcribe service region | Pin to your target geography |
| `AWS_TRANSCRIBE_S3_BUCKET` | S3 bucket for Transcribe audio files | Must be in the same region as `AWS_TRANSCRIBE_REGION` |
| `AWS_COMPREHEND_REGION` | Comprehend service region | Pin to your target geography |
| `AWS_TRANSLATE_REGION` | Translate service region | Pin to your target geography |
| `AWS_BEDROCK_GUARDRAIL_IDENTIFIER` | Guardrail applied to every route | Enforce your content and sensitive-information policy |
| `AWS_BEDROCK_GUARDRAIL_VERSION` | Version of that guardrail | Pin the exact policy version in force |
| `AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE` | Allow clients to override the configured guardrail | Keep `false` so no client can bypass the policy |
!!! warning "Service regions are unpinned by default"
Left unset, `AWS_POLLY_REGION`, `AWS_TRANSCRIBE_REGION`, `AWS_COMPREHEND_REGION` and `AWS_TRANSLATE_REGION` make every [`AWS_BEDROCK_REGIONS`](operations_configuration.md#aws-bedrock-regions) entry a candidate, with automatic failover between them. Restrict `AWS_BEDROCK_REGIONS` to compliant regions, or pin each service explicitly, so no request can be served outside your target geography.
---
## :material-map-marker-multiple: Region-Specific Configuration
=== ":fontawesome-solid-earth-europe: EU / GDPR"
To ensure all data processing stays within the European Union:
```bash
# Restrict Bedrock to EU regions
export AWS_BEDROCK_REGIONS=eu-west-1,eu-west-3,eu-central-1,eu-north-1
# Prevent Bedrock from using global profiles that could route outside the EU.
# Geography-pinned eu.* profiles are still used for failover within the EU.
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false
# S3 bucket in an EU region (must match the first entry in AWS_BEDROCK_REGIONS)
export AWS_S3_BUCKET=my-stdapi-eu-bucket
```
!!! tip "Ready-to-use Terraform example"
:fontawesome-solid-earth-europe: **Multi-region GDPR (EU)** — [getting_started_production_gdpr](https://github.com/stdapi-ai/samples/tree/main/getting_started_production_gdpr)
=== ":fontawesome-solid-earth-americas: United States"
To restrict all data processing to US AWS regions:
```bash
# Restrict Bedrock to US regions
export AWS_BEDROCK_REGIONS=us-east-1,us-west-2,us-east-2
# Prevent Bedrock from using global profiles that could route outside the US.
# Geography-pinned us.* profiles are still used for failover within the US.
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false
# S3 bucket in a US region
export AWS_S3_BUCKET=my-stdapi-us-bucket
```
!!! tip "Ready-to-use Terraform example"
:fontawesome-solid-earth-americas: **Multi-region US** — [getting_started_production_us](https://github.com/stdapi-ai/samples/tree/main/getting_started_production_us)
=== ":fontawesome-solid-earth-asia: Asia Pacific"
To restrict all data processing to Asia Pacific AWS regions:
```bash
# Restrict Bedrock to APAC regions
export AWS_BEDROCK_REGIONS=ap-northeast-1,ap-southeast-1,ap-southeast-2
# Prevent Bedrock from using global profiles that could route outside APAC.
# Geography-pinned apac.* profiles are still used for failover within APAC.
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false
# S3 bucket in an APAC region
export AWS_S3_BUCKET=my-stdapi-apac-bucket
```
---
## :material-gavel: US Law and Cloud Provider Obligations
AWS is incorporated and headquartered in the United States. Regardless of where your data is stored, AWS as a legal entity is subject to US law — including statutes with explicit extraterritorial reach. This section describes the statutes and the AWS commitments and technical controls that exist around them; it is not legal advice, and how these statutes apply to your data is a question for your own counsel.
### CLOUD Act
The **Clarifying Lawful Overseas Use of Data (CLOUD) Act, 2018** requires US-based cloud providers to produce customer data in response to a valid US law enforcement warrant or court order, **even when the data is stored outside the United States**. Content data requires a court-issued warrant establishing probable cause — the highest legal standard under US criminal procedure.
Key limits and AWS commitments ([AWS CLOUD Act page](https://aws.amazon.com/compliance/cloud-act/)):
- **Challenge mechanism**: AWS can file a motion to quash or modify requests that would require violating another country's laws, and preserves this right contractually.
- **Customer notification**: AWS's policy is to notify customers of government data requests unless prohibited by a court order (gag order).
- **Zero cross-border content disclosures**: AWS publicly reports that it has not disclosed enterprise or government content stored outside the US to the US government since it began reporting this metric in 2020.
- **Administrative access controls**: The [AWS Nitro System](https://aws.amazon.com/ec2/nitro/) restricts administrative access to customer workloads — including by AWS employees — a design validated by independent security audit (NCC Group).
### FISA Section 702
Section 702 of the **Foreign Intelligence Surveillance Act** authorizes US intelligence agencies to compel US-based electronic communication service providers to deliver communications of non-US persons located outside the United States — without a per-target warrant. This authority was reauthorized in April 2024 and has no provider challenge mechanism equivalent to the CLOUD Act.
The encryption controls described on this page apply here as they do everywhere else: data stored by stdapi.ai is encrypted with a **customer-managed KMS key (CMK)** whose policy and grants you control, and whose usage is visible to you in CloudTrail (see [KMS Encryption](#kms-encryption)). Contractual protections, likewise, cannot override applicable US law. Whether a given combination of controls satisfies your obligations is a legal assessment to make with your own counsel.
### Transfer Frameworks
**EU — Data Privacy Framework and SCCs.** The European Commission adopted the EU-US Data Privacy Framework (DPF) in July 2023, requiring proportionality constraints on US intelligence collection and establishing a Data Protection Review Court (DPRC) as a redress mechanism for EU individuals. AWS is certified under the DPF ([AWS DPF page](https://aws.amazon.com/compliance/eu-us-data-privacy-framework/)). A legal challenge by noyb is pending before the CJEU but has not resulted in invalidation as of this writing.
AWS also incorporates **Standard Contractual Clauses (SCCs)** into its Data Processing Addendum by default — providing a contractual transfer mechanism independent of the DPF. SCCs cannot override applicable US law, but they contractually bind AWS to notify customers of legal demands and challenge requests where legally permissible. Maintaining SCCs alongside the DPF is the recommended posture.
**APAC and other regions.** There is no unified bilateral transfer framework equivalent to the EU-US DPF for APAC jurisdictions. Enterprises typically rely on AWS contractual commitments, customer-managed encryption, and jurisdiction-specific legal analysis. AWS maintains local certifications (IRAP for Australia, ISMAP for Japan, K-ISMS for South Korea, MTCS for Singapore) relevant to respective regulatory environments.
!!! info "Recommended posture"
Deploy in AWS regions within your target geography, encrypt stored data with customer-managed KMS keys whose policy you control, and maintain SCCs as a transfer mechanism independent of DPF validity. This combines a residency control, an encryption control and a contractual transfer mechanism; assess with your own counsel whether it meets your obligations.
---
## :material-lightbulb-outline: Best Practices for High-Compliance Deployments
- :material-check: **Restrict all region settings to compliant regions** — `AWS_BEDROCK_REGIONS` is the primary control; all services default to it, and any optional per-service override must stay within your target geography.
- :material-check: **Set `AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false`** — stdapi.ai will then use geography-pinned inference profiles (`us.*`, `eu.*`, `apac.*`), whose destination region list AWS commits to keeping within the named geography.
- :material-check: **Opt out of AWS AI service improvement** via an AWS Organizations policy — one-time console action covering Polly, Transcribe, Comprehend, and Translate.
- :material-check: **Configure an Amazon Bedrock guardrail and leave `AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE` at `false`** — the policy then applies to every route in both directions and no client can weaken it. If you rely on it, also set `AWS_BEDROCK_MANTLE_ENABLED=false`, since guardrails do not apply to Mantle-served models. See [Content Safety](#content-safety).
- :material-check: **Use a CMK with a restrictive key policy** — the Terraform module creates one by default, so the key policy, its grants and its CloudTrail usage records are yours to control. For stricter control: bring your own key, limit decrypt to the ECS task role, enable automatic rotation, crypto-shredding for right-to-erasure, or a CloudHSM-backed store for FIPS 140-3 Level 3.
- :material-check: **Confirm `LOG_REQUEST_PARAMS` is disabled** (the default) in production — prompt and response content are then kept out of application logs entirely. If Bedrock invocation logging is enabled for audit purposes, configure a KMS CMK for the S3 or CloudWatch destination.
- :material-check: **Use AWS PrivateLink** for Bedrock and S3 to keep service calls off the public internet, and **enable TLS 1.3 on the ALB** to activate post-quantum hybrid key exchange.
- :material-check: **Enable AWS CloudTrail** in all configured regions to monitor API activity across Bedrock, S3, KMS, and AI services.
- :material-check: **Enable AWS Security Hub and GuardDuty** on the account — deploy via the Terraform module and set `compliance_vpc_endpoints_enabled=true`, `guardduty_vpc_endpoint_enabled=true`, and `dns_firewall_enabled=true` (dedicated VPC only) to close the remaining FSBP gaps, support GuardDuty Runtime Monitoring, and block DNS resolution of known-malicious domains. See [AWS Security Hub, GuardDuty & DNS Firewall Integration](operations_authentication_security.md#aws-security-hub-guardduty-dns-firewall-integration).
- :material-check: **EU users: maintain SCCs alongside the DPF** — SCCs (included by default in the AWS Data Processing Addendum) provide a transfer mechanism independent of DPF validity and contractually bind AWS to challenge requests and notify you.
---
## :material-domain: Regulated Industries
### :material-hospital-box: Healthcare (HIPAA)
**Amazon Bedrock is a HIPAA-eligible service** and is covered under the [AWS Business Associate Agreement (BAA)](https://aws.amazon.com/compliance/hipaa-eligible-services-reference/). For the current list of all HIPAA-eligible AWS services, see the [AWS HIPAA Eligible Services Reference](https://aws.amazon.com/compliance/hipaa-eligible-services-reference/).
**There is no stdapi.ai BAA to sign.** stdapi.ai runs entirely within your AWS account — you are the data controller. Your existing AWS BAA with Amazon covers the underlying services. No separate agreement with stdapi.ai is required.
Recommended configuration for PHI workloads:
- Configure the [AI services opt-out policy](#aws-ai-service-improvement-opt-out) before processing any PHI through Polly, Transcribe, Comprehend, or Translate
- Restrict `AWS_BEDROCK_REGIONS` to regions covered by your BAA geography requirements
- Enable Bedrock invocation logging to a KMS-encrypted S3 bucket if audit trails are required by your compliance programme
- Confirm `LOG_REQUEST_PARAMS` is disabled (the default) so PHI never appears in application logs
### :material-bank: Financial Services (SOC 2, PCI-DSS, GDPR)
Amazon Bedrock carries **SOC 1/2/3**, **PCI-DSS**, and **GDPR** certifications. See [AWS Compliance Programs](https://aws.amazon.com/compliance/programs/) for the current list.
stdapi.ai itself does not process or store payment card data. Your PCI-DSS scoping decision is determined by what data your application sends to the gateway — not by the gateway itself.
For the legal context around cross-border data access under US law (CLOUD Act, FISA 702), the AWS commitments that apply, and the region and encryption controls available to you, see [US Law and Cloud Provider Obligations](#us-law-and-cloud-provider-obligations).
### :material-office-building: Government / Public Sector (FedRAMP)
Amazon Bedrock has received **FedRAMP Moderate and High authorization**. See the [AWS FedRAMP page](https://aws.amazon.com/compliance/fedramp/) and the [FedRAMP Marketplace](https://marketplace.fedramp.gov/) for current authorization status.
Restrict `AWS_BEDROCK_REGIONS` to US regions and set `AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false` so that inference is served only from US regions and US geography-pinned inference profiles.
### :material-briefcase: Legal & Professional Services
Attorneys, consultants, accountants, and other professionals bound by confidentiality obligations cannot transmit client materials to third-party AI services. stdapi.ai runs inference on the AWS services and regions you enable — no third party sits between your users and your models, and the gateway contacts no vendor endpoint of its own choosing. The outbound paths that can exist are driven by your configuration or your own clients; see [Application Data Flow](#application-data-flow). This makes it a workable basis for AI-assisted document review, contract analysis, and research where client confidentiality is non-negotiable.
---
## :material-arrow-right: Next Steps
Aurora PostgreSQL
ElastiCache Valkey
NAT gateways · one per AZ
LobeHub"] --> stdapi["
Amazon Comprehend"]
s3["
AWS WAF (optional)
Your Python App
Docling Serve
Amazon SQS
RAGFlow"] --> stdapi["
Amazon OpenSearch Service
Home Assistant
wyoming-openai"]
proxy -->|OpenAI API| stdapi["
Amazon EFS