Troubleshooting¶

Your AWS profile does not have sufficient permissions. The stdapi.ai Terraform module provisions IAM roles, KMS keys, ECS, ALB, Route53 records, an optional WAF, and (for some samples) RDS and ElastiCache.

• Use an administrator-level AWS profile for the evaluation deployment.
• Recommended: deploy into a sandbox/non-production AWS account first, then replicate into your target account with scoped-down principals once validated.
• Verify your active identity: aws sts get-caller-identity.

The deployment serves the API from a name you control, so the module needs a public Route 53 hosted zone, in the same AWS account, that is authoritative for that name. It looks the zone up from alb_domain_name by taking the most specific parent zone it can find; when nothing matches, the plan fails before anything is created.

• Name the zone explicitly when it is not the immediate parent of the domain — api.eu.example.com served from the example.com zone needs -var alb_route53_zone_name=example.com.
• Confirm the zone exists in this account and is public:

  aws route53 list-hosted-zones-by-name --dns-name example.com \
    --query 'HostedZones[].{Name:Name,Private:Config.PrivateZone,Id:Id}'
  

  A private zone never matches: the module validates the name on the public internet.
• Wrong account or wrong profile produces the same message — check with aws sts get-caller-identity before re-applying.
• Bringing your own certificate and DNS instead: pass alb_certificate_arn and leave alb_domain_name unset, then point your own record at the ALB. See Advanced Deployment.

ACM issues the certificate only once the DNS validation record it asked for resolves on the public internet. The module writes that record into the zone it found, then waits. A few minutes is normal; longer than that means the record is not being answered, and because the resource waits rather than fails, it reads as a hang.

• Check what ACM is still waiting for:

  aws acm describe-certificate --certificate-arn <arn> \
    --query 'Certificate.DomainValidationOptions'
  

  ValidationStatus: PENDING_VALIDATION with a ResourceRecord names the exact record that must resolve.
• Resolve that record from outside AWS: dig +short <name> CNAME. Nothing back means the record is in a zone that does not serve the domain publicly.
• The usual cause is a zone found by name that is not the authoritative one — a duplicate zone in this account, or a registrar delegating elsewhere. Compare the registrar's name servers with the zone's: dig +short NS example.com against the zone's NS record set, and pass the right zone with -var alb_route53_zone_name=....
• Cancelling the apply leaves the certificate PENDING_VALIDATION; it is picked up again by the next apply once DNS answers, and no other resource depends on it having been re-created.

Terraform does not wait for the service to stabilise, so a successful apply means the resources exist, not that a task is running. The ALB returns 503 Service Unavailable until tasks pass health checks — either because the service is still coming up, or because no task can start at all.

• Wait 2–3 minutes after terraform apply completes.
• Check ECS service status: aws ecs describe-services --cluster <cluster> --services <service>. A CannotPullContainerError or an AccessDeniedException on the Marketplace ECR repository in its events means the AWS Marketplace subscription was never accepted for this account: the tasks cannot pull the licensed image and will never become healthy. Subscribe — it is a one-time legal acceptance per account with no API, so Terraform cannot do it for you — then force a new deployment.
• Check task logs in CloudWatch: /aws/ecs/<service-name>.

The AWS provider uses the region/profile from your environment, not a Terraform variable.

• Confirm before applying:

  aws sts get-caller-identity
  aws configure get region
  

• Set explicitly with AWS_PROFILE=... AWS_REGION=... terraform apply if needed.

The ElastiCache Valkey cache occasionally fails to create when the target availability zone is out of capacity.

Error: waiting for ElastiCache Replication Group ... create: unexpected state 'create-failed',
wanted target 'available'

• Remove the failed Valkey cache from the ElastiCache console (disable backups first, then wait for full deletion) and re-run terraform apply.
• If the problem persists, change node_type in valkey.tf (e.g. cache.t4g.micro → cache.t3.micro) and retry.

SELinux blocks container access to ~/.aws without a relabel.

• Add :z (or :Z for exclusive use) to the volume and --userns=keep-id:uid=65532,gid=65532:

  podman run --rm -p 8000:8000 \
    --userns=keep-id:uid=65532,gid=65532 \
    -v ~/.aws:/home/nonroot/.aws:ro,z \
    -e AWS_BEDROCK_REGIONS=us-east-1,us-west-2 \
    -e ENABLE_DOCS=true \
    ghcr.io/stdapi-ai/stdapi.ai-community:latest
  

• See Local Development for the full run command.

The image runs as the unprivileged user nonroot, uid/gid 65532, while the files under ~/.aws belong to your own account and are typically readable by it alone. The mount succeeds, every file inside it is unreadable, and the server exits as if no credentials had been given.

• Run the container as yourself: --user "$(id -u):$(id -g)" -e HOME=/home/nonroot. The HOME value matters — without it the AWS SDK looks for .aws in the wrong place. Run this from your own shell, never under sudo — under sudo, $(id -u):$(id -g) resolves to 0:0 and the container silently runs as root instead of nonroot.
• On rootless Podman use --userns=keep-id:uid=65532,gid=65532 instead, which maps your account onto the image's user.
• Or skip the mount entirely and pass AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN as environment variables — see Local Development.

Both images declare python as their entry point, so anything after the image name is passed to the interpreter rather than run as a program. The Marketplace image additionally ships no shell and no package manager, so sh, bash and any install command have nothing to run there; the community image is Debian-based and still has them.

• Pass Python arguments directly: docker run ... ghcr.io/stdapi-ai/stdapi.ai-community:latest -c "import stdapi; print(stdapi)".
• To run another binary in the image, name it explicitly: --entrypoint /usr/bin/ffmpeg.
• Server options are environment variables, not command arguments — see Configuration.

The ECS service is still starting up. Health checks take a few minutes.

• Wait 2–3 minutes after deployment and refresh.
• Check the ALB target group health in the AWS console.
• If it persists longer than 5 minutes, inspect CloudWatch logs for the ECS task. No log group and no task at all points at the image pull rather than the application — see terraform apply succeeds but nothing is reachable.

The images bind IPv4 only (GRANIAN_HOST=0.0.0.0), so a client that resolves the server to an IPv6 address reaches a port nothing is listening on. Clients disagree about which address to try first, which is why the same deployment looks reachable from one language and dead from another: Node.js prefers the AAAA record and fails outright, while most Python clients fall back to the A record and hide the problem.

• Typically hit with ECS service discovery, which publishes an AAAA record for every task in an IPv6-enabled subnet. Deployments fronted by an ALB are unaffected — the load balancer terminates the client connection itself and reaches the task over IPv4.
• Set GRANIAN_HOST=:: for a dual-stack socket answering both families; see the Container Runtime note in Configuration. The Terraform module sets it when the VPC has IPv6 enabled.
• After switching, extend PROXY_TRUSTED_HOSTS with the IPv4-mapped form of each range (::ffff:10.0.0.0/112) — a dual-stack listener reports IPv4 peers in that form, and an untrusted proxy means X-Forwarded-For is ignored and the load balancer's own address is logged as the client IP.

Host header validation applies to /health as well. A load balancer health check addresses the target directly, so its Host header carries the target's IP address — which a list of domain names does not match, and every probe is answered with 400. The target group stays unhealthy and the ALB keeps returning 503.

• Prefer host validation at the load balancer: an ALB listener rule on the Host header, with TRUSTED_HOSTS left unset.
• If the application-level allow-list is required, include the address the health check actually sends.
• The container's own HEALTHCHECK is unaffected: it derives its Host header from TRUSTED_HOSTS. See TRUSTED_HOSTS.

ECS ignores the image's own HEALTHCHECK, so a task definition that declares no healthCheck gets no container-level probe at all, and one that declares a probe carried over from an earlier version runs a command the current image no longer provides. Either way the container is reported unhealthy and the service replaces it.

• Copy the healthCheck block from the ECS task definition example, which declares the image's own probe, and re-copy it when upgrading.
• The Terraform module declares it for you.
• Do not substitute a curl or urllib one-liner: it sends an untrusted Host and is answered with 400 as soon as TRUSTED_HOSTS is set.

Neither status comes from stdapi.ai: the load balancer answers them itself when a request arrives with no healthy target to forward to, because the container behind it was just killed for using too much memory. The Terraform module's default Fargate task (512 MiB) sizes the task for text generation and embeddings, where it holds little more than the request in flight — it is not enough for the paths that hold bytes in memory: audio and video through the ffmpeg pipeline, inline input files up to MAX_INPUT_FILE_SIZE each, and MAX_CONCURRENT_INPUT_DOWNLOADS of them fetched at once.

• Confirm the diagnosis: in the ECS console, open the service's stopped tasks — a stoppedReason of Essential container in task exited, with the container's own reason OutOfMemoryError: container killed due to memory usage, is this failure, not a crash in the application. The same command works from the CLI:

  aws ecs describe-tasks --cluster <cluster> --tasks <stopped-task-arn> \
    --query 'tasks[].{Stopped:stoppedReason,Containers:containers[].reason}'
  

• Cross-check with CloudWatch: the service's MemoryUtilization metric peaking near 100% in the same window rules out a health-check or networking cause.
• Fix: raise the Terraform module's memory variable — 1024 or higher for a deployment that serves audio, video or image generation under real load. Valid cpu/memory pairs are constrained by Fargate; see Amazon ECS task CPU and memory. See Cost-Optimized Deployment for the same sizing note in context.

Neither alb_domain_name nor alb_certificate_arn was set, so the Terraform module creates no HTTPS listener at all and forwards port 80 straight to the tasks. Everything works, but the API — including every API key sent to it — crosses the network in the clear. Never send an API key to that endpoint.

• Set alb_domain_name to a name in a public Route 53 hosted zone in the account: the module then issues and renews an ACM certificate, serves HTTPS on port 443, and redirects port 80 to it.
• Or supply alb_certificate_arn for a certificate you already manage. Supplying it without a domain leaves the ALB on its generated *.elb.amazonaws.com name, which no certificate can cover — that is the one configuration that does produce a browser certificate warning.

With alb_waf_enabled = true and alb_waf_block_anonymous_ips = true (both set by the production getting-started sample), the AWS-managed AWSManagedRulesAnonymousIpList rule group rejects requests from VPNs, proxies, Tor exit nodes and hosting-provider address ranges. Your address is allowed by the security group and then blocked by the WAF, so the 403 is produced before the request reaches stdapi.ai — it is not an API-key problem, and no amount of key fixing changes it.

• Tell the two apart by the body: stdapi.ai answers every refusal with a JSON error object, the WAF does not. A 403 with no JSON body came from the load balancer.
• Confirm the rule that matched: WAF & Shield → Web ACLs → <deployment> → Sampled requests in the console, or the aws-waf-logs-<deployment> CloudWatch log group when alb_waf_logging_enabled = true (the default).
• Testing from a corporate VPN, AWS CloudShell, an EC2 instance, a CI runner or any cloud host is the common way to hit this — every one of those is an address range the rule group covers. Retry from an ordinary network first.
• Keep the rule and allow yourself through, or set alb_waf_block_anonymous_ips = false in the sample's main.tf and re-apply. The rest of the WAF, including the rate limit, is unaffected.
• A 403 with no User-Agent header on the request is the other WAF rule, NoUserAgent_HEADER — see AWS WAF's NoUserAgent_HEADER rule reads like an auth failure. A 403 that does carry a JSON permission_error body is per-user cost attribution, not the WAF.

Outbound traffic to AWS endpoints is blocked.

• Confirm the ECS task's security group allows outbound HTTPS (port 443).
• If using VPC endpoints (the Terraform module default), verify the endpoint security groups and policies permit traffic from the ECS task subnet.
• If ECS runs in a private subnet without VPC endpoints, confirm the NAT gateway / route table is configured.

The name in the request is not one this deployment serves. Most names need no change when you adopt the gateway: the Anthropic, OpenAI and Cohere models Bedrock serves are published under their providers' own names as well as their Bedrock IDs, derived mechanically from the ID, so anthropic.claude-opus-5 also answers to claude-opus-5, openai.gpt-oss-120b-1:0 to gpt-oss-120b and cohere.rerank-v3-5:0 to rerank-v3.5. This 404 is what a name that does differ looks like: one your application hard-codes for a model this deployment does not serve (gpt-4o, gpt-3.5-turbo, text-embedding-3-small, dall-e-3) resolves only if you map it yourself with MODEL_ALIASES. Nothing is substituted on your behalf, because a lookalike would serve a different model than the one you asked for.

• List what this deployment actually serves: GET /search_models (the default model-discovery endpoint). Filter by capability with query parameters — e.g. GET /search_models?input_modalities=IMAGE&route=/v1/chat/completions returns only vision-capable chat models. See the Search Models API reference.
• GET /v1/models is also available for strict OpenAI SDK compatibility (lighter payload, no capability metadata).
• Confirm the pipeline itself with a low-friction model: amazon.nova-micro-v1:0 (available in all standard Bedrock regions).
• Keep the name your application already sends by mapping it onto a served model with MODEL_ALIASES — an alias only ever points at a model this deployment serves, so what it resolves to is your choice, not a guess.
• Try the name with and without its anthropic. / openai. / cohere. prefix: both forms resolve for those three families, so anthropic.claude-fable-5 and claude-fable-5 reach the same model, as do openai.gpt-oss-120b-1:0 and gpt-oss-120b. Cohere additionally spells its versions with a dot, so cohere.embed-english-v3 answers to embed-english-v3.0 and cohere.rerank-v3-5:0 to rerank-v3.5 — not to embed-english-v3 or rerank-v3-5. Models from other providers are served under their Bedrock IDs only, so name the ID or alias it. A model version Bedrock has retired needs a current one, whichever form you use.
• Only if the model is one Bedrock serves and it is still missing: verify AWS_BEDROCK_REGIONS includes a region that offers it — see the Bedrock model availability table. Adding regions never makes a name Bedrock does not serve resolve.

Anthropic models are the one family Bedrock gates behind a one-time use case form, submitted once per AWS account or Organization. Until it is accepted, the gateway's startup availability check finds the model unentitled and never adds it to the served catalogue: it is missing from GET /v1/models and GET /search_models, and naming it answers 404 Not Found — not a 503. Every other Bedrock foundation model is enabled by default in commercial Regions, so there is no per-model switch to hunt for.

1. Confirm the diagnosis in the startup log: the model is listed under unavailable_models with an unentitled issue.
2. Open any Anthropic model in the Bedrock console → Model catalog and submit the use case details, or call PutUseCaseForModelAccess. The form asks for your company name, website, industry and intended users, so no deployment can fill it in on your behalf.
3. Restart the deployment — the catalogue is built at startup, so an accepted form takes effect on the next start, not on the next request.

4. Verify with GET /search_models; the model now appears with its capabilities.

5. Submit it in every account that runs a deployment: entitlement follows the account or Organization, not the region or the gateway.

6. A model listed through AWS Marketplace is gated differently: AWS creates the subscription on first invocation, which needs aws-marketplace:Subscribe and aws-marketplace:ViewSubscriptions on the task role and AWS_BEDROCK_MARKETPLACE_AUTO_SUBSCRIBE left enabled (the default). Without those permissions the first call fails with AccessDeniedException.
7. A model that is entitled but still missing is a region problem, not an access one — see the 404 entry above and check AWS_BEDROCK_REGIONS.

Two or more of its matches were released on the same date, so the server refuses to guess which one you meant, rather than spend your money on a model you never named — sibling models released together are often priced differently. The message names the tied models. openai.gpt-5.6-* (Sol, Terra, Luna) and stability.* on the image routes are real examples of ties.

• Name the model you meant explicitly, or narrow the pattern so it matches only one of the tied models.
• See everything a pattern matches, newest first, before relying on it: GET /search_models?model=<pattern> — see Search Models.
• See Model Wildcard Patterns for the full resolution rule.

A pattern only selects from what it can order by release date, and four kinds of match are skipped even when they are listed and match the glob:

• A model with no known release date. The server cannot order it, so it is never selected by a pattern, whatever else it matches — a property of that model, not a bug in the pattern or a gap in a particular backend's catalogue. It also never makes an otherwise-unique match ambiguous.
• A legacy model, one AWS_BEDROCK_DEPRECATED_MODEL_FALLBACK covers, or one whose first use would open a paid Marketplace subscription.

Name any of these explicitly and it still resolves exactly as it does today — only pattern resolution skips them. GET /search_models?model=<pattern> returns the whole match set a pattern draws from, newest first with unknown-release-date models last, so you can see what it will do before it surprises you — see Model Wildcard Patterns.

Expected, within a bounded window. The model list is discovered from Amazon Bedrock and kept for MODEL_CACHE_SECONDS (15 minutes by default); once it expires the request that notices is answered from the list in hand and the refresh runs behind it, so no request pays for the discovery pass. A model AWS has withdrawn — or that this account has lost access to — can therefore stay listed until that refresh lands, and a request naming it is accepted and then fails at the backend with the same 404 model_not_found as any unknown model. See Model List Refresh.

• A model that is new is never served from an expired list: naming one the list does not know makes the server refresh before it answers, so it is usable as soon as it exists.
• Shorten the window with a lower MODEL_CACHE_SECONDS, at the cost of more discovery calls.
• Remove it entirely with MODEL_CACHE_MAX_STALE_SECONDS=0, which makes every expiry refresh synchronously — the freshest and the slowest setting.
• Only if the list is very old, or a model that exists never appears, is this a fault rather than the window — see the next entry.

A refresh that keeps failing leaves the list frozen. It is reported in the server log rather than in a response, because the request that triggered it was answered from the list already in memory.

• Look for Refreshing the model list from AWS Bedrock failed in the server log. It is logged at warning, and at error once the list is more than two MODEL_CACHE_SECONDS old — the signal that the list is drifting toward the ceiling.
• The usual causes are a revoked bedrock:ListFoundationModels or bedrock:GetFoundationModelAvailability permission and a prolonged outage in every configured region. Check Bedrock IAM first.
• Beyond MODEL_CACHE_MAX_STALE_SECONDS (24 hours by default) requests stop being answered from the frozen list and wait for a refresh instead, so a deployment in this state eventually surfaces the real error to clients as a 503 rather than serving a list it cannot confirm.

A published model list is only read by servers running the same version, in the same AWS account, with the same AWS_BEDROCK_* and AWS_SAGEMAKER_* configuration — every setting under those prefixes, not only the ones discovery reads. Anything else reads as an empty cache and the server discovers the catalogue itself, exactly as it would with the feature off, so changing one of those settings costs the fleet one discovery pass each rather than pointing at a table or permission fault.

• During a rolling deployment this is expected: the new version recognises nothing the old one published, so the first server of each version performs a full discovery pass and publishes for the rest. Changing any AWS_BEDROCK_* setting has the same effect, once.
• If it never gets faster, the table itself is the suspect. Every failure is reported in the server log at WARNING, naming the DynamoDB action, the table and the setting to fix — a missing dynamodb:GetItem/PutItem/Query on the table ARN is the common one, see Shared Table IAM.
• A single-container deployment gains nothing by design: there is no second server to share with, and the first one still has to discover the catalogue.
• Setting it without AWS_DYNAMODB_TABLE fails startup with a message naming both settings.

An alias in MODEL_ALIASES maps to an object that is not a valid alias configuration, so the server refuses to start rather than ignore it.

• Extra inputs are not permitted names a field that does not exist — check its spelling against the alias fields.
• Field required on model means the object gives configuration but no target model.
• A guardrail needs both guardrail_id and guardrail_version.
• An alias that only maps a name to a model stays a plain string: {"my-model": "amazon.nova-lite-v1:0"}.

The model is not offered in every AWS Region, and the catalog only lists what the configured Regions serve. Check the model's regions.

• Add a Region that offers it to AWS_BEDROCK_REGIONS; us-east-1, us-west-2, ap-northeast-1 and eu-north-1 did at the time of writing, while eu-west-3 and eu-central-1 did not.
• Request model access for it in the Amazon Bedrock console for that Region.
• Grant the task role bedrock:InvokeModelWithBidirectionalStream, which this model needs on top of the usual invoke permissions — see IAM Permissions. Without it the session opens and then ends with a 503 saying live conversation is not available on this server; the server log names the permission.
• /search_models?route=openai_audio_transcription lists what this deployment can actually reach.

A Bedrock Marketplace model endpoint is discovered, never created — the gateway only ever lists an endpoint that already exists in your account. Work through the checks in order:

• The feature is off. AWS_BEDROCK_MARKETPLACE_ENDPOINTS_ENABLED defaults to false.
• The endpoint is in a region the server does not serve. It must be in AWS_BEDROCK_REGIONS, and in AWS_BEDROCK_MARKETPLACE_ENDPOINT_REGIONS too when that is set. A Marketplace model endpoint has no cross-region form, so the server cannot reach one outside the regions it is configured to call.
• The endpoint is not yet in service. Deployment takes roughly 10–15 minutes; it appears the next time the model cache refreshes, not the instant deployment finishes. An endpoint you update — a scale, an instance change — is a different case: SageMaker keeps serving it on the previous configuration throughout, so it stays listed and keeps answering.
• Amazon Bedrock could not register the endpoint. Its own console reports this — check there before assuming a gateway-side gap.
• The server role is missing the discovery permissions. The server reports this itself, in its own log at WARNING — see Bedrock Marketplace Model Endpoints IAM.

Two endpoints deployed from the same Marketplace listing in the same region publish only one model, because both carry the listing's name. Set AWS_BEDROCK_ALLOW_MARKETPLACE_ENDPOINT_ARN to let a client name the other by its ARN, or delete the duplicate.

Not a fault. AWS bills a Bedrock Marketplace model endpoint by the instance-hour, whether or not it ever serves a request, so there is no per-token rate to report against the requests that use it — nothing is invented in its place. Read the real charge from AWS Cost Explorer or the SageMaker hosting line of your bill. The same holds for a SageMaker AI endpoint.

Expected, and it is the feature working. An endpoint scaled to zero has no capacity to answer with, and the request itself is what makes AWS provision an instance again — a few minutes for the scaling alarm, the instance and the model to load. stdapi.ai holds the connection and retries rather than failing, so the caller gets a slow answer instead of an error, and callers arriving during the wait share it rather than each starting one.

• Bound it with AWS_SAGEMAKER_WARMUP_TIMEOUT, or set it to 0 to fail fast instead of waiting.
• Keep the endpoint warm by giving its Application Auto Scaling target a minimum capacity of one — at the cost of an instance-hour bill that never stops.
• Check that any load balancer, proxy or client SDK in front of the deployment tolerates that much silence before the first byte.

The endpoint never came back up, and the server waited its whole warm-up budget before saying so. The message to the caller is deliberately generic; the server log carries the real cause at WARNING, naming the endpoint and the elapsed budget.

By far the most likely cause is the scaling alarm. An endpoint with a minimum instance count of zero and no CloudWatch alarm on NoCapacityInvocationFailures never scales back up at all: every request fails, forever. Check the alarm exists, that its step-scaling policy is its alarm action, and that it is not stuck in INSUFFICIENT_DATA:

aws cloudwatch describe-alarms --region <region> \
  --query "MetricAlarms[?MetricName=='NoCapacityInvocationFailures'].[AlarmName,StateValue,AlarmActions]"

Then check the component is actually being asked to grow, and why it might not be:

aws sagemaker describe-inference-component --region <region> \
  --inference-component-name <name> --query 'RuntimeConfig'

A DesiredCopyCount of 1 with a CurrentCopyCount of 0 that never moves means the copy cannot be placed — most often the component asks for more memory or accelerators than the instance type has, which CreateInferenceComponent accepts and the placement then fails on. A longer budget will not fix either case.

An endpoint is served because you named it, so the entry is the first thing to check.

• The model ID collides with a model already in the catalogue. The declaration is ignored and the reason is reported in the startup log; give the endpoint a model ID of its own.
• The declaration is malformed. AWS_SAGEMAKER_ENDPOINTS is a JSON object and an unknown field is refused at startup rather than ignored, so the server would not have started.
• The catalogue has not refreshed. A declaration is picked up when the server starts; changing the environment needs a restart, as every setting does.

The endpoint refused the server's credentials. The OpenAI-compatible route takes a short-term API key derived from the server's own role, which needs sagemaker:CallWithBearerToken on * in addition to sagemaker:InvokeEndpoint on the endpoint ARN — a policy carrying only the second is refused with a 403 the server reports as a feature being unavailable. See SageMaker AI Endpoints IAM.

The catalog lists the classic Bedrock models and none of the ones served through Amazon Bedrock Mantle, in every configured region at once. A per-region or per-model gap looks different: this is the shape of a network policy that reaches one endpoint and not the other.

Bedrock Mantle is served from bedrock-mantle.<region>.api.aws — a different domain from classic Bedrock's bedrock-runtime.<region>.amazonaws.com, so an allowlist, firewall rule, proxy exception or VPC endpoint written for amazonaws.com does not cover it.

• Allow bedrock-mantle.<region>.api.aws outbound, for every region in AWS_BEDROCK_REGIONS.
• On a private deployment, create the interface VPC endpoint com.amazonaws.<region>.bedrock-mantle with private DNS enabled.
• Behind a proxy, make sure HTTPS_PROXY is set in the task environment and that NO_PROXY does not exclude api.aws.
• Confirm the task role carries the Bedrock Mantle permissions — without them the models are simply not listed rather than refused. See IAM Permissions.
• Check what was discovered: GET /search_models returns every model the server found, with its regions.

See Outbound Network Requirements for the full destination list.

The startup warning names the region, the endpoint address and the exception chain behind the failure, so a blocked route (ConnectionTimeoutError), a refused connection (ConnectionRefusedError), an intercepting proxy's certificate (SSLCertVerificationError) and an unresolvable address (ClientConnectorDNSError) are told apart without further instrumentation.

The startup log lists the region under bedrock_mantle_regions_without_endpoint instead of unreachable_bedrock_regions. Bedrock Mantle is offered in fewer regions than classic Bedrock, and where it is not offered bedrock-mantle.<region>.api.aws has no DNS record at all — nothing to retry, and no network policy to change. See model availability by endpoint.

• Remove the region from AWS_BEDROCK_MANTLE_REGIONS, or unset it to fall back to the regions of AWS_BEDROCK_REGIONS that offer Mantle. Classic Bedrock in that region is unaffected either way.
• If no configured region offers Mantle, the log says so as well and no Mantle model is served — set AWS_BEDROCK_MANTLE_ENABLED to false to stop the warning.
• If AWS has since added the region, list it explicitly in AWS_BEDROCK_MANTLE_REGIONS: an explicit list is used as given.
• If the address should resolve because a VPC endpoint provides it, this is instead the private-DNS case covered by Every Bedrock Mantle model is missing above.

The gateway reached the AWS service, but the ECS task role (or your local AWS credentials) lacks permission for the action it used. AWS returns AccessDeniedException, which stdapi.ai answers as a feature this deployment cannot run: HTTP 503, error code feature_unavailable, and the same generic message whatever is missing. This is an IAM misconfiguration, not a client API-key problem, and the client is deliberately told nothing about it — the server log names the operation, the model and the permission AWS refused, under error_detail.

• Confirm the task role grants bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream (and the bedrock:Converse* actions) for the target model ARNs.
• For models invoked through an inference profile, allow both the profile ARN and the underlying foundation-model ARNs in the policy.
• By a distance the most likely cause for a Marketplace model endpoint that is listed but every call to it is refused: a policy scoped to foundation-model/* and inference-profile/* — copied from an example that predates Marketplace model endpoints — denies every call to one. It needs arn:aws:bedrock:*:ACCOUNT_ID:marketplace/model-endpoint/* as well, on the task role and, when per-user cost attribution is enabled, on the end user role too — see Bedrock Marketplace Model Endpoints IAM.
• Anthropic models only, and not this 503: Claude requires a one-time use case form per AWS account or Organization. Without it the model never enters the served catalogue and naming it answers 404 Not Found — see the use case form entry for the steps.
• Audio, embeddings, and file features need permissions for the relevant services (Polly, Transcribe, Translate, Comprehend, S3) — see IAM Permissions for the full IAM reference.
• A 403 permission_error on a model call means the opposite: per-user cost attribution is enabled and the end user's role was denied, so the policy to fix is that role's, not the task role's.
• A 400 invalid_request_error naming an s3:// input also means the opposite: the denial is on the object the request pointed at, in one of the external buckets declared in AWS_S3_ACCEPTED_BUCKETS. Only the caller can fix that one — a wrong key, or a bucket policy that does not grant this deployment's role s3:GetObject on the object. Objects in the deployment's own buckets keep the 503 above.

The startup log lists the region under bedrock_regions_missing_iam_permission, never under unreachable_bedrock_regions: AWS answered, and refused. The entry names the IAM action, the region, and the resource the call was attempted on when AWS supplied one.

• Read the wording first, it says whose problem it is. the server role is missing the IAM permission <action> means IAM evaluated a policy and denied it — grant the action and restart. AWS denied <action> … unless the service does not offer the operation there means AWS refused without naming an action, which is equally how a region that does not offer the operation answers; check regional availability before editing a policy.
• provisioned model discovery skipped: … is not a lost region. Provisioned throughput is offered in a subset of regions and the listing is refused in the others, so the region keeps its on-demand models and loses only any model it serves exclusively as provisioned throughput. As of 2026-08-31 that is the answer in af-south-1, ap-east-2, ap-southeast-5, ap-southeast-6, ap-southeast-7, ca-west-1, il-central-1 and mx-central-1.
• Model discovery calls bedrock:ListFoundationModels, bedrock:ListProvisionedModelThroughputs and bedrock:ListInferenceProfiles in every configured region. Only the provisioned-throughput listing degrades; a denial on either of the other two costs the whole region's models. See IAM Permissions.
• With AWS_BEDROCK_MARKETPLACE_ENDPOINTS_ENABLED set, bedrock:ListMarketplaceModelEndpoints and bedrock:GetMarketplaceModelEndpoint are called too, and a denial is reported the same way under <region> (Marketplace endpoints). It costs only the endpoints of that region, never a foundation model.
• Do not read any of this as a model retirement: the models are still there, this deployment just may not list them.
• The same distinction applies per request. A denied call names its permission in the request log's error_detail and answers 503 feature_unavailable, never a region failover.

The start log event reports server_start_time_ms in the tens of thousands where a healthy deployment reports a few thousand. Startup reads the model catalogs of every configured region, and the ECS task metadata endpoint before them; a destination that never answers is only given up on after a timeout.

• Read the server_warnings of the same event first. ECS container metadata endpoint answered after N attempts in X s accounts for that many seconds on its own: the endpoint is served by the ECS agent over the task ENI and answers slowly when the task is CPU-starved at boot. Raise the task CPU, or the cpu of the Fargate task definition, so the agent is scheduled promptly.
• unreachable_bedrock_regions and bedrock_mantle_regions_without_endpoint each name a region that spent its full timeout budget. Removing the region from AWS_BEDROCK_REGIONS or AWS_BEDROCK_MANTLE_REGIONS removes the delay.
• AWS_CONNECT_TIMEOUT bounds each connection attempt and the model-catalog fetch that follows it, so lowering it lowers what an unreachable region can cost. It also bounds failover between healthy regions, so keep it above your real inter-region latency.
• Model discovery is per region and runs in parallel, so the count of regions costs far less than one unreachable region does.

Some work is started outside the request that asked for it, so the caller is answered without waiting: temporary file cleanups, vector store file indexing, and the release of live audio sessions. When a task is stopped, the server waits SHUTDOWN_DRAIN_TIMEOUT seconds for that work, then cancels the rest — and reports the counts as abandoned_background_tasks in a stop log event raised to warning.

• A vector store file reported failed after a deployment, with last_error saying the indexing was interrupted, or a temporary object that outlived its request, is what those counts look like from the outside. Attach the file again, or let the object's lifecycle rule expire it. A file is never left in_progress: one whose indexing is gone is settled the next time it, its store, or the store's file list is read.
• Raise SHUTDOWN_DRAIN_TIMEOUT and the container stop timeout together — the container runtime sends SIGKILL a fixed delay after the stop signal (30 seconds by default on Amazon ECS), so raising the wait alone changes nothing past that delay.
• Counts that persist after raising both mean the work itself is long, not that the wait is short: indexing a large file outlasts any stop timeout. Attach large files when a deployment is not in flight, and treat the operation as retryable.
• The wait is best effort in every deployment: a Spot interruption or a hard kill ends the process regardless, so never rely on it for anything whose completion matters.

A guardrail is configured — AWS_BEDROCK_GUARDRAIL_IDENTIFIER or a MODEL_ALIASES entry carrying one — while AWS_BEDROCK_MANTLE_PREFERRED_MODELS routes models to Amazon Bedrock Mantle, which cannot apply guardrails. Those models would be served unfiltered, so startup stops instead. The setting defaults to openai.gpt-5.6, so a guardrailed deployment meets this without having set it.

• Keep the guardrail: export AWS_BEDROCK_MANTLE_PREFERRED_MODELS= (empty). Every dual-homed model, the GPT-5.6 family included, is then served by the classic endpoint under the guardrail — and web_search and code_interpreter are refused with a 400 for that family.
• Keep Mantle routing: remove the guardrail configuration, or point the offending alias at a model that stays on the classic endpoint.
• The message names the routed entries, so a deployment that listed several sees which ones are at stake.

AWS returned ThrottlingException, TooManyRequestsException, or ServiceQuotaExceededException — mapped to HTTP 429 with error type rate_limit_error. You've hit the per-region Bedrock quota.

• Add more regions to AWS_BEDROCK_REGIONS. Each region has its own independent quota — three regions ≈ triple the throughput.
• See Resilience & Failover for multi-region routing configuration.
• Check quotas in the AWS Service Quotas console for Amazon Bedrock.
• When the router put a region on a quota backoff while serving the request, the response carries a retry-after header (in seconds) telling the client exactly how long to wait — OpenAI, Anthropic and Cohere SDKs honour it automatically instead of guessing an exponential backoff, up to the 60 s ceiling their retry loops apply to server-supplied delays.

A specific model is unavailable or requests to it are rejected because your account's data retention mode is incompatible with what that model requires.

Amazon Bedrock enforces retention compatibility at invocation time: each model declares the retention modes it accepts, and if your effective mode is not among them, the request is blocked.

Common scenarios:

• Your account is set to zero data retention (none) but the model requires default or provider_data_share for safety or abuse-prevention purposes. Bedrock blocks the request to honour your retention policy. To access the model, either switch to a compatible retention mode or contact your AWS account manager to request ZDR eligibility for that specific model.
• Your account is set to default but the model exclusively requires provider_data_share (typically models with mandatory provider-side safety review). The model will appear as unavailable. Enabling provider_data_share grants access but means AWS will share your inference data with the model provider — see Data Privacy before enabling it.

Bedrock rejected the request parameters (ValidationException / BadRequestException), mapped to HTTP 400 with error type invalid_request_error — for example an unsupported parameter for the chosen model, an out-of-range value, or content that exceeds the model's limits.

• Read the message detail returned in the response (correlate with x-request-id in the server logs).
• Confirm the parameter is supported by the model — see the per-API Feature Compatibility tables.

Request fields stdapi.ai does not declare are forwarded to Amazon Bedrock as provider-specific inference parameters, so any parameter a model accepts can be passed through — including the highly model-specific ones no common API surface exposes. Some OpenAI-SDK-based clients also use that same channel for their client-side settings — LiteLLM-derived ones send drop_params, api_key or custom_llm_provider in extra_body — and Bedrock answers ValidationException for a field no model declares. The symptom is a route that fails for one client and works for every other.

• The known LiteLLM control parameters are stripped by a built-in denylist, so this only appears for a name it does not yet cover. The rejected field is in the Bedrock message detail, correlated via x-request-id in the server logs.
• Add that name to EXTRA_MODEL_PARAMS_DENYLIST — it is merged with the built-in list, and every other extra parameter keeps being forwarded.
• If no client needs the passthrough, EXTRA_MODEL_PARAMS_DROP_ALL disables it outright. Per-model defaults set through DEFAULT_MODEL_PARAMS are unaffected — only request-supplied extras are dropped.

Changing the tool set between the turns of one conversation works: the tools a turn declares are the only ones the model may call, and a tool dropped from that list is not offered again. The exception is the empty tool set — a turn that declares no tool at all, or sets tool_choice to none ({"type": "none"} on /v1/messages), while the conversation it replays already contains a tool call. Those earlier tools stay available for the turn, so the answer can be a call to one of them.

• Remove the tool turns from the history — the earlier tool_calls/tool messages, tool_use/tool_result blocks or function_call/function_call_output items — and send the conversation as plain text. Nothing then keeps a tool available.
• Or keep the history and ignore the returned call: the message is otherwise a normal answer, and the call names a tool the conversation already used, never an unknown one.
• Declaring a different tool set on that turn does not help: none withdraws the tools the request declares, whichever they are, and only the conversation's own tools remain.
• No setting changes this, and it is the same on /v1/chat/completions, /v1/responses and /v1/messages.

Some configuration reaches the model that the client did not send, or the value the client sent is not the one applied. Two layers of server-side configuration sit behind every request, and both are deliberate.

• The model name may be an alias carrying its own configuration — check the entry in MODEL_ALIASES for that name. Requests naming it get its service tier, guardrail, metadata and model parameters; requests naming the target model directly do not.
• The reverse also happens: alias and server-wide configuration apply to models served through Amazon Bedrock's Converse and InvokeModel operations. A model served through Amazon Bedrock Mantle applies the request's own values only, so a configured service tier, metadata or model parameters are ignored there by design — see the scope note.
• A request value is discarded on purpose when its override setting is disabled: AWS_BEDROCK_ALLOW_SERVICE_TIER_OVERRIDE for the tier, AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE for the guardrail headers.
• Otherwise the value comes from the server-wide setting for that field — DEFAULT_MODEL_SERVICE_TIERS, DEFAULT_MODEL_PARAMS or AWS_BEDROCK_GUARDRAIL_IDENTIFIER. The order is always the request, then the alias, then the setting.

AWS_BEDROCK_GUARDRAIL_SCOPE_TURNS is set. On the chat routes the guardrail then evaluates the text of that many trailing user turns only — the saving it buys is paid for in detection; see Evaluation Scope for exactly what stops being checked.

• The conversation history is no longer submitted — the earlier user turns, and the answers the client attributes to the model. A forbidden word or denied topic placed there is not detected. Raise the number of turns to widen the window, or unset the setting to evaluate the whole conversation again.
• An image sent alongside the newest question is not submitted either: only the text of a scoped turn is. Unset the setting on a deployment that relies on the guardrail to screen images on the chat routes.
• Model output is always evaluated in full, and the other routes — like a chat model served through InvokeModel rather than Converse — check the content of the request they are given, whatever this is set to.

The result is correct: the classification hit something the OpenAI moderation categories have no field for, so it is reported on the overall flagged field alone. Which one depends on the model that served it.

• A guardrail resource: a denied topic, a word filter, a sensitive-information policy, a prompt attack filter or contextual grounding — every policy other than the content filters. Read the guardrail's own configuration in Amazon Bedrock to see which ones it applies.
• amazon.bedrock-runtime-guardrail-checks: the prompt attack or personal data check, enabled with AWS_BEDROCK_GUARDRAIL_CHECKS_PROMPT_ATTACK or AWS_BEDROCK_GUARDRAIL_CHECKS_PII_ENTITIES. Broad PII entity types flag ordinary prose — a city name is an ADDRESS — so narrow the list to the types your policy covers. Each enabled check is also billed separately, which is the other symptom it produces.
• amazon.comprehend-toxicity: the overall toxicity score, or a label with no OpenAI counterpart such as profanity.

They are served by Amazon Bedrock Mantle by default, so that their web_search and code_interpreter tools work. Mantle has no cross-region inference profiles, so those requests no longer ride the Global profile the classic endpoint uses by default, and pay the In-Region rate — exactly 10% above the Global one, on every token dimension.

• Accept the routing and the rate: nothing to do. It is what makes OpenAI GPT web search available, and the models' usage now appears under Bedrock Mantle rather than Bedrock in cost reporting.
• Prefer the Global rate: export AWS_BEDROCK_MANTLE_PREFERRED_MODELS= (empty). The family returns to the classic endpoint, its server tools are refused with a 400, and token counting works again.
• A deployment that had already disabled AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL paid the In-Region rate before and pays it now: nothing changed for it.

The message states the length the server accepts ('input' is limited to 3,000 characters…, or 20,000 with a generative voice). A generative voice speaks up to 20,000 characters unaided; every other voice, and longer generative input, is synthesized into an S3 bucket co-located with the region serving the request, and none is configured there.

• Set AWS_S3_BUCKET for the first region of AWS_BEDROCK_REGIONS, and an AWS_S3_REGIONAL_BUCKETS entry for every other region that may serve speech — the server log names the one that was missing.
• Grant the task role polly:StartSpeechSynthesisStream (generative voices beyond 3,000 characters), polly:StartSpeechSynthesisTask, polly:GetSpeechSynthesisTask, and S3 read/write/delete on those buckets — see IAM Permissions.
• A generative request that still behaves like the others — rejected without a bucket, or slower than expected with one — is missing polly:StartSpeechSynthesisStream; the server log names the failure.
• Up to 3,000 characters (6,000 including SSML markup) never needs a bucket; the limits and the expected latency are in Long Input.

The message names the condition: amazon.nova-2-sonic-v1:0 serves json and text only, and accepts at most 10 minutes of audio per request. It returns no timestamps and does not report a detected language, so subtitles, verbose_json, diarized_json and timestamp_granularities cannot be produced from it.

• Request json or text, or send the same audio to amazon.transcribe, which produces timestamps, SRT/VTT subtitles, speaker diarization and longer recordings — see Transcriptions.
• The same model and the same limits apply on Translations.

The message is Amazon Transcribe's own failure reason for the job, and it names KMS. Only requests that stage audio in a bucket are affected: AWS_TRANSCRIBE_OUTPUT_ENCRYPTION_KEY_ARN encrypts the job's output, so streamed transcriptions, which write nothing, keep working — which is what makes the failure look model-specific at first.

• Grant the task role kms:GenerateDataKey and kms:Decrypt on that key, and allow the same role in the key's own policy — a grant on only one of the two denies the job. See Speech-to-Text.
• With AWS_TRANSCRIBE_REGION unset, a job runs in whichever candidate Region has a co-located bucket, so a single-Region key fails as soon as failover moves the job. Use a multi-Region key, or pin the Region.
• A key policy conditioned on the encryption context must not require stdapi-ai.user_id: it is sent only when the request identifies an end user, so requiring it denies every anonymous call.

stream=true always answers with server-sent events, but they arrive phrase by phrase only when the request can be served that way. Otherwise every event is delivered at once, after the whole recording has been read — which reads as a stream that hangs and then completes in a single burst. See Streaming.

• Name the language expected: send language, or two or more languages. A request naming neither has to read the recording before it can tell which language it is in.
• Set AWS_TRANSCRIBE_STREAM_LANGUAGES to the languages your callers actually send, which gives phrase-by-phrase delivery to requests that name none.
• Drop provider-specific parameters other than VocabularyName, VocabularyFilterName and VocabularyFilterMethod. MaxSpeakerLabels, ShowSpeakerLabels, ChannelIdentification, MaxAlternatives, ToxicityDetection, ContentRedaction, IdentifyMultipleLanguages and the rest are honoured in full, but the request that uses one is answered in a single burst rather than phrase by phrase.
• response_format=diarized_json on its own does not cost the phrase-by-phrase delivery: its transcript.text.segment events are interleaved with the deltas like any other.

Real-time audio requests must become ready within AWS_CONNECT_TIMEOUT in each candidate region — that budget covers the connection, the initial handshake and the first response together, not the connection alone. On a high-latency or NAT-fronted network the default of 5 seconds can expire in every region, and the request then ends as a 503.

• Raise AWS_CONNECT_TIMEOUT to 10 seconds or more; AI_RESPONSE_TIMEOUT governs the response itself and has no effect here.
• The server log names each Region that was abandoned and why.

The upgrade request never reached the Realtime API route — either the URL is wrong, or something in front of the deployment does not forward a WebSocket upgrade at all.

• Check the path: the route lives at ${OPENAI_ROUTES_PREFIX}/v1/realtime (/v1/realtime with the default empty prefix). A client dialling the wrong prefix reaches no route and gets 404 from the framework itself, before authentication is even checked.
• An Application Load Balancer forwards a WebSocket upgrade by default, but anything placed in front of it — a CDN, an API Gateway REST API (which has no WebSocket support at all), a reverse proxy that does not pass through Connection: Upgrade / Upgrade: websocket — answers its own 404 or refuses the upgrade first. See WebSocket-Capable Deployment.
• Confirm the deployment actually ships the Realtime API: this is a versioned feature, not every earlier release includes it.

The WebRTC transport is an operator opt-in: without REALTIME_WEBRTC_ENABLED the endpoint answers 404, and the error message says so. Enabling it is a deployment-architecture decision — a UDP media path straight to the task, a single instance — not a flag to flip casually; see WebRTC calls terminated by the gateway and the deployment section first.

• A browser does not need it. It connects to the same WS /v1/realtime with an ephemeral client secret in the Sec-WebSocket-Protocol list, and captures and plays the audio itself.
• A client that must speak SIP — a phone line — belongs behind a voice-agent framework that terminates the media itself and reaches this API over the WebSocket; the SIP call verbs (accept, reject, refer) answer 400 on every deployment. See Put WebRTC or a phone line in front of the gateway.
• Do not try to route the media UDP through the deployment's ALB: its listeners carry HTTP and HTTPS only. The Terraform module's media mode gives the task the public UDP path instead.

The SDP exchange rides ordinary HTTPS through the ALB, so it succeeds even when the media path is broken: the failure only appears as a call that connects and stays silent (the browser's iceConnectionState sticks at checking or failed).

• The task has no public UDP path. The default deployment keeps tasks in private subnets behind an ALB that cannot carry UDP. Enable the module's realtime_webrtc_media_enabled, which assigns the task a public IP and opens the UDP port range in both directions — see the deployment section.
• The subnets' network ACLs drop the media. On the VPC the module creates, media mode widens the application subnets' NACLs itself, so this only bites on subnets you brought through subnet_ids — the module cannot write rules in a VPC it did not create. Allow the UDP media range inbound and the ephemeral range 1024-65535 in both directions on those subnets; without it the SDP exchange still succeeds over TCP and no media ever arrives, which is exactly this symptom.
• The answer advertises a private address. Behind 1:1 NAT (an ECS task with a public IP) the process sees only its private address; without REALTIME_WEBRTC_STUN_SERVER the ICE candidates it sends are unreachable. The module's media mode sets it.
• The security group does not cover the negotiated port. Media binds OS-assigned ephemeral UDP ports; the opened range must match the kernel's ephemeral range (the module opens 32768-60999 by default).
• The caller's network blocks UDP. Corporate networks often do; only a TURN relay you run gets media through — see REALTIME_WEBRTC_TURN_SERVER.
• Call control lands on another instance. hangup or WS /v1/realtime?call_id=... answering 404 naming the instance means the deployment runs more than one task; media mode requires a single instance.

The offer named only addresses the gateway will not probe. It screens the candidates a caller offers before answering, because those addresses are where its own ICE checks go — see What a WebRTC call demands of the deployment.

• The caller is on the deployment's own network (same VPC, on-premises, a LAN), so every candidate it has is a private address. Set REALTIME_WEBRTC_ALLOW_PRIVATE_CANDIDATES; it is off by default so an untrusted caller cannot aim UDP probes inside the VPC.
• The caller offered only mDNS (.local) candidates, which browsers use to hide the host address. Those are dropped whatever the setting is — nothing resolves them from a server — and the caller needs a server-reflexive candidate: configure a STUN server in the client's own RTCPeerConnection, or a TURN relay.

Expected: the model session behind the call is capped at 480 seconds by Amazon Nova Sonic, and when it ends the peer connection is torn down with it — see Session Lifecycle and Limits. A phone-length conversation needs the caller to reconnect with a fresh offer; context does not carry over.

Two different things end a Realtime session, and the close code tells them apart — inspect it in the client's WebSocket close handler.

• Close code 1000, reason session_expired: expected. Every session is capped at 8 minutes; reconnect to continue the conversation — see Session Lifecycle and Limits.
• Closes with no code from the gateway at all, well before 8 minutes: the load balancer's idle timeout fired on a quiet stretch between spoken turns. Raise alb_idle_timeout to at least 8 minutes (480 seconds) — see The idle timeout bounds a session.
• Closes during a deployment, scale-in, or Spot interruption: the ECS task holding the session was replaced. There is no live handoff between tasks — see A deploy truncates open sessions.
• Close code 1001, reason server_shutdown: the deployment was shutting down when the session was still open; reconnect once it is back.
• Close code 3000: a fatal error, not a limit. The reason is <error type>.<error code>, and a terminal error event carrying the same detail was sent just before the close frame.

The model is waiting for the result. A function call suspends the conversation until a function_call_output item naming the same call_id comes back, and nothing else is spoken in the meantime.

• Answer every call, including a failed one. Return {"error": "..."} rather than nothing; there is no timeout that resumes the conversation for you.
• Reply to the call_id the call carried, as response.function_call_arguments.done reports it. Any call_id is accepted and carried to the model, so a wrong one is not refused — it simply leaves the real call unanswered and the session silent.
• Answer each call once. A repeated answer is carried to the model too, which then sees a turn the application did not intend.
• A session.update that changes tools or tool_choice after the model has answered anything is refused: declare them before the conversation opens, as with the voice and the audio formats.

If the deployment's WAF is enabled with the AWS-managed Common Rule Set (alb_waf_enabled = true), its NoUserAgent_HEADER rule blocks any request — including a WebSocket upgrade — that carries no User-Agent header, with a plain 403 that is easy to mistake for a rejected credential.

• Every mainstream WebSocket client library sets a User-Agent automatically; this only surfaces with a hand-rolled client (a bespoke SIP/telephony bridge, a minimal test script).
• Check the WAF sampled requests in the console to confirm NoUserAgent_HEADER is the rule that matched, before assuming the API key or ephemeral secret is wrong.
• Have the client send any non-empty User-Agent, or exclude the rule for the Realtime path. See AWS WAF's NoUserAgent_HEADER rule reads like an auth failure.

A Realtime API ephemeral client secret is a signed token with nothing stored server-side, verified by re-checking its signature against a shared key — every instance must sign with the same key for that to work.

• With no API_KEY-family setting configured at all, each instance falls back to a random signing key generated per process, so a secret minted by one instance never verifies on another — the symptom is intermittent rejection that tracks which instance the client's connection happened to land on.
• Set REALTIME_CLIENT_SECRET_KEY explicitly to a value shared by every instance; this also covers a deployment with no API key by design (e.g. behind an IP-restricted ALB).
• A deployment that already configures an API key is unaffected: the signing key is derived from it automatically, and that same key is already shared across instances.
• Rotating the API key or REALTIME_CLIENT_SECRET_KEY invalidates every client secret minted before the change, the same as an expired one.

The S3 bucket is missing, unreachable, or in the wrong region.

• The Terraform module creates the bucket automatically unless you pass your own via aws_s3_bucket.
• If you're using your own bucket: AWS_S3_BUCKET must point to a bucket in the same region as the first entry in AWS_BEDROCK_REGIONS.
• Verify the ECS task IAM role has s3:PutObject / s3:GetObject on the bucket.
• 503 saying transcription is not available on the current server means no region that can run a transcription has a bucket at all: set AWS_S3_BUCKET, AWS_TRANSCRIBE_S3_BUCKET or an AWS_S3_REGIONAL_BUCKETS entry for it. The server log names the settings.
• 503 saying the 'url' response format is not available on the current server means there is no bucket to host the images a url response points at: set AWS_S3_BUCKET, or request response_format="b64_json", which needs no storage. Image requests are refused before anything is generated, so a misconfigured deployment is never billed for an image it cannot serve; with a guardrail configured, the prompt's check runs first and is billed, because a prompt the guardrail blocks is answered as such rather than as a missing bucket.

The Batch API and the Message Batches API refuse at submit what the backend would otherwise fail hours later, so most surprises land in the create call.

• 503 (or 529 on /anthropic/...) on every batch endpoint: the deployment declares no batch service role. Set AWS_BEDROCK_BATCH_ROLE_ARN and grant the policies in Batch Inference IAM. The server also reports the disabled feature in the server_warnings field of its start log event.
• 400 naming a minimum of 100 requests: batches run on a backend with a floor of 100 requests per model. A batch naming several models must reach it for each of them; combine the small ones or send them without batching. The 100 is the default of the per-model Amazon Bedrock quota Minimum number of records per batch inference job (Amazon Bedrock quotas), and it is the value checked here whatever your own account's quota says.
• The batch stays validating (in_progress) for several minutes: expected. Validation alone takes a few minutes before any request runs, and the whole batch has a 24-hour window. Poll rather than resubmit — a resubmission is a second, separately billed batch.
• The create call fails where you expected a batch that later reports failed: every problem in the input file — an unparseable or truncated line, a url that is not the batch's endpoint, a method other than POST, a missing body, a repeated custom_id, a file uploaded with another purpose — is answered by POST /v1/batches itself, with a 400 naming what to fix. No batch is created, so there is none to poll and no errors.data[] to read; What the input file is checked for lists every case. failed is reported only for a batch that was accepted and then could not run at all.
• 400 naming tool use or a structured output schema: neither is available in a batch. Remove tools/tool_choice and response_format of type json_schema, or send those requests without batching.
• 400 saying the model is not available for batched requests: not every model can run batched. Pick another one; when the batch named several models, no sibling job is left running.
• A model that batches fine is missing from search_models?batch=true, or reports batch: false: that flag is a discovery hint published on a best-effort basis and is never used to reject anything — submit the batch and let the answer decide. It is reported for no model at all while COST_TRACKING is disabled, and for a few seconds after startup while the catalogue is still being built.
• 503 on creation, with nothing wrong with the request: the backend refused the job for a reason that is not the model — the account's batch quota for that model, the service role, or a restriction such as a model the provider marked legacy and the account has not used in the last 30 days. The client message is deliberately generic; the server log carries the reason the backend gave, as a warning.
• 503 on creation, after the endpoints answered normally: the task role is missing bedrock:CreateModelInvocationJob or the iam:PassRole statement on the batch service role; the server log names which. A batch that starts and then fails without results usually means the service role itself cannot read or write the bucket under AWS_S3_BATCHES_PREFIX — the reason the backend gives is logged as a warning when a job reports Failed.
• A batch reports no cached tokens, whatever its requests asked for: prompt caching does not apply to batched requests, on any model. A cache hint — cache_control on /anthropic/v1/messages/batches, prompt_cache_key or prompt_cache_breakpoint on /v1/batches — is accepted and dropped rather than refused, so the request is answered normally and no cached tokens are reported for it. There is no discount to lose: batched requests are billed at the batch rate already.
• The first read after the batch ends is slow: the results are translated and published on that read. Later reads are immediate.
• The result files are gone, or never go away: a batch created with output_expires_after deletes both files that long after they are written, and one created without it keeps them until they are deleted with the Files API. The clock starts when the results are published, not when the batch was created.

The listing answers from a window of the most recent batches, found by a seek bounded to a fixed number of storage requests. A burst of thousands of batches created inside the same minute can outrun that budget, and the ones beyond it fall outside the window.

• The record is intact: GET /v1/batches/{batch_id} returns it, and so does cancelling or reading its output. Only the listing is bounded — see Listing Order.
• A cursor that points outside the current window returns an empty page rather than an error, so a paginating client stops early instead of failing.
• Record the id each POST /v1/batches returns and address batches by it, rather than rediscovering them through the listing. Amazon Bedrock's own batch quotas bound how fast batches can realistically be created, so this density is hard to reach by accident.

Indexing runs after the response is sent, so a file is in_progress for a moment by design — see Vector Stores.

• 503 on every vector store endpoint: the deployment declares no vector storage. Set AWS_S3_VECTORS_BUCKET and AWS_S3_VECTORS_REGION, keep AWS_S3_BUCKET set, and grant the Vector Stores IAM permissions.
• 503 on one operation only (creating a store, searching, deleting): a single s3vectors action is missing from the task role. The client message is deliberately the same as above; the server log names the action and the bucket.
• Creating a store fails: the vector bucket must already exist, in the Region named by AWS_S3_VECTORS_REGION. A vector bucket is a Region-local resource with no failover, so a bucket in another Region is not reachable at all.
• 400 on the attach, or a file settling as failed with unsupported_file: the store does not index that file type. A file attached on its own is refused there and then, since its content type answers the question; one attached in a batch is reported on the file. Read the message — it names what this store indexes, since that differs per store. A store the server owns indexes text only, so a PDF or an office document settles here; convert it first — RAG Pipelines shows a conversion stage — or attach it to a knowledge base store, which indexes those formats as they stand. When the message names formats and the file is already one of them, its bytes are not what the content type claims.
• A file settles as failed with server_error: indexing hit a backend error, or was interrupted before it finished — last_error.message says which. Interrupted means the server was replaced, scaled in or killed while it was indexing, and nothing else is wrong. Check the background log event sharing the request's id for the backend case; attach the file again in both. To stop losing that work at every deployment, give the deployment an indexing queue — AWS_SQS_VECTOR_STORE_QUEUE_URL — and another server finishes the job instead.
• A file stays in_progress far longer than the others: a large file is many passages, each embedded in turn, and indexing is bounded server-wide, so a file attached while others are being indexed waits its turn. A file no server is indexing any more settles as failed rather than waiting for good, so an unchanging in_progress is work that is still queued.
• A search returns nothing after attaching: the store is still indexing (status is in_progress), the store has passed its expiration (status is expired), or the filters match no file. A filter applies to the file's attributes, never to its content.
• 409 on an update: several requests are changing the same store at once. Retry the request.

Only deployments that set AWS_SQS_VECTOR_STORE_QUEUE_URL hand indexing to a queue; without it, indexing runs in the server that accepted the request and everything below is expected behaviour rather than a fault.

• The server refuses to start, naming the setting: the URL is not an Amazon SQS queue URL (https://sqs.<region>.amazonaws.com/<account-id>/<queue-name>), it names a FIFO queue, or AWS_S3_VECTORS_BUCKET is unset. FIFO is refused because its deduplication would silently drop a legitimate re-attach of the same files.
• A start log warning says the queue could not be described: the queue does not exist, or the task role lacks sqs:GetQueueAttributes. The deployment still runs and still queues, but it cannot read your redrive policy, so it falls back to its own retry count. Grant the Durable Vector Store Indexing permissions.
• A start log warning says the queue has no dead-letter queue: add a redrive policy. Without one, the message of a file that cannot be indexed is dropped once its retries run out instead of being kept for inspection.
• Every file still settles as failed after a deployment: the send is failing, which the server log reports at error naming sqs:SendMessage. A deployment that cannot queue keeps indexing in-process, which is exactly the behaviour the setting was meant to replace, so the symptom looks like the setting doing nothing.
• Files sit in_progress for minutes under load: a server only takes jobs off the queue while it is not busy answering requests, so indexing yields to clients by design. Scale out, or wait.
• A file settles as failed although the queue is configured: the job ran out of deliveries. Its message is in your dead-letter queue; the server log says so at error. Attach the file again once the underlying cause is fixed.
• Files attached with a tenant key, or by an identified end user, never reach the queue: a queued job runs under the server's own identity, so a request whose embeddings are billed to a tenant AWS credential or attributed to an end user under AWS_BEDROCK_USER_ROLE_ARN is indexed in the server that accepted it, keeping the spend where the request put it. Those files settle as failed when that server stops first; attach them again.

A knowledge base store is addressed, never created, so most of these are configuration rather than a bad request.

• 404 on every route of a vs_kb_... identifier: the knowledge base is not listed in AWS_BEDROCK_KNOWLEDGE_BASE_IDS, or it does not exist in the first AWS_BEDROCK_REGIONS entry, or the task role lacks the read permission on it. The three cases answer identically by design, so the allowlist cannot be probed by a client; the server log says which one it was. Check the setting first, then grant the Knowledge Base Vector Stores permissions on the knowledge base ARN.
• 503 when attaching a file, while search and listing work: the knowledge base has more than one data source, so which one a document belongs to is ambiguous. Name it in the setting as <knowledgeBaseId>/<dataSourceId>.
• 400 saying files cannot be attached to the store, while search and listing work: the allowlisted data source keeps its corpus in sync from somewhere else — a bucket, or another connected service — and takes no file handed to it. Only a custom data source does. Point the entry at one, as <knowledgeBaseId>/<dataSourceId>; the server log names the data source that refused. A knowledge base can hold both kinds, and the store keeps serving search and listing meanwhile.
• 400 when deleting a file a search returned: that document belongs to the corpus behind the store rather than to the files attached here, so it is readable and never removable. Remove it where the corpus comes from.
• 400 on an update, a delete, a chunking strategy, a file batch, a score_threshold or a file's content: none of these apply to a store managed outside the server. The refusal table lists each one and what to do instead.

A conversation has a bounded lifetime and a bounded number of writes, and both are reached silently.

• After 30 days, a conversation and its items are removed and every route on it returns 404. Long-lived agents must create a new conversation rather than reusing one indefinitely.
• 1,000 requests that add or delete items is the per-conversation ceiling; a response bound to a conversation counts as one, whatever its number of output items. Past it, a listing stops early rather than adding failing: the gateway reads at most 1,000 invocation steps, and a single large item spans several. Start a new conversation, seeding it with the items you still need.
• 503 saying the API is not available on the current server means the IAM role is missing the Bedrock Session Storage permissions, including bedrock:UpdateSession, which only the metadata update uses — a deployment created before conversations shipped fails on POST /v1/conversations/{id} alone. The client message is the same whichever one is absent; the server log names it.
• Items added by a streamed response appear when the stream ends, not while it runs; a client that reads them from a callback fired on the terminal event must wait for the stream to close.

Per-user attribution reaches the AWS bill through two AWS-side steps that are easy to miss, and neither is instant:

• The Cost and Usage Report export must include caller identity. Create a Data Exports CUR 2.0 export with Include caller identity (IAM principal) allocation data enabled; an existing export cannot be changed and must be re-created. The identity then appears in line_item_iam_principal as assumed-role/<role>/<session>.
• The session tag must be activated as a cost allocation tag, in the AWS Billing console under Cost allocation tags, filtered by type IAM principal. It is only listed there after that identity has made at least one call, and takes up to 24 hours to appear in Cost Explorer.
• Requests that identify no end user are billed to the server, by design. The request log's aws_role_session_name field is absent on exactly those requests — use it to find the clients that send no identifier, then enable AWS_BEDROCK_USER_ROLE_REQUIRE_IDENTITY.
• Only model invocations are attributed. Video generation, guardrail evaluations, speech, transcription and translation stay on the server's own identity.

Model calls run under a session of AWS_BEDROCK_USER_ROLE_ARN, and a session that cannot be opened fails the request rather than silently falling back to the server's identity. The server also reports this at startup, in the server_warnings field of its start log event. Five causes, in order of likelihood:

• The trust policy allows only sts:AssumeRole. Tagging the session is a separate action: add sts:TagSession to both the trust policy of the end user role and the server's own policy — see Per-User Cost Attribution IAM. Setting AWS_BEDROCK_USER_ROLE_TAG_KEY to null removes the need for it, at the cost of Cost Explorer grouping.
• The role was just created. A new or newly-edited trust policy takes a few seconds to propagate; a task started immediately after logs the startup warning and recovers on its own.
• 403 on every request: the end user role lacks bedrock:InvokeModel or bedrock:InvokeModelWithResponseStream, or its Resource list misses an ARN form requests actually reach. A cross-region inference profile also needs arn:aws:bedrock:*::foundation-model/... for every Region it routes to, and an application inference profile, a prompt router or a prompt ARN each has to be named in its own right — see Per-User Cost Attribution IAM.
• 403 once a guardrail is configured: a guardrail applied during an invocation — AWS_BEDROCK_GUARDRAIL_IDENTIFIER, a model alias carrying one, or a request-level moderation parameter — is evaluated as part of the call the end user signed, so the end user role needs bedrock:ApplyGuardrail on the guardrail ARN as well.
• 400 naming safety_identifier: AWS_BEDROCK_USER_ROLE_REQUIRE_IDENTITY is enabled and the client sends no end user identifier. Either have the client send one, or disable that setting.

USAGE_API is off, which is the default — the endpoints exist and refuse, so this is neither a wrong path nor a rejected credential.

• Set USAGE_API=true and restart — together with CLOUDWATCH_METRICS, which publishes the metrics these endpoints are answered from: enabling USAGE_API without it fails startup rather than serving endpoints that could never answer. /v1/organization/costs additionally needs COST_TRACKING, which is only a startup warning when missing. The server log names whichever one is missing.
• Read what a query costs before turning it on: every query is billed per metric read, and enabling it also stores additional metric series.
• A 404 instead means the prefix is wrong: the routes live under ${OPENAI_ROUTES_PREFIX}/v1/organization/... — see OPENAI_ROUTES_PREFIX.
• GET /v1/usage is not served at any setting: the retired endpoint is absent from OpenAI's current API surface and from the openai SDK. Use GET /v1/organization/usage/... instead.

Nothing was refused — the query simply found no published metric in the range it covers. In order of likelihood:

• /v1/organization/costs alone is empty: COST_TRACKING is disabled, so no cost is computed and no Cost metric is published. The usage endpoints are unaffected — CLOUDWATCH_METRICS cannot be disabled while these endpoints answer at all, since USAGE_API now refuses to start without it.
• The range predates the feature: usage exists only from the moment CLOUDWATCH_METRICS was enabled, and nothing is backfilled. Per-endpoint buckets start later still — only from the moment USAGE_API was enabled, since that is what publishes the Operation dimension, so a query grouped or filtered by endpoint is empty over traffic served before it.
• A multi-region deployment reports one region's traffic: the endpoints read a single Amazon CloudWatch region, CLOUDWATCH_METRICS_REGION, defaulting to the first entry of AWS_BEDROCK_REGIONS. Point it at the region the metrics are actually ingested in.

They are an administrator surface, so the credential that calls the models is not the credential that reads them.

• A tenant API key is never accepted, whatever its scope: these endpoints report the whole deployment's consumption and spend, not one tenant's. Read them with the deployment's own API key.
• A user pool token must carry every scope named in USAGE_API_ADMIN_SCOPES — all of them, not one of them.
• With that list empty (the default) no token is accepted at all, and only the deployment's own API key may read them. Name a scope there to let an operator's token in.

A query outside these bounds is refused rather than truncated or quietly answered at a resolution it did not ask for — each one is billed by Amazon CloudWatch per metric read, and a partial answer would be indistinguishable from a quiet period.

• Too many metric series: the query matched more than USAGE_API_MAX_METRICS (500 by default, which is also CloudWatch's own per-request maximum). Narrow it with models, or ask for fewer group_by keys.
• The range is too long: it spans more than USAGE_API_MAX_RANGE_DAYS (92 days by default). Split it across several queries, or raise the setting.
• bucket_width=1m over an older range: one-minute buckets are reported for the last 15 days only, one-hour and one-day buckets for the last 455. Request 1h or 1d, or move the range inside 15 days.

The built-in web search tool is gated by both an IAM permission and a server setting, and each failure looks different.

• The model answers from its training data and says it could not search: the task role is missing bedrock-websearch:InvokeSearch / bedrock-websearch:InvokeFetch. Add them — see Web Search IAM. The request itself still succeeds, so this shows up as a weak answer rather than an error.
• 400 on external_web_access: the request asked for a value the server does not allow. By default searches stay inside the AWS boundary; set AWS_BEDROCK_EXTERNAL_WEB_ACCESS to change what the server does, or AWS_BEDROCK_ALLOW_EXTERNAL_WEB_ACCESS_OVERRIDE to let requests choose. It travels as an extra model parameter (a top-level request field), not as a field of the tool: a client that sets it on the tool changes nothing.
• 400 on external_web_access saying it is not available with this model: the override is enabled, but only the models that serve web search natively — the OpenAI GPT-5.x family — take a web access choice per request. Everywhere else the parameter must match the configured value, and is refused rather than accepted and ignored. Send it only with those models, or drop it.
• External web access was enabled but results still look cached: bedrock-websearch:ExternalWebAccess is missing from the task role, so the search falls back to the Amazon Bedrock web index.
• 400 naming filters.allowed_domains or user_location: the model's own search cannot restrict which sources it uses, and running it unrestricted would answer from the very sources the request excluded. Drop the restriction, or send the request to a model that serves web search natively — the OpenAI GPT-5.x family. web_search_options.user_location on /v1/chat/completions is refused for the same reason.
• 400 saying web_search_options is not available with this model: the Chat Completions parameter reaches the model's own search, and this model runs none — answering it without searching would return something other than what was asked. Send it to an Amazon Nova model that supports web grounding, or ask for the search with the web_search tool of /v1/responses.
• The tool is rejected outright: web search is served on /v1/responses for the OpenAI GPT-5.x family, in us-east-1, us-east-2 and us-west-2. On /v1/messages and /v1/chat/completions it is not available for these models.

The request succeeds and the model answers plausibly, but no MCP server was ever contacted. The MCP connector asks the model to act as an MCP client during the turn, and the models this API serves do not open those connections. The connector is therefore accepted and ignored rather than refused, so nothing in the response says it was dropped — no setting enables it.

• The server log carries a warning naming exactly what was ignored (mcp_servers, mcp_toolset), correlated with x-request-id — the caller sees an ordinary 200, so that line is the only place this is reported.
• Run the MCP client yourself: connect to the server, declare its tools in tools, and return each result as a tool_result block. Every other tool in tools is kept and behaves normally.
• A tool_choice is dropped with the toolsets when they were the only entries in tools, and a cache_control breakpoint carried by a dropped mcp_toolset goes with it — move the breakpoint to a tool that survives, or the cached prefix is shorter than the one the request paid to write.
• mcp_tool_use and mcp_tool_result blocks replayed from a connector-enabled transcript are read as an ordinary tool_use and tool_result, so a repeated call comes back as a plain tool_use block for the client to run.
• Unrelated to this deployment being an MCP server itself, which is the opposite direction: that lets an AI agent call these endpoints as tools, and is unaffected.

Both are Anthropic Messages features whose backing runs outside this gateway, and they fail in opposite ways — one loudly, one silently.

• 400 The 'browser_toolset_20260801' toolset is not available for this model. (computer_toolset_* reads the same): the toolsets run only on Bedrock Mantle-served Claude models, which forward them to the upstream Messages API. A Converse-served model, and a Mantle model whose request is converted to an OpenAI shape, refuse them rather than answer without the browser or the desktop. A replayed browser_state tool result is refused the same way, on the same paths.
• AWS_BEDROCK_EXTERNAL_WEB_ACCESS neither enables nor blocks a toolset: it governs the built-in web search only. Where a Mantle Claude model serves the toolset, turning AWS_BEDROCK_MANTLE_ENABLED off is what leaves no path serving it.
• container is accepted and the skills are never loaded: both the identifier string and the {id, skills} object Agent Skills requests use validate, and neither reaches any backend — no container is created or returned, and the response answers as if the field had not been sent. Nothing reports it: unlike the MCP connector above, no warning is logged, so an answer that ignored a skill's instructions is the only signal.

The Ollama-compatible endpoints answer at /api/* on the deployment's base URL, and the models they list are this deployment's, not the ones a local Ollama had pulled.

• Point the client at the deployment URL followed by OLLAMA_ROUTES_PREFIX — https://your-host/ollama by default, not https://your-host/v1. Set OLLAMA_ROUTES_PREFIX to an empty value to mount at the root instead, for a drop-in swap with a stock Ollama host.
• Send the deployment's API key as a Bearer token: a local Ollama needs no credentials, so a client configured against one usually has no field filled in, and every endpoint here answers 401 without it.
• Choose a model from GET /api/tags. A name learned from ollama.com such as llama3.2:3b is not served and answers 404; a trailing :latest on a name that is served is accepted.
• A client that probes its base URL for Ollama is running is answered at the routes prefix — https://your-host/ollama, with or without a trailing slash — not at /, which serves the deployment's own root document. With OLLAMA_ROUTES_PREFIX set to an empty value the two collide and the probe is not served at all; have the client probe GET /api/version instead.

Tokens per second is computed from eval_count divided by eval_duration, and eval_duration is only reported when the response streamed.

• Request the response with "stream": true (the default on /api/chat and /api/generate) and the durations are measured and reported.
• On a non-streamed response prompt_eval_duration and eval_duration are omitted, because a buffered answer carries no split between reading the prompt and generating the answer, and a number with nothing behind it would be an invention. load_duration is never reported for the same reason: nothing is loaded, since models are served on demand.
• The token counts themselves (prompt_eval_count, eval_count) and total_duration are always reported.

Both are correct. Models here are served on demand and none is stored on the deployment.

• POST /api/pull reports success immediately for any model /api/tags lists, because it is already usable — there is nothing to transfer. A model this deployment does not serve answers 404 instead.
• size is 0 and the parameter-count, quantization and format details are empty because they describe a model file that does not exist here. digest is a stable identifier derived from the model name, usable as a cache key but not a hash of any content.
• create, copy, push and delete answer 403: there is no model store for them to change, and reporting success would tell the client that state changed when nothing did.

message.thinking follows the deployment-wide reasoning setting.

• CHAT_COMPLETIONS_REASONING_FIELD set to none suppresses the reasoning text on every dialect, including this one. Set it back to reasoning_content or reasoning to have it emitted.
• think must also be set on the request; without it a model returns its answer only.
• The capabilities list from /api/show never advertises thinking, so a client gating its toggle on that list will not offer it. think can still be sent to any model — one that does not reason simply returns no thinking text.

A slow or hung generation exceeded a timeout somewhere between the model and the client. The status code tells you where: 503 comes from stdapi.ai's own gateway timeout; 504/408 come from an edge or proxy timeout in front of it.

• 503 from the gateway: Check AI_RESPONSE_TIMEOUT — it closes stalled upstream model connections; raise it for workloads with long-running generations. The request is not retried in another region: the model already ran and AWS bills it either way, so a failover would pay twice for the same generation.
• 504/408 from the edge/proxy: Check the Terraform module's alb_idle_timeout (default: 3600 s) — if you lowered it, or front the deployment with your own load balancer or reverse proxy at a shorter idle timeout, streaming responses can be cut off mid-flight before the gateway's own timeout fires. See ALB Resilience.

Either an attachment exceeds what the chosen model reads, the application-level file-size cap, or an edge control rejected the request.

• Attachments larger than … are not available on the current server means the attachment is too large to travel inside the request and there is nowhere to stage it: no region able to serve that model has an S3 bucket. Set AWS_S3_BUCKET for the first region of AWS_BEDROCK_REGIONS and an AWS_S3_REGIONAL_BUCKETS entry for every other region that may serve the model — the server log names the one that was missing. The same request succeeds unchanged once a bucket exists.
• An attached file is too large: this model accepts at most … bytes per file means that model only reads attachments sent inside the request. Send a smaller file, or choose a model that reads attachments from storage — see Attachment Size.
• The attached files are too large: this model accepts at most … bytes of attachments per request means the same, for the request as a whole: each file fits on its own, but their total does not. Split them across several requests, or choose a model that reads attachments from storage.
• A part sent as an inline JSON body must not exceed … bytes means one part of an upload session was sent in the JSON form and carried more than the 64 MiB that form may hold — it is decoded in memory, so it is capped. Send the part as a binary multipart/form-data upload, which is streamed and bounded only by S3's 5 GiB per part, or split the content into more parts. The binary form never raises this error; the official OpenAI client uses it and already splits a file at 64 MiB.
• Check MAX_INPUT_FILE_SIZE — it caps the bytes of any single file loaded into memory for model input; disabled by default, so if it's set and the error appears, raise it or reduce the input size.
• If the deployment sits behind the Terraform module's WAF (alb_waf_enabled=true), check for a SizeConstraintStatement rule on the request body — see Request Size & Resource Limits.
• If fronted by Amazon API Gateway instead of an ALB, remember its hard 10 MB payload limit.

The API key is missing, wrong, or not configured.

• Pass the key in the Authorization: Bearer <key> header (OpenAI-style) or X-API-Key header.
• Retrieve the generated key with terraform output -raw api_key.
• If api_key_create = true was not set, no API key is configured and requests pass through without authentication by default (useful for testing behind IP-restricted ALB, not for production).
• See Authentication & Security for all options.

stdapi.ai's own AWS credentials are missing, invalid, or expired — AWS returns UnrecognizedClientException, InvalidSignatureException, or ExpiredTokenException, which stdapi.ai maps to HTTP 401 with error type authentication_error. This is distinct from the client-facing API-key 401 above (which concerns your Authorization / X-API-Key header).

• Locally: refresh with aws sso login (or update AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN) and restart the container.
• On ECS: confirm the task is assuming its IAM role rather than relying on stale static keys.

The Anthropic SDK uses a different auth header than OpenAI.

• Use x-api-key: <your-key> (not Authorization: Bearer).
• Set the base URL to https://<your-endpoint>/anthropic (not /v1).
• See API Overview → Anthropic-Compatible API.

The response body is always the same opaque Unauthorized; the check that failed is in the server's request log (error_detail). Work through the checks in order:

• Wrong app client: the token's app client must be listed in AWS_COGNITO_CLIENT_IDS.
• Wrong token: an identity token is rejected unless AWS_COGNITO_ACCEPT_ID_TOKEN is enabled. Send the access token.
• Wrong issuer: pools on the Essentials and Plus tiers can issue https://issuer-cognito-idp.<region>.amazonaws.com/.... Set AWS_COGNITO_ISSUER_TYPE to updated for those, and confirm the pool ID matches the pool that minted the token.
• Missing scope: a token obtained by signing in with a username and password carries only aws.cognito.signin.user.admin. Clear AWS_COGNITO_REQUIRED_SCOPES, or have clients obtain tokens from the pool's OAuth 2.0 token endpoint.
• Expired token: tokens are accepted up to one minute past expiry only. Refresh the token, and check the container clock if expiry errors are constant.
• See Authentication & Security → Amazon Cognito User Pool Tokens.

A half-applied credential configuration is refused rather than accepted, so the deployment never runs unauthenticated by accident. The startup log's error_detail names the exact rule:

• Missing allowlist: AWS_COGNITO_CLIENT_IDS is required with a pool.
• Setting without a pool: any other AWS_COGNITO_* variable requires AWS_COGNITO_USER_POOL_ID.
• Mode conflict: AUTHENTICATION_MODE must not demand a method that is unconfigured, nor ignore one that is configured — use any to accept both.
• Empty API key: the SSM parameter or Secrets Manager secret named by API_KEY_SSM_PARAMETER or API_KEY_SECRETSMANAGER_SECRET exists but holds an empty value — populate it, or unset the setting to run without an API key deliberately.
• API key secret is not text: the secret named by API_KEY_SECRETSMANAGER_SECRET holds binary data, or the field selected by API_KEY_SECRETSMANAGER_KEY is not a string. Store the key as a plain string, or as a JSON object whose selected field is a string.
• Signing keys unreachable: the pool's public keys are read at startup over HTTPS. Check the pool ID, and that the task can reach the internet or a suitable endpoint for outbound HTTPS.

Discovery is off until it is configured, and a client that finds nothing falls back to asking the user for a key.

• Nothing published: GET /.well-known/oauth-protected-resource answering 404 means OAUTH_RESOURCE_IDENTIFIER is unset. That is the default, not a fault — set it to turn discovery on. With an AWS_COGNITO_USER_POOL_ID configured it is the only setting needed; otherwise set OAUTH_AUTHORIZATION_SERVERS too.
• Nothing in the challenge: a 401 carrying a bare WWW-Authenticate: Bearer means the same thing. Once configured it also carries resource_metadata="…" and, with OAUTH_SCOPES_SUPPORTED, scope="…".
• Browser-hosted client: a client running in a page reads the document cross-origin and needs its origin in CORS_ALLOW_ORIGINS.
• The client asks for /.well-known/openid-configuration: stdapi.ai is a resource server and deliberately does not serve it; the client should follow authorization_servers to the issuer's own document.
• See Authentication & Security → Authentication Discovery for Agents.

Clients compare the published resource against the URL they dialled character by character, so any difference aborts the flow.

• Scheme: set OAUTH_RESOURCE_IDENTIFIER to https://… when clients reach the deployment over TLS, even though the container itself listens on plain HTTP behind the load balancer.
• Host: it must be the public hostname clients use, not the internal service or task address.
• Port: an explicit :443 on the client's URL does not match an identifier without one, and vice versa. Drop the default port on both sides.
• Path: the identifier is an origin. Do not append /mcp, /v1, or a trailing slash — one document at the root already covers every surface of the deployment.

The three settings describe one document, so an incomplete set is refused rather than published half-formed. The startup log's error_detail names the rule:

• No authorization server: OAUTH_AUTHORIZATION_SERVERS is required with OAUTH_RESOURCE_IDENTIFIER unless AWS_COGNITO_USER_POOL_ID is set, which supplies the issuer; a document naming none leaves a client unable to obtain a token.
• Issuer contradicting the pool: with a user pool configured, the published issuers must include the pool's own — otherwise clients are sent to an authorization server whose tokens every request refuses. Leave OAUTH_AUTHORIZATION_SERVERS empty to publish exactly the pool issuer, or list it alongside the others.
• Setting without an identifier: the authorization servers and the scopes describe a document that is not published without OAUTH_RESOURCE_IDENTIFIER.
• Malformed value: the identifier is an origin with no path or query, each issuer is an https URL with no query or fragment, and a scope carries no space or quote.

A 503 for an sk-std-... credential means the key could not be checked, not that it is wrong: the tenant records in the DynamoDB table are unreachable, and the gateway fails closed rather than guessing. The server log's error_detail names what to fix:

• Missing permission: the task role needs the shared table permissions (dynamodb:GetItem in particular) on the table ARN.
• Missing or wrong table: AWS_DYNAMODB_TABLE and AWS_DYNAMODB_REGION must name the table holding the tenant records.
• Record from a newer build: during a rolling deployment, an instance on the previous version refuses records written with a newer layout; the refusals stop when the rollout completes.
• A rate-limited tenant whose counter cannot be written — the log names dynamodb:UpdateItem, or a throttled table: a limit the operator declared is never silently dropped, so the tenant's requests are refused once the slots already reserved are used. Grant the counter statement on the LIMIT#* items. The permission is probed at startup only, over the limits declared at that moment: a limit added to a tenant record later is not probed until the service restarts, and the request log's error_detail names the action either way. Anthropic-compatible routes answer 529; a tenant limited on tokens alone has no reserved requests to serve first and is refused at once.
• A record whose requests_per_minute or tokens_per_minute is not a whole number of at least 1: zero is refused rather than read as unlimited, so a malformed limit never opens the door. The log names the attribute.
• Other credential kinds are unaffected: the deployment API key and Cognito tokens keep working through the outage.

The key reached its per-tenant limit — its record's requests_per_minute or tokens_per_minute, or the deployment's TENANT_RATE_LIMIT_REQUESTS_PER_MINUTE / TENANT_RATE_LIMIT_TOKENS_PER_MINUTE. The message names the key ID and the limit; the retry-after header names the seconds left in the minute, and on OpenAI- and Anthropic-compatible routes the x-ratelimit-* or anthropic-ratelimit-* headers on every response of that key show how much of the minute is left before the refusal. The Cohere- and Ollama-compatible routes answer their own plain 429 envelope with retry-after and no rate-limit headers.

• Raise or remove the limit: the record's value wins over the deployment default, so one tenant can be widened without touching the others; a record change reaches every instance within TENANT_KEY_CACHE_SECONDS.
• Refused before the declared number of requests, on several instances: request slots are reserved per instance in batches and never returned within the minute, so a tenant whose traffic is spread thin — or moved by a scale event or a rolling deployment — can be refused below its requests_per_minute, the more so the more instances it spans. Declare the limit with headroom for the instance count, or keep the tenant's traffic on fewer instances.
• A token limit refused a request that was well under it, with nothing billed yet: on an instance that has not billed a request of the key — every instance, after a deployment or a scale-out — each request in flight counts as an eighth of the token limit, so about eight concurrent requests are admitted whatever their size and the next is refused while the minute's total reads zero. Raising the limit does not change that ratio; the refusals stop as soon as one request of the key bills on that instance. A request limit sized for the tenant's concurrency is what raises the admission bound.
• A token limit refused a request that was well under it, mid-minute: requests are admitted on the key's mean tokens per billed request and reconciled from what the model billed, so a burst of large responses in flight can exhaust the minute before their totals are known — and a token limit alone admits at most 64 requests in flight per instance, refused with the same 429. The refusal lasts until the minute ends. Pair the token limit with a request limit: it replaces the in-flight ceiling, and the pair bounds the minute's spend.
• Only one client is loud: the limit is per key, not per client — every client of the tenant shares it. Give a noisy integration its own tenant record.
• A Realtime session opened but the next connection is refused: the session's turns count against the token limit and the open session is never cut; the next handshake is what the limit refuses — and on the wire it is not a 429. The upgrade succeeds (101), then the server sends one error event of type invalid_request_error with code rate_limit_exceeded and closes with code 3000, reason invalid_request_error.rate_limit_exceeded. No retry-after reaches a WebSocket client, so the client backs off itself; only the server request log shows the 429. A session opened with an ephemeral client secret also costs a second request, for the POST /v1/realtime/client_secrets that mints it.

Declaring a per-tenant limit puts writes to the shared table on the admission path, and a request finding no slot reserved waits for one. Which requests wait follows the limit that is declared:

• A request limit of 16 or more: slots are reserved in batches that start at one and double within the minute, capped at an eighth of the limit, so the first two requests of each minute for that key on each instance wait, and a later one waits whenever the background refill has not landed yet. The rest of the minute is admitted from what the instance holds.
• A request limit below 16: an eighth of the limit rounds down to a single slot, so every request of that tenant waits for a write. Declare at least 16 where the table's round trip must stay off the request path.
• A token limit alone: one write per minute per key per instance, on the request that reads the window's totals; billed tokens are flushed in the background.

A latency step of the table's round trip on those requests is expected; more than that is the table answering slowly:

• Throttling: a provisioned table whose write capacity is below the number of limited keys times instances makes the write retry with backoff before it succeeds, or fails it — the request log then names dynamodb:UpdateItem with ProvisionedThroughputExceededException. The Terraform module creates the table on-demand, which scales with the load.
• A distant region: AWS_DYNAMODB_REGION is far from the instances; keep the table in the deployment's own region.
• A table that does not answer is given up on within seconds, and the request is then refused with 503 rather than served unlimited.

The server mints pending tenants at startup and then once a minute, so the parameter appears within about a minute of terraform apply — if it can:

• Missing permission: minting needs ssm:PutParameter and ssm:GetParameter on the delivery prefix, and the shared table permissions to record the hash. The refusal is in the server log.
• A parameter already exists at that name with something that is not this tenant's key: the server refuses to adopt it. Delete the parameter and let the next cycle mint a fresh key.
• The feature is off: TENANT_API_KEYS must be true on the running service, not only in the table.

Each instance caches a validated key for TENANT_KEY_CACHE_SECONDS — 60 seconds by default — so a revocation, a disabled = true or a scope change takes up to that long to reach every instance. That window is the documented trade against a table read per request; lower the setting if a minute is too long, 0 disables the cache entirely.

By design, for TENANT_KEY_ROTATION_OVERLAP_SECONDS — 7 days by default: the key a rotation superseded stays readable as the secret's AWSPREVIOUS version and keeps authenticating, so a client that has not re-read its secret yet is not locked out. The cutoff is exact, whatever the cache: the key stops at the end of the window on every instance at once. Only the last superseded key is kept, so a further rotation inside the window — raising key_generation again — retires it there and then.

• Shorter window: lower the setting; 0 refuses the superseded key as soon as the new one is promoted to AWSCURRENT.
• Compromised key: set disabled = true on the tenant, which refuses both keys within TENANT_KEY_CACHE_SECONDS, whatever the overlap.

A rotation writes the new key as a pending version of the tenant's secret, records it, then moves the AWSCURRENT label — the gateway accepts the new key before it becomes current, so a client re-reading the secret is never handed a key that is refused. When a key nonetheless does not rotate, or the rotated key answers 401:

• Keys are delivered through Parameter Store: rotation needs the Secrets Manager store — TENANT_KEY_SECRETSMANAGER_PREFIX on the service, or tenant_key_rotation_days / key_generation in the Terraform module. A key_generation raised without it is reported once in the server log and ignored.
• Missing permission: the task role needs the rotation permissions on the secret prefix, and kms:GenerateDataKey and kms:Decrypt on the key encrypting the secrets. The refusal names the action in the server log; the key keeps working unchanged.
• The rotation is not due: TENANT_KEY_ROTATION_DAYS counts from the mint or the last rotation, and key_generation only rotates when it exceeds the generation recorded at the previous rotation — raise it again for another rotation.
• The client read the secret before the promotion: a read that names no version stage answers AWSCURRENT, which moves within a second of the record being written; a client that reads AWSPENDING explicitly can read a key that is not recorded yet. Read the current version, or nothing more specific.
• A mixed fleet: instances on a release without rotation accept the new key but refuse the superseded one at once.
• The promotion was not confirmed: the server log warns that a version could not be confirmed as AWSCURRENT, naming the secret and the key. The superseded key keeps working and the promotion is retried at the next reconciliation.

The fixed message "The AWS credential registered for this API key could not be used" means the gateway could not open (or keep) a session of the tenant's role; "Your AWS account does not have access to this model" means the session opened but the tenant's account refused the invocation. The full AWS detail is in the server log only. In order of likelihood:

• Trust policy: the tenant role must trust the deployment's account with sts:AssumeRole, conditioned on the exact ExternalId the server minted — read it from the external_id attribute of the tenant's secret#<key id> record. A wrong or missing ExternalId is indistinguishable from a revoked trust, on purpose.
• Gateway-side permission: the task role needs sts:AssumeRole on the tenant role.
• Model access in the tenant's account: the tenant must have been granted access to the model in its account, in the serving Region — including enabling opt-in Regions a cross-Region profile routes to.
• The role's own policy: it must allow the Bedrock invocation actions on the model (and its inference profiles).

A 503 "…could not be used right now. Retry the request" is not one of these: it means AWS STS itself was throttled or unreachable, or the deployment's own session had expired. Nothing on the tenant's side is wrong, and the request is worth retrying — a client SDK retries it on its own.

A declared role is never silently ignored — the gateway refuses the key rather than billing the deployment for a tenant that expects its own account. The server log names which of these it is:

• The feature is off: TENANT_AWS_CREDENTIALS must be true, or the attribute removed.
• The ExternalId is not minted yet: for a tenant created before this feature existed, the server mints one within a minute of the role being declared; the refusal covers that window.
• The ARN is malformed: aws_role_arn must be an IAM role ARN, arn:aws:iam::<account>:role/<name>.
• A guardrail is configured: the combination is refused at startup — see the incompatibility.

A model outside a tenant's scope answers the standard model_not_found, indistinguishable from a model that does not exist — by design, so the catalogue leaks nothing. Check the tenant's models_allow and models_deny patterns against the resolved model ID (after aliases — the ID the working requests are logged with), and remember an empty models_allow list allows nothing, while an absent one restricts nothing. GET /v1/models is not filtered per tenant, so a model appearing there can still be refused at invocation.

Authentication is enforced by the ALB listener, not stdapi.ai.

• Verify the OIDC issuer URL, client ID, client secret, and redirect URI in the ALB listener rule.
• For Cognito, confirm the app client is configured as a "confidential" client with a client secret.
• See Authentication & Security → via Application Load Balancer (ALB).