Troubleshooting¶

Your AWS profile does not have sufficient permissions. The stdapi.ai Terraform module provisions IAM roles, KMS keys, ECS, ALB, Route53 records, an optional WAF, and (for some samples) RDS and ElastiCache.

• Use an administrator-level AWS profile for the evaluation deployment.
• Recommended: deploy into a sandbox/non-production AWS account first, then replicate into your target account with scoped-down principals once validated.
• Verify your active identity: aws sts get-caller-identity.

Terraform completed but the ECS service is still coming up. The ALB returns 503 Service Unavailable until tasks pass health checks.

• Wait 2–3 minutes after terraform apply completes.
• Check ECS service status: aws ecs describe-services --cluster <cluster> --services <service>.
• Check task logs in CloudWatch: /aws/ecs/<service-name>.

The AWS provider uses the region/profile from your environment, not a Terraform variable.

• Confirm before applying:

  aws sts get-caller-identity
  aws configure get region
  

• Set explicitly with AWS_PROFILE=... AWS_REGION=... terraform apply if needed.

The ElastiCache Valkey cache occasionally fails to create when the target availability zone is out of capacity.

Error: waiting for ElastiCache Replication Group ... create: unexpected state 'create-failed',
wanted target 'available'

• Remove the failed Valkey cache from the ElastiCache console (disable backups first, then wait for full deletion) and re-run terraform apply.
• If the problem persists, change node_type in valkey.tf (e.g. cache.t4g.micro → cache.t3.micro) and retry.

SELinux blocks container access to ~/.aws without a relabel.

• Add :z (or :Z for exclusive use) to the volume and --userns=keep-id:

  podman run --rm -p 8000:8000 \
    --userns=keep-id \
    -v ~/.aws:/home/nonroot/.aws:ro,z \
    -e AWS_BEDROCK_REGIONS=us-east-1,us-west-2 \
    -e ENABLE_DOCS=true \
    ghcr.io/stdapi-ai/stdapi.ai-community:latest
  

• See Local Development for the full run command.

The ECS service is still starting up. Health checks take a few minutes.

• Wait 2–3 minutes after deployment and refresh.
• Check the ALB target group health in the AWS console.
• If it persists longer than 5 minutes, inspect CloudWatch logs for the ECS task.

Host header validation applies to /health as well. A load balancer health check addresses the target directly, so its Host header carries the target's IP address — which a list of domain names does not match, and every probe is answered with 400. The target group stays unhealthy and the ALB keeps returning 503.

• Prefer host validation at the load balancer: an ALB listener rule on the Host header, with TRUSTED_HOSTS left unset.
• If the application-level allow-list is required, include the address the health check actually sends.
• The container's own HEALTHCHECK is unaffected: it derives its Host header from TRUSTED_HOSTS. See TRUSTED_HOSTS.

The ALB uses the default *.elb.amazonaws.com domain, which has no trusted certificate. This is expected and safe to bypass for testing.

• For a production-grade certificate, configure a custom domain — the Terraform module supports ACM-managed certificates via alb_domain_name.

The API key is missing, wrong, or not configured.

• Pass the key in the Authorization: Bearer <key> header (OpenAI-style) or X-API-Key header.
• Retrieve the generated key with terraform output -raw api_key.
• If api_key_create = true was not set, no API key is configured and requests pass through without authentication by default (useful for testing behind IP-restricted ALB, not for production).
• See Authentication & Security for all options.

The gateway reached Amazon Bedrock, but the ECS task role (or your local AWS credentials) lacks permission for the requested action. AWS returns AccessDeniedException, which stdapi.ai maps to HTTP 403 with error type permission_error. This is an IAM misconfiguration, not a client API-key problem.

• Confirm the task role grants bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream (and the bedrock:Converse* actions) for the target model ARNs.
• For models invoked through an inference profile, allow both the profile ARN and the underlying foundation-model ARNs in the policy.
• Some models require one-time activation in the Bedrock console → Model access page before they can be invoked.
• Audio, embeddings, and file features need permissions for the relevant services (Polly, Transcribe, Translate, Comprehend, S3) — see Configuration → IAM Permissions for the full IAM reference.

stdapi.ai's own AWS credentials are missing, invalid, or expired — AWS returns UnrecognizedClientException, InvalidSignatureException, or ExpiredTokenException, which stdapi.ai maps to HTTP 401 with error type authentication_error. This is distinct from the client-facing API-key 401 above (which concerns your Authorization / X-API-Key header).

• Locally: refresh with aws sso login (or update AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN) and restart the container.
• On ECS: confirm the task is assuming its IAM role rather than relying on stale static keys.

The model ID isn't available in your configured region(s).

• Start with a low-friction model to confirm the pipeline works: amazon.nova-micro-v1:0 (available in all standard Bedrock regions).
• List every discovered model with full details: GET /search_models (the default model-discovery endpoint). Filter by capability with query parameters — e.g. GET /search_models?input_modalities=IMAGE&route=/v1/chat/completions returns only vision-capable chat models. See the Search Models API reference.
• GET /v1/models is also available for strict OpenAI SDK compatibility (lighter payload, no capability metadata).
• Verify AWS_BEDROCK_REGIONS includes a region that offers the model — see the Bedrock model availability table.
• For Anthropic SDK clients, use either the full Bedrock ID (anthropic.claude-fable-5) or the Anthropic alias (claude-fable-5) — both resolve automatically.

AWS returned ThrottlingException, TooManyRequestsException, or ServiceQuotaExceededException — mapped to HTTP 429 with error type rate_limit_error. You've hit the per-region Bedrock quota.

• Add more regions to AWS_BEDROCK_REGIONS. Each region has its own independent quota — three regions ≈ triple the throughput.
• See Resilience & Failover for multi-region routing configuration.
• Check quotas in the AWS Service Quotas console for Amazon Bedrock.
• When the router put a region on a quota backoff while serving the request, the response carries a retry-after header (in seconds) telling the client exactly how long to wait — OpenAI, Anthropic and Cohere SDKs honour it automatically instead of guessing an exponential backoff, up to the 60 s ceiling their retry loops apply to server-supplied delays.

A specific model is unavailable or requests to it are rejected because your account's data retention mode is incompatible with what that model requires.

Amazon Bedrock enforces retention compatibility at invocation time: each model declares the retention modes it accepts, and if your effective mode is not among them, the request is blocked.

Common scenarios:

• Your account is set to zero data retention (none) but the model requires default or provider_data_share for safety or abuse-prevention purposes. Bedrock blocks the request to honour your retention policy. To access the model, either switch to a compatible retention mode or contact your AWS account manager to request ZDR eligibility for that specific model.
• Your account is set to default but the model exclusively requires provider_data_share (typically models with mandatory provider-side safety review). The model will appear as unavailable. Enabling provider_data_share grants access but means AWS will share your inference data with the model provider — see Data Privacy before enabling it.

Bedrock rejected the request parameters (ValidationException / BadRequestException), mapped to HTTP 400 with error type invalid_request_error — for example an unsupported parameter for the chosen model, an out-of-range value, or content that exceeds the model's limits.

• Read the message detail returned in the response (correlate with x-request-id in the server logs).
• Confirm the parameter is supported by the model — see the per-API Feature Compatibility tables.

The S3 bucket is missing, unreachable, or in the wrong region.

• The Terraform module creates the bucket automatically unless you pass your own via aws_s3_bucket.
• If you're using your own bucket: AWS_S3_BUCKET must point to a bucket in the same region as the first entry in AWS_BEDROCK_REGIONS.
• Verify the ECS task IAM role has s3:PutObject / s3:GetObject on the bucket.

Outbound traffic to AWS endpoints is blocked.

• Confirm the ECS task's security group allows outbound HTTPS (port 443).
• If using VPC endpoints (the Terraform module default), verify the endpoint security groups and policies permit traffic from the ECS task subnet.
• If ECS runs in a private subnet without VPC endpoints, confirm the NAT gateway / route table is configured.

Either the application-level file-size cap or an edge control rejected the request.

• Check MAX_INPUT_FILE_SIZE — it caps the bytes of any single file loaded into memory for model input; disabled by default, so if it's set and the error appears, raise it or reduce the input size.
• If the deployment sits behind the Terraform module's WAF (alb_waf_enabled=true), check for a SizeConstraintStatement rule on the request body — see Request Size & Resource Limits.
• If fronted by Amazon API Gateway instead of an ALB, remember its hard 10 MB payload limit.

A slow or hung generation exceeded a timeout somewhere between the model and the client. The status code tells you where: 503 comes from stdapi.ai's own gateway timeout; 504/408 come from an edge or proxy timeout in front of it.

• 503 from the gateway: Check AI_RESPONSE_TIMEOUT — it closes stalled upstream model connections; raise it for workloads with long-running generations. The request is not retried in another region: the model already ran and AWS bills it either way, so a failover would pay twice for the same generation.
• 504/408 from the edge/proxy: Check the Terraform module's alb_idle_timeout (default: 3600 s) — if you lowered it, or front the deployment with your own load balancer or reverse proxy at a shorter idle timeout, streaming responses can be cut off mid-flight before the gateway's own timeout fires. See ALB Resilience.

The Anthropic SDK uses a different auth header than OpenAI.

• Use x-api-key: <your-key> (not Authorization: Bearer).
• Set the base URL to https://<your-endpoint>/anthropic (not /v1).
• See API Overview → Anthropic-Compatible API.

Authentication is enforced by the ALB listener, not stdapi.ai.

• Verify the OIDC issuer URL, client ID, client secret, and redirect URI in the ALB listener rule.
• For Cognito, confirm the app client is configured as a "confidential" client with a client secret.
• See Authentication & Security → via Application Load Balancer (ALB).