AWS Services and Regions¶

Which AWS regions serve your models, how a request fails over between them, and how the shared AWS clients behave. Part of the Configuration Guide.

Settings Summary¶

AWS Client¶

AWS AI Services¶

Resilience & Failover¶

Bedrock Mantle¶

General Configuration¶

Settings shared by every AWS service client the gateway opens: retry pacing, connection pool size, and connection timeout.

AWS_ADAPTIVE_RETRY¶

Purpose : Enable adaptive retry mode that adjusts retry pacing based on observed error rates across all AWS service calls

Type : Boolean (true / false)

Default : false

Behavior : When enabled, the retry strategy dynamically responds to real-time congestion signals. If errors are occurring frequently, retries are spaced further apart to avoid amplifying load on an already-stressed endpoint. Once conditions improve, the pacing returns to normal. When disabled, retries follow a standard exponential backoff strategy with fixed intervals. Applies to all AWS services (Bedrock, S3, Polly, Transcribe, etc.).

# Default: standard exponential backoff
export AWS_ADAPTIVE_RETRY=false

# Enable adaptive retry (recommended under sustained high load)
export AWS_ADAPTIVE_RETRY=true

AWS_MAX_POOL_CONNECTIONS¶

Purpose : Maximum number of concurrent HTTP connections per AWS service client

Type : Integer (must be > 0)

Default : 50

Behavior : Each AWS service client (one per service per region) maintains its own connection pool up to this limit. Under high concurrency, increasing this value prevents requests from queuing for an available connection. Setting it too high may exhaust system file descriptors.

# Default
export AWS_MAX_POOL_CONNECTIONS=50

# High-concurrency deployment
export AWS_MAX_POOL_CONNECTIONS=100

AWS_CONNECT_TIMEOUT¶

Purpose : Timeout in seconds for establishing a connection to an AWS service endpoint

Type : Integer (must be > 0)

Default : 5

Behavior : Limits how long the client waits when opening a new connection. It also bounds, per candidate region, the time a real-time audio session (generative-voice speech, speech-to-speech) may take to become ready: connection, initial handshake and the first response together. A short value allows fast failover to another region when an endpoint is unreachable. Increase it only if you see spurious connection timeouts on high-latency networks, or real-time audio requests failing with a 503 a few seconds after they start.

# Default: 5 seconds
export AWS_CONNECT_TIMEOUT=5

# High-latency network
export AWS_CONNECT_TIMEOUT=10

Bedrock Configuration¶

Which regions serve your Bedrock models, how a request is routed and retried across them, and the Bedrock Mantle endpoint.

AWS_BEDROCK_REGIONS¶

Purpose : List of AWS regions where Bedrock models are available

Format : Comma-separated string

Default : Current AWS SDK region if not specified

Behavior : Models are discovered in the same order as the listed regions. The first region is the primary region where your server should be hosted on AWS for optimal performance. Your S3 bucket (AWS_S3_BUCKET) must also be in this region. If a model is unavailable in the primary region, subsequent regions are checked in order

export AWS_BEDROCK_REGIONS=us-east-1,us-west-2,eu-west-1

AWS_BEDROCK_CROSS_REGION_INFERENCE¶

Purpose : Enable automatic cross-region routing when a model isn't available in the primary region

Type : Boolean

Default : true

export AWS_BEDROCK_CROSS_REGION_INFERENCE=true

AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL¶

Purpose : Allow global cross-region inference routing to any region worldwide

Type : Boolean

Default : true

export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false

AWS_BEDROCK_REGION_ROUTING¶

Purpose : Automatic region routing strategy for distributing Bedrock requests across configured regions

Type : String

Default : ordered

Behavior : When multiple regions are configured in AWS_BEDROCK_REGIONS, this setting controls how requests are distributed across them. The router automatically handles quota/throttling errors and regional unavailability by temporarily avoiding affected regions

Requirement : Requires at least 2 regions in AWS_BEDROCK_REGIONS to take effect

Available strategies:

# Use ordered routing (default)
export AWS_BEDROCK_REGION_ROUTING=ordered

# Use lowest latency routing
export AWS_BEDROCK_REGION_ROUTING=lowest_latency

# Disable routing
export AWS_BEDROCK_REGION_ROUTING=disabled

AWS_BEDROCK_REGION_ROUTING_QUOTA_BACKOFF_SECONDS¶

Purpose : Duration to temporarily avoid a region after receiving a quota or throttling error

Type : Integer (seconds, must be > 0)

Default : 60

Behavior : This is the base backoff value. When a Bedrock API call fails due to quota limits (ThrottlingException, TooManyRequestsException, ServiceQuotaExceededException), the affected region is temporarily blocked. The actual delay doubles with each consecutive quota error on the same region (exponential backoff), up to a hard ceiling of 1 hour. The counter resets after a successful request. Subsequent requests are routed to other available regions during the backoff period.

# Default: 60 seconds (base value — actual delay doubles per consecutive error)
export AWS_BEDROCK_REGION_ROUTING_QUOTA_BACKOFF_SECONDS=60

# Shorter base backoff for aggressive retry
export AWS_BEDROCK_REGION_ROUTING_QUOTA_BACKOFF_SECONDS=30

# Longer base backoff for conservative approach
export AWS_BEDROCK_REGION_ROUTING_QUOTA_BACKOFF_SECONDS=120

AWS_BEDROCK_REGION_ROUTING_UNAVAILABLE_BACKOFF_SECONDS¶

Purpose : Duration to temporarily avoid a region after receiving an unavailability error

Type : Integer (seconds, must be > 0)

Default : 30

Behavior : When a Bedrock API call fails due to service unavailability (ServiceUnavailableException, ModelNotReadyException), the affected region is temporarily blocked for this many seconds. These errors are typically shorter-lived than quota limits, so the default is shorter

# Default: 30 seconds
export AWS_BEDROCK_REGION_ROUTING_UNAVAILABLE_BACKOFF_SECONDS=30

# Longer backoff for stability
export AWS_BEDROCK_REGION_ROUTING_UNAVAILABLE_BACKOFF_SECONDS=60

AWS_BEDROCK_REGION_ROUTING_MAX_QUOTA_BACKOFF_SECONDS¶

Purpose : Hard ceiling in seconds on the exponential quota backoff for a single region

Type : Integer (seconds, must be > 0)

Default : 3600 (1 hour)

Behavior : Quota backoff grows exponentially with consecutive errors (base interval × 2^n). This setting caps how large that value can become, preventing a region from being blocked indefinitely. Reduce it to allow faster recovery; increase it to keep a misbehaving region sidelined for longer.

# Default: 1 hour ceiling
export AWS_BEDROCK_REGION_ROUTING_MAX_QUOTA_BACKOFF_SECONDS=3600

# More aggressive recovery
export AWS_BEDROCK_REGION_ROUTING_MAX_QUOTA_BACKOFF_SECONDS=600

AWS_BEDROCK_REGION_ROUTING_QUOTA_STALE_FACTOR¶

Purpose : Multiplier applied to the max quota backoff to compute the stale-error reset threshold

Type : Integer (must be > 0)

Default : 2 (threshold = 2 × max quota backoff = 2 hours with defaults)

Behavior : If the most recent quota error on a region occurred more than max_quota_backoff × factor seconds ago, the consecutive-error counter is reset and the next error is treated as a fresh start rather than an escalation. A higher value keeps memory of past errors for longer before resetting the counter.

# Default: reset counter after 2× the max backoff window
export AWS_BEDROCK_REGION_ROUTING_QUOTA_STALE_FACTOR=2

# Longer memory of past errors
export AWS_BEDROCK_REGION_ROUTING_QUOTA_STALE_FACTOR=4

AWS_BEDROCK_MAX_RETRIES¶

Purpose : Maximum number of retries per Bedrock invocation, each retry escalating to the next available region

Type : Integer (must be 0 or greater; 0 disables retries)

Default : 9

Behavior : Controls the retry budget for each Bedrock API call. When region routing is enabled, every retry escalates to the next region in priority order and each candidate region is tried at most once, so the attempts are bounded by the smaller of AWS_BEDROCK_MAX_RETRIES + 1 and the number of candidate regions for the model — with 3 regions and the default 9 retries, a request makes at most 3 attempts. A region that just failed is still blocked by its own backoff, and retrying it would only extend that backoff instead of recovering the request. When routing is disabled, or the region is pinned by S3 inputs, the full budget is spent as SDK retries against that single region.

# Default: 9 retries (10 total attempts)
export AWS_BEDROCK_MAX_RETRIES=9

# Fail faster (e.g. low-latency interactive use cases)
export AWS_BEDROCK_MAX_RETRIES=3

# Deeper in-region retrying for single-region or S3-pinned requests
export AWS_BEDROCK_MAX_RETRIES=18

AWS_FAILOVER_MAX_RETRIES¶

Purpose : Maximum SDK retry attempts per candidate region for the multi-region failover services (Polly, Transcribe, Translate, Comprehend)

Type : Integer (must be 0 or greater)

Default : 2

Behavior : Only applied when a service has several candidate regions (no explicit region setting): each region attempt uses this reduced retry budget (2 retries = 3 attempts per region) before failing over, so failover across regions replaces deep in-region retrying. When a service is pinned to a single region, the standard retry budget from AWS_BEDROCK_MAX_RETRIES applies instead.

# Default: 2 retries (3 attempts) per candidate region
export AWS_FAILOVER_MAX_RETRIES=2

# Fail over after a single attempt per region
export AWS_FAILOVER_MAX_RETRIES=0

AWS_BEDROCK_MANTLE_ENABLED¶

Purpose : Expose models served by the Amazon Bedrock Mantle endpoint (OpenAI/Anthropic-compatible APIs) in addition to the classic Bedrock Converse models

Type : Boolean

Default : true

Behavior : Mantle-only models (e.g. OpenAI GPT, xAI Grok, Google Gemma 4) become available on the chat completions, responses, messages, and completions routes. Models available on both the classic bedrock-runtime endpoint and Mantle are served by bedrock-runtime unless listed in AWS_BEDROCK_MANTLE_PREFERRED_MODELS, which defaults to the OpenAI GPT-5.6 family.

Authentication requires no static secrets: short-term bearer tokens are derived automatically (SigV4-presigned) from the same AWS credential chain the server already uses, and refreshed transparently.

When Bedrock Mantle is unreachable or the IAM role lacks `bedrock-mantle` permissions, Mantle models are simply not listed and a warning is logged at startup — no configuration change required.

export AWS_BEDROCK_MANTLE_ENABLED=false

Bedrock Mantle Models feature overview

AWS_BEDROCK_MANTLE_REGIONS¶

Purpose : List of AWS regions used for Amazon Bedrock Mantle, in failover priority order

Type : Comma-separated string of AWS region identifiers

Default : The regions of AWS_BEDROCK_REGIONS that offer Bedrock Mantle

Behavior : Model availability differs per region; the served model catalog is the union of all listed regions. Region failover, quota backoff, and health tracking work exactly like classic Bedrock region routing.

export AWS_BEDROCK_MANTLE_REGIONS=us-east-1,eu-west-1

AWS_BEDROCK_MANTLE_ENDPOINT_URL¶

Purpose : Override the Amazon Bedrock Mantle endpoint URL template

Type : String — URL template with a {region} placeholder

Default : None (https://bedrock-mantle.{region}.api.aws)

Behavior : The {region} placeholder is substituted with the target region.

export AWS_BEDROCK_MANTLE_ENDPOINT_URL='https://bedrock-mantle.{region}.api.aws'

AWS_BEDROCK_MANTLE_SERVICE_HEADER¶

Purpose : Honor the x-stdapi-service: bedrock-mantle request header to route a model available on both endpoints through Bedrock Mantle for that request instead of the default bedrock-runtime serving

Type : Boolean

Default : false

export AWS_BEDROCK_MANTLE_SERVICE_HEADER=true

AWS_BEDROCK_MANTLE_PROJECT¶

Purpose : Default Amazon Bedrock Project/Workspace ID attributed to Bedrock Mantle inference requests for cost tracking and observability

Type : String — a bare project ID (e.g. proj_abc123 or default), not an ARN

Default : None (requests fall to the account's default project)

Behavior : Bedrock Projects (OpenAI-compatible APIs) and Workspaces (Anthropic Messages API) are the same underlying resource; the value is sent as the OpenAI-Project header on the Chat Completions and Responses APIs, and as the anthropic-workspace header on the Anthropic Messages API. When unset, requests fall to the account's default project — no failure.

export AWS_BEDROCK_MANTLE_PROJECT=proj_abc123

AWS_BEDROCK_ALLOW_MANTLE_PROJECT_OVERRIDE¶

Purpose : Allow a request to override the configured Mantle project via the OpenAI-Project / anthropic-workspace header

Type : Boolean

Default : false

Behavior : When true, a request may set its own project through the OpenAI-Project (Chat Completions, Responses) or anthropic-workspace (Anthropic Messages) header. When false and AWS_BEDROCK_MANTLE_PROJECT is configured, the request header is ignored and the server default applies. When no default project is configured, the request header is always honored regardless of this flag. A malformed request-supplied project ID returns 400.

export AWS_BEDROCK_ALLOW_MANTLE_PROJECT_OVERRIDE=true

AWS_BEDROCK_MODEL_REGION_RESTRICT¶

Purpose : Restrict a model to specific region(s) only, useful when a model provides important features only in certain regions

Type : JSON object (keys: Bedrock model IDs or prefixes, values: ordered lists of allowed regions)

Default : {} (empty — no model-specific region restriction)

Behavior : When set, the model is made available only in the listed regions (intersected with the regions where it is actually available), and the list order defines the routing priority when the default ordered routing strategy is used. No fallback to other regions occurs. Keys can be exact model IDs or prefixes that match the beginning of a model ID

# Restrict Nova Pro to us-east-1 for grounding support
export AWS_BEDROCK_MODEL_REGION_RESTRICT='{"amazon.nova-pro-v1:0": ["us-east-1"]}'

Other AWS Services¶

AWS_POLLY_REGION¶

Purpose : Region for Amazon Polly text-to-speech service

Default : All regions in AWS_BEDROCK_REGIONS, with per-engine regional discovery and automatic failover

Behavior : When unset, voice availability is discovered per engine in every AWS_BEDROCK_REGIONS entry at startup: an engine (Standard, Neural, Long-form, Generative) is exposed as a model when at least one candidate region offers it, and each synthesis call routes to the regions offering the requested engine and voice, failing over on region-level errors. Setting an explicit region pins Polly to that single region — engines it does not offer are then disabled.

export AWS_POLLY_REGION=us-east-1

AWS_COMPREHEND_REGION¶

Purpose : Region for the Amazon Comprehend services (language detection and toxicity moderation)

Default : All regions in AWS_BEDROCK_REGIONS, tried in order with automatic failover

Behavior : When unset, Comprehend calls try each AWS_BEDROCK_REGIONS entry in order and fail over to the next region on region-level errors (throttling, service unavailability, network issues, or a region that does not offer Comprehend or the requested operation). Setting an explicit region pins Comprehend to that single region with no failover.

export AWS_COMPREHEND_REGION=us-east-1

AWS_TRANSCRIBE_REGION¶

Purpose : Region for Amazon Transcribe speech-to-text service

Default : All regions in AWS_BEDROCK_REGIONS that have a co-located S3 bucket, tried in order with automatic failover

Behavior : Transcription jobs need an S3 bucket in the job's region. When unset, every AWS_BEDROCK_REGIONS entry with a usable bucket is a candidate — the primary region is served by AWS_TRANSCRIBE_S3_BUCKET (or AWS_S3_BUCKET), the others by their AWS_S3_REGIONAL_BUCKETS entry. On a region-level error while starting a job, the audio is server-side copied to the next candidate's bucket and the job restarts there. Setting an explicit region pins Transcribe to that single region with no failover.

export AWS_TRANSCRIBE_REGION=us-east-1

AWS_TRANSLATE_REGION¶

Purpose : Region for Amazon Translate text translation service

Default : All regions in AWS_BEDROCK_REGIONS, tried in order with automatic failover

Behavior : When unset, translation calls try each AWS_BEDROCK_REGIONS entry in order and fail over to the next region on region-level errors (throttling, service unavailability, network issues, or a region that does not offer Translate). Setting an explicit region pins Translate to that single region with no failover.

export AWS_TRANSLATE_REGION=us-east-1

Compliance and Latency Optimization¶

Strategic region configuration is critical for both regulatory compliance and performance optimization. This section provides best practice configurations for common scenarios.

GDPR and Data Residency Compliance¶

For applications serving European users, data residency regulations like GDPR may require that data processing occurs within specific geographic boundaries.

# Use only European regions
export AWS_S3_BUCKET=my-stdapi-eu-bucket
export AWS_BEDROCK_REGIONS=eu-west-1,eu-west-3,eu-central-1

# Disable global cross-region inference to prevent data routing outside Europe
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false

# Keep cross-region inference enabled for failover within EU regions
export AWS_BEDROCK_CROSS_REGION_INFERENCE=true

Latency Optimization¶

For applications prioritizing low latency and high performance, configure regions closest to your users and application infrastructure.

North America:

# Primary region for lowest latency, with fallbacks
export AWS_S3_BUCKET=my-stdapi-us-east-1-bucket
export AWS_BEDROCK_REGIONS=us-east-1,us-west-2,us-east-2

# Enable all cross-region inference for maximum model availability
export AWS_BEDROCK_CROSS_REGION_INFERENCE=true
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=true

Asia-Pacific:

# Use Asia-Pacific regions for lowest latency to APAC users
export AWS_S3_BUCKET=my-stdapi-ap-southeast-1-bucket
export AWS_BEDROCK_REGIONS=ap-southeast-1,ap-northeast-1,us-west-2

# Enable global inference for fallback to US regions if needed
export AWS_BEDROCK_CROSS_REGION_INFERENCE=true
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=true

Global Multi-Region:

# Balanced configuration with worldwide coverage
export AWS_S3_BUCKET=my-stdapi-us-east-1-bucket
export AWS_BEDROCK_REGIONS=us-east-1,eu-west-1,ap-southeast-1,us-west-2

# Enable global inference for best availability
export AWS_BEDROCK_CROSS_REGION_INFERENCE=true
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=true

Hybrid Approach: Compliance with Performance¶

Balance compliance requirements with performance needs:

# EU primary with US fallback (for model availability)
export AWS_S3_BUCKET=my-stdapi-eu-bucket
export AWS_BEDROCK_REGIONS=eu-west-1,eu-central-1,us-east-1

# Allow cross-region but restrict to specific regions only
export AWS_BEDROCK_CROSS_REGION_INFERENCE=true
export AWS_BEDROCK_CROSS_REGION_INFERENCE_GLOBAL=false