Batch API¶

Run a large set of chat completion or embedding requests asynchronously, at a lower price than the synchronous API, through the OpenAI Batch API shape.

A batch is created from a file of requests, runs without a connection held open, and is read back from the result files it produces — exactly the OpenAI workflow, so the official OpenAI SDKs work by changing the base URL.

Why Choose the Batch API?¶

Available Endpoints¶

Feature Compatibility¶

Model Support¶

Any chat or embedding model available for batch inference in your configured Amazon Bedrock regions can be used — the same identifiers as Chat Completions and Embeddings. To shortlist them, call search_models with route=openai_chat_completion&batch=true, or route=openai_embedding&batch=true for embeddings; each entry also carries a batch field.

A model that cannot serve batched requests is refused when the batch is created, naming the model. A model this deployment normally serves through another Amazon Bedrock endpoint is batched under the identifier the batch endpoint knows it by, so it needs nothing from you.

Workflow¶

1. Upload the requests¶

One JSON object per line, uploaded with purpose="batch". Every line names the same model, targets the same endpoint, carries a unique custom_id, and holds the request itself in body.

{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "amazon.nova-micro-v1:0", "messages": [{"role": "user", "content": "Summarize: ..."}]}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "amazon.nova-micro-v1:0", "messages": [{"role": "user", "content": "Summarize: ..."}]}}

Embedding requests are written the same way, against /v1/embeddings:

{"custom_id": "doc-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "amazon.titan-embed-text-v2:0", "input": "First passage of the corpus"}}
{"custom_id": "doc-2", "method": "POST", "url": "/v1/embeddings", "body": {"model": "amazon.titan-embed-text-v2:0", "input": "Second passage of the corpus"}}

Both samples above are abridged: a real file needs at least 100 requests, the minimum a batch carries.

from openai import OpenAI

client = OpenAI(base_url="https://your-host/v1", api_key="...")

requests_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")

2. Create the batch¶

batch = client.batches.create(
    input_file_id=requests_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

Example request (curl):

curl -X POST "https://your-host/v1/batches" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-06fvfg3lbdqarbad8kbo55g0sg5h3s4a",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

Example response:

{
  "id": "batch_06fvfg3lbdqarbad8kbo55g0sg5h3s4a",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "input_file_id": "file-06fvfg3lbdqarbad8kbo55g0sg5h3s4a",
  "completion_window": "24h",
  "status": "validating",
  "created_at": 1786568013,
  "expires_at": 1786654413,
  "request_counts": {"total": 100, "completed": 0, "failed": 0},
  "model": "amazon.nova-micro-v1:0"
}

3. Poll until it ends¶

batch = client.batches.retrieve(batch.id)
print(batch.status, batch.request_counts)

4. Read the results¶

results = client.files.content(batch.output_file_id).text

Each line pairs a custom_id with the completion it produced:

{"id": "batch_req_9f2c...", "custom_id": "req-1", "response": {"status_code": 200, "request_id": "batch_req_9f2c...", "body": {"id": "chatcmpl-req-1", "object": "chat.completion", "choices": [{"index": 0, "message": {"role": "assistant", "content": "..."}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 22, "completion_tokens": 9, "total_tokens": 31}}}, "error": null}

Requests that failed are collected in a separate file, named by error_file_id.

batch = client.batches.create(
    input_file_id=requests_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    output_expires_after={"anchor": "created_at", "seconds": 7 * 24 * 3600},
)

Limits¶

A batch below the minimum, or past any of these caps, is refused when it is created and the message names the shortfall, rather than accepted and failed later.

Prerequisites¶

The Batch API is disabled until the deployment declares an AWS IAM service role that Amazon Bedrock assumes to read the requests and write the results:

• AWS_BEDROCK_BATCH_ROLE_ARN — the service role.
• AWS_S3_BUCKET — the bucket holding the batch data.
• AWS_S3_BATCHES_PREFIX — the prefix it is stored under.

The permissions the role and the server need are listed in IAM Permissions. While the role is unset, every batch endpoint answers 503.

Billing¶

Batched requests are billed at the published batch rate for the model, roughly half the on-demand rate. Usage is recorded once, when the batch ends. See Cost Management.

See Also¶

• Files API — upload the requests, download the results
• Chat Completions API — the per-request body
• Embeddings API — the per-request body of an embeddings batch
• Message Batches API — the Anthropic-shaped equivalent
• Configuration — enabling batches