Batch API¶

Run a large set of chat completion or embedding requests asynchronously, at a lower price than the synchronous API, through the OpenAI Batch API shape.

A batch is created from a file of requests, runs without a connection held open, and is read back from the result files it produces — exactly the OpenAI workflow, so the official OpenAI SDKs work by changing the base URL.

At a glance¶

• Four endpoints — create, list, retrieve and cancel, served by Amazon Bedrock batch inference, see Endpoints.
• The batch rate, roughly half the on-demand rate — usage is recorded once, when the batch ends, see Billing.
• 100 to 50,000 requests per batch, in up to 200 MB of JSONL — a batch outside those bounds, or holding a line that cannot be run as it stands, is refused when it is created, see What the input file is checked for.
• A 24-hour processing window, no connection held — completion_window is 24h as upstream, counted from created_at and reported as expires_at, see Listing order.
• Requests and results in your own S3 bucket — the results are read back through the Files API as output_file_id and error_file_id, with no traffic to third-party endpoints.
• No tools, json_schema, streaming or n above 1 — a batch carrying one of them is refused when it is created, see Feature compatibility.
• A few creations at a time — the create call reads and translates the whole file, so a caller past that bound is answered 429 or 503 rather than queued, see How many batches may be created at once.
• No DELETE /v1/batches/{batch_id} — deletion is served by the Anthropic Message Batches surface, for the batches created there, see Deleting a batch.

curl -X POST "$BASE/v1/batches" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-06fvfg3lbdqarbad8kbo55g0sg5h3s4a",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

Endpoints¶

Feature compatibility¶

Models¶

Any chat or embedding model available for batch inference in your configured Amazon Bedrock regions can be used — the same identifiers as Chat Completions and Embeddings. To shortlist them, call search_models with route=openai_chat_completion&batch=true, or route=openai_embedding&batch=true for embeddings; each entry also carries a batch field.

A model that cannot serve batched requests is refused when the batch is created, naming the model. A model this deployment normally serves through another Amazon Bedrock endpoint is batched under the identifier the batch endpoint knows it by, so it needs nothing from you.

A batch's own model field, and the model field of each request in the JSONL body, may name a wildcard pattern instead of an exact model. It is resolved once, when the batch is created: the concrete model that pattern meant that day is what the batch reports and runs for its whole life, even after a newer release would resolve it differently.

Listing order¶

Batches are listed most recent first, ordered by the created_at each one reports, so paging with the after cursor and sorting a page on created_at give the same sequence. Batches sharing a second are ordered by identifier, so the sequence is stable from one page to the next. created_at is the moment the batch was created, and the 24-hour processing window reported as expires_at runs from it.

Prerequisites¶

The Batch API is disabled until the deployment declares an AWS IAM service role that Amazon Bedrock assumes to read the requests and write the results:

• AWS_BEDROCK_BATCH_ROLE_ARN — the service role.
• AWS_S3_BUCKET — the bucket holding the batch data.
• AWS_S3_BATCHES_PREFIX — the prefix it is stored under.

The permissions the role and the server need are listed in IAM Permissions. While the role is unset, every batch endpoint answers 503.

Billing¶

Batched requests are billed at the published batch rate for the model, roughly half the on-demand rate. Usage is recorded once, when the batch ends. See Cost Management.

Limits and behaviour to know¶

Caps enforced at creation¶

A batch below the minimum, or past any of these caps, is refused when it is created and the message names the shortfall, rather than accepted and failed later.

What the input file is checked for¶

The whole file is read and checked while POST /v1/batches is answered. A file that cannot be batched as it stands is refused there: the call returns 400 naming the line at fault, and no batch is created — no request runs, and there is nothing to poll or to cancel.

The file as a whole is read the same way: one uploaded with a purpose other than batch, one holding no request at all, one past the caps above, and one naming more than one model are each refused by the create call.

How many batches may be created at once¶

POST /v1/batches does its work while the call is open: the input file is read and every request in it is translated, which fetches whatever those requests point at — an image URL, a document, an audio file. That work is bounded so a few large files cannot exhaust the server, and the bound is answered rather than queued.

The per-key cap counts tenant API keys only: it is a fairness rule between tenants, and a deployment key is the operator's own traffic, held by the server-wide slots like everything else.

Both are refusals of the create call alone: nothing is read, no batch is created, and no other endpoint is affected. Both are safe to retry, and the message names what to wait for. Neither carries a retry-after header.

A creation that is admitted is never held up by another for longer than one translation wave: the server-wide slots are given back between waves, not at the end of a file, so a 50,000-request batch whose inputs are slow to fetch shares the server with every other caller instead of owning it until it is done. A client that disconnects while its creation waits releases its place.

Deleting a batch¶

This surface serves no DELETE /v1/batches/{batch_id}: a batch record created here keeps its place in the bucket for as long as the bucket holds it. Its result files are ordinary Files API objects, so DELETE /v1/files/{file_id} removes them, and output_expires_after set at creation expires them on their own.

The Anthropic Message Batches API does serve DELETE /v1/messages/batches/{id}, but a batch record is readable only through the API it was created with, so that endpoint deletes Anthropic-created batches only.

Try it¶

1. Upload the requests¶

One JSON object per line, uploaded with purpose="batch". Every line names the same model, targets the same endpoint, carries a unique custom_id, and holds the request itself in body.

{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "amazon.nova-micro-v1:0", "messages": [{"role": "user", "content": "Summarize: ..."}]}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "amazon.nova-micro-v1:0", "messages": [{"role": "user", "content": "Summarize: ..."}]}}

Embedding requests are written the same way, against /v1/embeddings:

{"custom_id": "doc-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "amazon.titan-embed-text-v2:0", "input": "First passage of the corpus"}}
{"custom_id": "doc-2", "method": "POST", "url": "/v1/embeddings", "body": {"model": "amazon.titan-embed-text-v2:0", "input": "Second passage of the corpus"}}

Both samples above are abridged: a real file needs at least 100 requests, the minimum a batch carries.

from openai import OpenAI

client = OpenAI(base_url="https://your-host/v1", api_key="...")

requests_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")

2. Create the batch¶

batch = client.batches.create(
    input_file_id=requests_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

Example request (curl):

curl -X POST "$BASE/v1/batches" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-06fvfg3lbdqarbad8kbo55g0sg5h3s4a",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

Example response:

{
  "id": "batch_06fvfg3lbdqarbad8kbo55g0sg5h3s4a",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "input_file_id": "file-06fvfg3lbdqarbad8kbo55g0sg5h3s4a",
  "completion_window": "24h",
  "status": "validating",
  "created_at": 1786568013,
  "expires_at": 1786654413,
  "request_counts": {"total": 100, "completed": 0, "failed": 0},
  "model": "amazon.nova-micro-v1:0"
}

3. Poll until it ends¶

batch = client.batches.retrieve(batch.id)
print(batch.status, batch.request_counts)

4. Read the results¶

results = client.files.content(batch.output_file_id).text

Each line pairs a custom_id with the completion it produced:

{"id": "batch_req_9f2c...", "custom_id": "req-1", "response": {"status_code": 200, "request_id": "batch_req_9f2c...", "body": {"id": "chatcmpl-req-1", "object": "chat.completion", "choices": [{"index": 0, "message": {"role": "assistant", "content": "..."}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 22, "completion_tokens": 9, "total_tokens": 31}}}, "error": null}

Requests that failed are collected in a separate file, named by error_file_id.

batch = client.batches.create(
    input_file_id=requests_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    output_expires_after={"anchor": "created_at", "seconds": 7 * 24 * 3600},
)

Next steps¶

Next: Files API · Chat Completions API · Embeddings API · Message Batches API, the only surface with a delete endpoint