Realtime API¶

Hold a live, bidirectional speech-to-speech conversation over a single WebSocket, through the OpenAI Realtime API shape. Audio flows in both directions on the same connection: send the caller's speech as it is captured, and receive the model's spoken answer as it is generated — no request/response round trip per turn.

At a glance¶

• The same client and server event vocabulary as the OpenAI Realtime API — client.realtime.connect(model=...) works by changing the base URL — see Feature compatibility.
• Ephemeral client secrets valid 10 to 7,200 seconds — mint an ek_... credential server-side and hand it to an untrusted browser or mobile client; your API key never leaves your backend — see Ephemeral client secrets.
• A signed token, not a server-side record — any instance behind a load balancer verifies a secret minted by any other, with no shared session store — see Ephemeral client secrets.
• 24 kHz PCM by default, or G.711 at 8 kHz — audio/pcmu and audio/pcma interoperate directly with telephony and SIP media — see Feature compatibility.
• WebSocket always, WebRTC by operator opt-in, SIP never terminated here — POST /v1/realtime/calls answers 404 until REALTIME_WEBRTC_ENABLED is set — see Transports.
• The model calls your own functions — declare them on the session, run the call, hand back the result, and the answer is spoken with it — see Function calling.
• One session lasts at most 8 minutes — the connection then closes with reason session_expired, and reconnecting continues the conversation — see Session lifecycle and limits.

import asyncio

from openai import AsyncOpenAI

client = AsyncOpenAI(api_key="YOUR_API_KEY", base_url="https://your-gateway/v1")


async def main() -> None:
    async with client.realtime.connect(model="amazon.nova-2-sonic-v1:0") as connection:
        await connection.conversation.item.create(
            item={
                "type": "message",
                "role": "user",
                "content": [
                    {"type": "input_text", "text": "Say hello in one sentence."}
                ],
            }
        )
        await connection.response.create()

        async for event in connection:
            if event.type == "response.output_audio_transcript.delta":
                print(event.delta, end="", flush=True)
            elif event.type == "response.done":
                break


asyncio.run(main())

Endpoints¶

Feature compatibility¶

Voice, instructions and audio formats are fixed once the conversation opens¶

The model's voice, its system instructions, both audio formats and the session's tools are set when the conversation with the model opens, and cannot change for the rest of that session. The conversation opens on the first thing sent into it — the first input_audio_buffer.append under the default server voice activity detection, or the first input_audio_buffer.commit, response.create or conversation.item.create under manual turns — which is well before the model has answered anything.

Send session.update with these settings before sending anything else, or open a new session to change them. Afterwards, a session.update touching only other fields (turn_detection, max_output_tokens, transcription settings, and so on) is still accepted; one that would change voice, instructions, an audio format, tools or tool_choice is refused with an error event.

Answering a written turn¶

conversation.item.create carrying an input_text part adds the text to the conversation without starting an answer. Follow it with response.create, and the model answers it exactly as it answers a spoken one — the same response.output_audio.delta chunks and the same transcript — so a written nudge into a voice session ("the caller has been on hold", "wrap up now") needs no second channel.

What a response object reports¶

response.created and response.done carry the same response object, and every field the upstream API sends is present on both — a voice framework validates each frame against its own models, and a missing field is the event never arriving rather than a cosmetic difference.

Truncating an answer the caller spoke over¶

The model generates speech faster than it is played, so a caller who interrupts has heard less of the answer than was sent. Send conversation.item.truncate with the item's id, content_index: 0 and the audio_end_ms your player actually reached; the session cuts its record of that item to what was heard and answers conversation.item.truncated.

• The item's transcript is removed whole, not trimmed: nothing aligns a transcript to a position in the audio, and leaving text the caller never heard in the record is the failure this event exists to prevent.
• audio_end_ms past the end of the item's audio, an item that is not an assistant message, and an item this session never sent are each refused with an error.
• What the model itself remembers of the answer is the model's own; truncation aligns the record this session reports through conversation.item.retrieved.

Function calling¶

Declare functions on the session, and the model asks for one whenever it needs what only your application knows — an account balance, a booking, the state of a device. The call arrives as a finished answer, so the client can run it immediately; the result goes back as a conversation item, and the model speaks its reply with what the function returned.

await connection.session.update(
    session={
        "type": "realtime",
        "tools": [
            {
                "type": "function",
                "name": "get_weather",
                "description": "Get the current weather for a city.",
                "parameters": {
                    "type": "object",
                    "properties": {"location": {"type": "string"}},
                    "required": ["location"],
                },
            }
        ],
        "tool_choice": "auto",
    }
)

A call is reported as its own conversation item and its own response:

1. response.output_item.added carries a function_call item with the function's name and the call_id to answer it by. conversation.item.created and conversation.item.added announce the same item.
2. response.function_call_arguments.delta then .done carry the arguments as a JSON string. They arrive whole, in one delta.
3. response.done ends that answer with status: "completed", the function_call item in its output.
4. Send the result back with a function_call_output item naming the same call_id:

await connection.conversation.item.create(
    item={
        "type": "function_call_output",
        "call_id": call_id,
        "output": '{"temperature_c": 14, "condition": "rain"}',
    }
)

The model starts answering as soon as the function_call_output arrives — step 5 upstream, an explicit response.create, is not needed here. Sending one anyway is safe: it ends the open turn exactly as it would have, and the answer is still a single response.

Things to know before wiring an agent to it:

• Answer every call. A model waiting for a result says nothing else for the rest of the session, so return one even when the function failed — an {"error": "..."} payload is a usable answer, silence is not.
• output is free text, and a JSON object travels best. Anything that is not one is carried as the result field of one, so a function returning structured data should return it as JSON.
• Declare the tools before the conversation opens — with the voice, the instructions and the audio formats (above). A session.update changing tools or tool_choice afterwards is refused with an error.
• One call per answer. Each call ends the answer that produced it, and anything the model says afterwards is a new response; a client tracking responses sees more of them in a session that calls functions.
• Any call_id is accepted. A function_call_output is carried to the model whatever it names, so a client may replay a conversation's history into a fresh session, or resend an answer after reconnecting past the session cap, and the model itself decides what to do with it. Nothing is refused for naming a call this particular connection did not produce.

Guardrail coverage¶

When the deployment configures an Amazon Bedrock guardrail, a realtime session is checked per turn: what the caller said (as the model transcribes it) as INPUT, and each completed answer as OUTPUT. A blocked turn ends the session with a terminal error event and close code 3000.

Unlike a request/response route, the check cannot come before the content reaches the client: the model's speech is streamed while it is being generated and its transcript is only complete once the answer is over, so a blocked answer may already have been partly heard when the session ends. Written items sent with conversation.item.create are checked as INPUT before they reach the model, as on every other route. So is what a function returns; the arguments the model asks a function to run with are not checked, since they are a request for data rather than content spoken to the caller.

Models¶

Every deployment's catalog differs, and a model's own name is never guaranteed stable across accounts. Find which models serve this route:

curl "$BASE/search_models?route=openai_realtime" \
  -H "Authorization: Bearer $OPENAI_API_KEY"

Pass the returned model ID as model on the WebSocket URL, or in the session.model field of an ephemeral secret's configuration. See the Search Models API for the full filter syntax.

Authentication¶

Open the WebSocket with a credential, carried in whichever of three channels the client can use:

Any credential the deployment accepts on its HTTP routes works here — its own API key, a tenant API key where tenants are configured, an Amazon Cognito user pool access token where a pool is — and so does an ephemeral client secret (ek_...). A handshake made with a tenant key counts as one request against that tenant's rate limits, and the tenant's endpoint and model restrictions apply to /v1/realtime as to any other route. The model query parameter (/v1/realtime?model=<model id>) selects the model serving the session; it may be omitted when the credential is an ephemeral secret whose session configuration already names one. See Authentication & Security for how the API key itself is configured.

Ephemeral client secrets¶

POST /v1/realtime/client_secrets mints a short-lived credential — a value starting with ek_ — that carries a session configuration. Hand it to a browser or mobile client so it can open a session directly, without ever holding the deployment's own API key.

curl -X POST "$BASE/v1/realtime/client_secrets" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "expires_after": {"anchor": "created_at", "seconds": 60},
    "session": {
      "type": "realtime",
      "model": "amazon.nova-2-sonic-v1:0",
      "instructions": "You are a helpful support agent."
    }
  }'

{
  "value": "ek_...",
  "expires_at": 1731000060,
  "session": {
    "type": "realtime",
    "model": "amazon.nova-2-sonic-v1:0",
    "instructions": "You are a helpful support agent.",
    "audio": { "...": "..." }
  }
}

• expires_after.seconds accepts 10 to 7,200 seconds, defaulting to 600 (10 minutes) when omitted. This bounds how long the secret can be used to open a session — a session already opened with it keeps running for its own session limit.
• session accepts the same configuration a client would otherwise send in a session.update event; it is applied to every session opened with the secret. It also accepts {"type": "transcription"}, which opens sessions that only transcribe the caller and never answer out loud — the only way to ask for one, since the socket itself takes no session type. Such a session still produces response.* text events, and still costs what a spoken session costs.
• By default the carried configuration is a default, not a constraint: the client may name another model on the ?model= query string and change the configuration with its own session.update, as it can upstream. Set REALTIME_ALLOW_SESSION_OVERRIDE=false to make the model, the instructions and max_output_tokens the secret was minted with final — a mismatching ?model= is then refused at connect, and a session.update changing one of them answers an error.

Transports¶

Upstream offers a realtime session over three transports — WebSocket, WebRTC and SIP. This API always serves the WebSocket; WebRTC is an operator opt-in; SIP is never terminated here. POST /v1/realtime/calls trades an SDP offer for an answer once REALTIME_WEBRTC_ENABLED is set — see WebRTC calls terminated by the gateway for what that transport delivers and what it demands of the deployment — and answers 404 otherwise.

A browser connects to that same WebSocket¶

There is no separate browser transport to be missing. The page opens wss://<host>/v1/realtime?model=<id> itself, carrying an ephemeral client secret in the way Authentication describes, so the deployment's own API key never leaves your backend. It is two steps — mint the secret server-side, connect with it client-side — and the browser example below is both of them.

What the page owns in exchange is the media. Capturing the microphone, resampling it to the session's input format and playing back the response.output_audio.delta chunks are its own work, because a WebSocket carries the audio bytes handed to it and nothing else: no jitter buffer, no packet-loss concealment, no echo cancellation. On a good network that is unremarkable; on a lossy one it is audible, and it is the reason to reach for a media stack rather than write one.

WebRTC calls terminated by the gateway¶

With REALTIME_WEBRTC_ENABLED set — and the webrtc optional dependencies installed, which the container images ship — the gateway terminates the whole WebRTC media path itself: ICE, DTLS-SRTP and Opus. A browser (or any WebRTC client) posts its SDP offer and connects directly, with nothing in between:

• POST /v1/realtime/calls accepts the offer as a raw application/sdp body — with ?model=<id> on the query string, exactly as upstream's browser flow — or as multipart/form-data with an sdp field and an optional session JSON field. Either encoding authenticates with an ephemeral client secret or the deployment's credentials; a secret whose session is locked refuses a session field. A JSON body is refused with unsupported_content_type, which is what the upstream endpoint answers too. The response is 201 with the SDP answer as its body and the call's identifier in the Location header (/v1/realtime/calls/rtc_...).
• Audio rides the media tracks as Opus, both directions — no base64, no response.output_audio.delta events. Events ride a data channel the client opens under the label oai-events, in exactly the WebSocket vocabulary: session.created arrives on it once the channel opens, session.update, response.create and the rest work unchanged. The session's audio formats are fixed by the media negotiation, so a session.update changing them is refused with an error.
• Playback is the gateway's, so barge-in is too. The model generates speech faster than it is played, and on a call the unplayed tail sits in the gateway rather than in your player. A caller who starts speaking over it — input_audio_buffer.speech_started — stops hearing it immediately, and the drop is reported as output_audio_buffer.cleared, as it is upstream. An answer ended by response.cancel or interrupted mid-generation stops the same way.
• POST /v1/realtime/calls/{call_id}/hangup ends the call; the SDP answer's Location header is where the identifier comes from.
• A sideband WebSocket — WS /v1/realtime?call_id=<id>, API credentials only, never an ephemeral secret — observes the call's server events and may send client events into its session, mirroring upstream's monitoring channel.
• Call control follows the credential that opened the call. A call opened under a tenant API key can be ended or observed only with that tenant's key or the deployment's own credentials; any other caller is answered the same 404 as an unknown call.

Put WebRTC or a phone line in front of the gateway¶

The route to a browser peer connection or a phone call does not have to run through POST /v1/realtime/calls, and for SIP it never does. The two frameworks most voice agents are already built on — LiveKit Agents and Pipecat — terminate WebRTC themselves (and SIP, through their telephony transports), and reach the model over exactly the WebSocket this API serves. The caller speaks WebRTC or SIP to the framework; the framework speaks this API. Pointing one at this deployment asks no more of it than the rest of this gateway does: the base URL, the deployment's API key, and the model name only where the name differs.

LiveKit Agents takes an HTTP base URL and derives the WebSocket from it, exactly as the official SDK does:

from livekit.agents import AgentSession
from livekit.plugins import openai

session = AgentSession(
    llm=openai.realtime.RealtimeModel(
        model="amazon.nova-2-sonic-v1:0",
        base_url="https://your-deployment.example.com/v1",
        api_key="YOUR_API_KEY",
    )
)

base_url also reads from the OPENAI_BASE_URL environment variable, and api_key from OPENAI_API_KEY. A base URL ending in /v1 has /realtime appended for you; a deployment served under a non-default OPENAI_ROUTES_PREFIX has to name the full path itself.

Pipecat takes the WebSocket URL whole, /v1/realtime included:

import os

from pipecat.services.openai.realtime.llm import OpenAIRealtimeLLMService

llm = OpenAIRealtimeLLMService(
    base_url="wss://your-deployment.example.com/v1/realtime",
    api_key=os.environ["OPENAI_API_KEY"],
    settings=OpenAIRealtimeLLMService.Settings(model="amazon.nova-2-sonic-v1:0"),
)

On a telephony leg, set the session's audio formats to audio/pcmu or audio/pcma — G.711 at 8 kHz is what a phone call already carries, so nothing resamples it twice.

Why WebRTC is a different kind of endpoint¶

An HTTP request that returns an SDP answer is the small, visible part of WebRTC. The rest is a media transport in its own right: a UDP path negotiated separately from the HTTPS connection, carrying encrypted RTP, with its own address discovery, NAT traversal and congestion control. Serving it means terminating that media path, not routing one more request — and the two are not the same thing to build, nor the same thing to put an ingress in front of. SIP has the same shape under different names: signalling on one connection, audio on another.

That is also why a framework remains the shorter path for anything the gateway-terminated transport does not cover — telephony, more than one instance, calls past 8 minutes, hostile networks without your own TURN. LiveKit and Pipecat already own a media stack, already run wherever your users are, and already speak this API on the other side. WebRTC and SIP need their own ingress covers what terminating media costs a deployment, whichever process does it.

Billing¶

A realtime session bills audio and text tokens continuously, in both directions, for as long as the connection is open — not per request. Usage is reported per answer, in each response.done event, and recorded in the gateway's usage log the same way, so a session that drops mid-conversation still accounts for everything spoken before it. Speech tokens are priced well above text tokens by AWS, and the two are recorded and priced separately here. See Cost Management for how usage becomes cost.

Limits and behaviour to know¶

What a session refuses and how it ends. The event-level gaps are in the feature compatibility table, and what a WebRTC call additionally demands is under WebRTC calls.

Session lifecycle and limits¶

• Duration cap — a session lasts at most 8 minutes. When it is reached, the server closes the connection with WebSocket close code 1000 and reason session_expired; reconnect to continue the conversation. A WebRTC call is under the same cap: at 8 minutes its session ends and the peer connection is torn down.
• Conversation ended by the model side — when the model ends the conversation itself, the connection closes with close code 1000 and reason session_ended. Normal, and reconnecting starts a new session.
• Server shutdown — a session still open when the deployment shuts down is closed with close code 1001 and reason server_shutdown.
• Fatal errors — a fatal error sends a terminal error event, then closes the connection with close code 3000, whose reason is <error type>.<error code> (e.g. invalid_request_error.model_not_found).
• Event size — a single client event may carry at most 4 MiB, base64 included; a larger one answers an error and is dropped. Append audio in the small chunks it is captured in rather than whole files.
• Uncommitted audio — under manual turns (turn_detection: null), at most 5.7 MB of decoded audio may be buffered before an input_audio_buffer.commit (about 2 minutes of 24 kHz PCM, longer for G.711); past that the append answers an error. Commit each turn, or clear the buffer with input_audio_buffer.clear.
• Addressable items — the session keeps its 200 most recent conversation items addressable, dropping the oldest past that. conversation.item.truncate, .retrieve and .delete answer an error for an item that has fallen out; the model's own memory of the conversation is unaffected.

Request headers¶

The Amazon Bedrock guardrail headers apply to a realtime session. Send them on the WebSocket handshake, or on POST /v1/realtime/calls for a WebRTC call. All headers are optional.

Content Safety (Guardrails)¶

The guardrail selected here is the one applied per turn, as described under Guardrail coverage. Both headers are honoured only when AWS_BEDROCK_ALLOW_GUARDRAIL_OVERRIDE is enabled — otherwise the deployment's configured guardrail applies. X-Amzn-Bedrock-Trace is accepted but has no effect on this route — no guardrail trace is returned.

Try it¶

Python (server-side, with the official SDK)¶

Requires the openai package with realtime support (pip install "openai[realtime]"). Point the client's base_url at this deployment; client.realtime.connect(...) derives the WebSocket URL from it automatically.

import base64

from openai import AsyncOpenAI

client = AsyncOpenAI(
    api_key="YOUR_API_KEY", base_url="https://your-deployment.example.com/v1"
)


async def main() -> None:
    async with client.realtime.connect(model="amazon.nova-2-sonic-v1:0") as connection:
        await connection.session.update(
            session={
                "type": "realtime",
                "instructions": "You are a concise, friendly voice assistant.",
            }
        )

        # Stream 24 kHz, 16-bit, mono, little-endian PCM in the small chunks it
        # is captured in -- ~100 ms each here, well under the per-event limit.
        with open("question.pcm", "rb") as audio_file:
            while chunk := audio_file.read(4800):
                await connection.input_audio_buffer.append(
                    audio=base64.b64encode(chunk).decode()
                )
        await connection.input_audio_buffer.commit()
        await connection.response.create()

        async for event in connection:
            if event.type == "response.output_audio.delta":
                # event.delta is base64-encoded audio in the session's output format.
                ...
            elif event.type == "response.output_audio_transcript.delta":
                print(event.delta, end="", flush=True)
            elif event.type == "response.done":
                break

Browser (ephemeral client secret)¶

Mint the secret from your backend — never expose the deployment's own API key to the browser — then connect directly from client-side JavaScript using the openai-insecure-api-key.<secret> subprotocol, since a browser cannot set custom WebSocket headers:

// Fetched from your own backend, which called POST /v1/realtime/client_secrets
const { value: ephemeralSecret } = await fetch("/api/realtime-secret").then((r) => r.json());

const ws = new WebSocket(
  "wss://your-deployment.example.com/v1/realtime?model=amazon.nova-2-sonic-v1:0",
  ["realtime", `openai-insecure-api-key.${ephemeralSecret}`],
);

ws.addEventListener("open", async () => {
  const stream = await navigator.mediaDevices.getUserMedia({ audio: true });

  // Your own capture code: resample the microphone to the session's input
  // format (24 kHz, 16-bit, mono PCM by default) and hand over small chunks.
  captureAudioChunks(stream, (pcmChunk) => {
    ws.send(
      JSON.stringify({
        type: "input_audio_buffer.append",
        audio: btoa(String.fromCharCode(...new Uint8Array(pcmChunk))),
      }),
    );
  });
  // Nothing else is needed: server voice activity detection ends each turn and
  // starts the answer. Under `turn_detection: null`, send
  // `input_audio_buffer.commit` yourself instead.
});

ws.addEventListener("message", (event) => {
  const serverEvent = JSON.parse(event.data);
  if (serverEvent.type === "response.output_audio.delta") {
    // serverEvent.delta is base64-encoded audio in the session's output format.
  }
});

Next steps¶

Next: Search Models API · Text to Speech API · Transcriptions API · WebRTC and SIP ingress