Realtime API¶

Hold a live, bidirectional speech-to-speech conversation over a single WebSocket, through the OpenAI Realtime API shape. Audio flows in both directions on the same connection: send the caller's speech as it is captured, and receive the model's spoken answer as it is generated — no request/response round trip per turn.

Why Choose the Realtime API?¶

Available Endpoints¶

Feature Compatibility¶

Voice, instructions and audio formats are fixed once the conversation opens¶

The model's voice, its system instructions, and both audio formats are set when the conversation with the model opens, and cannot change for the rest of that session. The conversation opens on the first thing sent into it — the first input_audio_buffer.append under the default server voice activity detection, or the first input_audio_buffer.commit, response.create or conversation.item.create under manual turns — which is well before the model has answered anything.

Send session.update with these settings before sending anything else, or open a new session to change them. Afterwards, a session.update touching only other fields (turn_detection, max_output_tokens, transcription settings, and so on) is still accepted; one that would change voice, instructions or an audio format is refused with an error event.

Answering a written turn¶

conversation.item.create carrying an input_text part adds the text to the conversation without starting an answer. Follow it with response.create, and the model answers it exactly as it answers a spoken one — the same response.output_audio.delta chunks and the same transcript — so a written nudge into a voice session ("the caller has been on hold", "wrap up now") needs no second channel.

What a response object reports¶

response.created and response.done carry the same response object, and every field the upstream API sends is present on both — a voice framework validates each frame against its own models, and a missing field is the event never arriving rather than a cosmetic difference.

Truncating an answer the caller spoke over¶

The model generates speech faster than it is played, so a caller who interrupts has heard less of the answer than was sent. Send conversation.item.truncate with the item's id, content_index: 0 and the audio_end_ms your player actually reached; the session cuts its record of that item to what was heard and answers conversation.item.truncated.

• The item's transcript is removed whole, not trimmed: nothing aligns a transcript to a position in the audio, and leaving text the caller never heard in the record is the failure this event exists to prevent.
• audio_end_ms past the end of the item's audio, an item that is not an assistant message, and an item this session never sent are each refused with an error.
• What the model itself remembers of the answer is the model's own; truncation aligns the record this session reports through conversation.item.retrieved.

Guardrail coverage¶

When the deployment configures an Amazon Bedrock guardrail, a realtime session is checked per turn: what the caller said (as the model transcribes it) as INPUT, and each completed answer as OUTPUT. A blocked turn ends the session with a terminal error event and close code 3000.

Unlike a request/response route, the check cannot come before the content reaches the client: the model's speech is streamed while it is being generated and its transcript is only complete once the answer is over, so a blocked answer may already have been partly heard when the session ends. Written items sent with conversation.item.create are checked as INPUT before they reach the model, as on every other route.

Model Support¶

Every deployment's catalog differs, and a model's own name is never guaranteed stable across accounts. Find which models serve this route:

curl "$BASE/search_models?route=openai_realtime" \
  -H "Authorization: Bearer $OPENAI_API_KEY"

Pass the returned model ID as model on the WebSocket URL, or in the session.model field of an ephemeral secret's configuration. See the Search Models API for the full filter syntax.

Authentication¶

Open the WebSocket with one of three credentials, carried in whichever channel the client can use:

Either the deployment's own API key or an ephemeral client secret (ek_...) works as the credential. The model query parameter (/v1/realtime?model=<model id>) selects the model serving the session; it may be omitted when the credential is an ephemeral secret whose session configuration already names one. See Authentication & Security for how the API key itself is configured.

Ephemeral Client Secrets¶

POST /v1/realtime/client_secrets mints a short-lived credential — a value starting with ek_ — that carries a session configuration. Hand it to a browser or mobile client so it can open a session directly, without ever holding the deployment's own API key.

curl -X POST "$BASE/v1/realtime/client_secrets" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "expires_after": {"anchor": "created_at", "seconds": 60},
    "session": {
      "type": "realtime",
      "model": "amazon.nova-2-sonic-v1:0",
      "instructions": "You are a helpful support agent."
    }
  }'

{
  "value": "ek_...",
  "expires_at": 1731000060,
  "session": {
    "type": "realtime",
    "model": "amazon.nova-2-sonic-v1:0",
    "instructions": "You are a helpful support agent.",
    "audio": { "...": "..." }
  }
}

• expires_after.seconds accepts 10 to 7,200 seconds, defaulting to 600 (10 minutes) when omitted. This bounds how long the secret can be used to open a session — a session already opened with it keeps running for its own session limit.
• session accepts the same configuration a client would otherwise send in a session.update event; it is applied to every session opened with the secret. It also accepts {"type": "transcription"}, which opens sessions that only transcribe the caller and never answer out loud — the only way to ask for one, since the socket itself takes no session type.
• By default the carried configuration is a default, not a constraint: the client may name another model on the ?model= query string and change the configuration with its own session.update, as it can upstream. Set REALTIME_ALLOW_SESSION_OVERRIDE=false to make the model, the instructions and max_output_tokens the secret was minted with final — a mismatching ?model= is then refused at connect, and a session.update changing one of them answers an error.

Transports¶

Upstream offers a realtime session over three transports — WebSocket, WebRTC and SIP. This API serves the WebSocket, and only the WebSocket. POST /v1/realtime/calls, the endpoint upstream uses to trade an SDP offer for a WebRTC peer connection or to accept an inbound SIP call, has no route here and answers 404; it is planned for a later release.

A browser connects to that same WebSocket¶

There is no separate browser transport to be missing. The page opens wss://<host>/v1/realtime?model=<id> itself, carrying an ephemeral client secret in the way Authentication describes, so the deployment's own API key never leaves your backend. It is two steps — mint the secret server-side, connect with it client-side — and the browser example below is both of them.

What the page owns in exchange is the media. Capturing the microphone, resampling it to the session's input format and playing back the response.output_audio.delta chunks are its own work, because a WebSocket carries the audio bytes handed to it and nothing else: no jitter buffer, no packet-loss concealment, no echo cancellation. On a good network that is unremarkable; on a lossy one it is audible, and it is the reason to reach for a media stack rather than write one.

Put WebRTC or a phone line in front of the gateway¶

The route to a browser peer connection or a phone call does not run through POST /v1/realtime/calls. The two frameworks most voice agents are already built on — LiveKit Agents and Pipecat — terminate WebRTC themselves (and SIP, through their telephony transports), and reach the model over exactly the WebSocket this API serves. The caller speaks WebRTC or SIP to the framework; the framework speaks this API. Pointing one at this deployment asks no more of it than the rest of this gateway does: the base URL, the deployment's API key, and the model name only where the name differs.

LiveKit Agents takes an HTTP base URL and derives the WebSocket from it, exactly as the official SDK does:

from livekit.agents import AgentSession
from livekit.plugins import openai

session = AgentSession(
    llm=openai.realtime.RealtimeModel(
        model="amazon.nova-2-sonic-v1:0",
        base_url="https://your-deployment.example.com/v1",
        api_key="YOUR_API_KEY",
    )
)

base_url also reads from the OPENAI_BASE_URL environment variable, and api_key from OPENAI_API_KEY. A base URL ending in /v1 has /realtime appended for you; a deployment served under a non-default OPENAI_ROUTES_PREFIX has to name the full path itself.

Pipecat takes the WebSocket URL whole, /v1/realtime included:

import os

from pipecat.services.openai.realtime.llm import OpenAIRealtimeLLMService

llm = OpenAIRealtimeLLMService(
    base_url="wss://your-deployment.example.com/v1/realtime",
    api_key=os.environ["OPENAI_API_KEY"],
    settings=OpenAIRealtimeLLMService.Settings(model="amazon.nova-2-sonic-v1:0"),
)

On a telephony leg, set the session's audio formats to audio/pcmu or audio/pcma — G.711 at 8 kHz is what a phone call already carries, so nothing resamples it twice.

Why WebRTC is a different kind of endpoint¶

An HTTP request that returns an SDP answer is the small, visible part of WebRTC. The rest is a media transport in its own right: a UDP path negotiated separately from the HTTPS connection, carrying encrypted RTP, with its own address discovery, NAT traversal and congestion control. Serving it means terminating that media path, not routing one more request — and the two are not the same thing to build, nor the same thing to put an ingress in front of. SIP has the same shape under different names: signalling on one connection, audio on another.

That is also why a framework is the shorter path rather than a stopgap. LiveKit and Pipecat already own a media stack, already run wherever your users are, and already speak this API on the other side. If you do intend to run a media terminator of your own, WebRTC and SIP need their own ingress covers what that costs a deployment.

Session Lifecycle and Limits¶

• Duration cap — a session lasts at most 8 minutes. When it is reached, the server closes the connection with WebSocket close code 1000 and reason session_expired; reconnect to continue the conversation.
• Conversation ended by the model side — when the model ends the conversation itself, the connection closes with close code 1000 and reason session_ended. Normal, and reconnecting starts a new session.
• Server shutdown — a session still open when the deployment shuts down is closed with close code 1001 and reason server_shutdown.
• Fatal errors — a fatal error sends a terminal error event, then closes the connection with close code 3000, whose reason is <error type>.<error code> (e.g. invalid_request_error.model_not_found).
• Event size — a single client event may carry at most 4 MiB, base64 included; a larger one answers an error and is dropped. Append audio in the small chunks it is captured in rather than whole files.
• Uncommitted audio — under manual turns (turn_detection: null), at most 5.7 MB of decoded audio may be buffered before an input_audio_buffer.commit (about 2 minutes of 24 kHz PCM, longer for G.711); past that the append answers an error. Commit each turn, or clear the buffer with input_audio_buffer.clear.
• Addressable items — the session keeps its 200 most recent conversation items addressable, dropping the oldest past that. conversation.item.truncate, .retrieve and .delete answer an error for an item that has fallen out; the model's own memory of the conversation is unaffected.

Billing¶

A realtime session bills audio and text tokens continuously, in both directions, for as long as the connection is open — not per request. Usage is reported per answer, in each response.done event, and recorded in the gateway's usage log the same way, so a session that drops mid-conversation still accounts for everything spoken before it. Speech tokens are priced well above text tokens by AWS, and the two are recorded and priced separately here. See Cost Management for how usage becomes cost.

Try It Now¶

Python (server-side, with the official SDK)¶

Requires the openai package with realtime support (pip install "openai[realtime]"). Point the client's base_url at this deployment; client.realtime.connect(...) derives the WebSocket URL from it automatically.

import base64

from openai import AsyncOpenAI

client = AsyncOpenAI(
    api_key="YOUR_API_KEY", base_url="https://your-deployment.example.com/v1"
)


async def main() -> None:
    async with client.realtime.connect(model="amazon.nova-2-sonic-v1:0") as connection:
        await connection.session.update(
            session={
                "type": "realtime",
                "instructions": "You are a concise, friendly voice assistant.",
            }
        )

        # Stream 24 kHz, 16-bit, mono, little-endian PCM in the small chunks it
        # is captured in -- ~100 ms each here, well under the per-event limit.
        with open("question.pcm", "rb") as audio_file:
            while chunk := audio_file.read(4800):
                await connection.input_audio_buffer.append(
                    audio=base64.b64encode(chunk).decode()
                )
        await connection.input_audio_buffer.commit()
        await connection.response.create()

        async for event in connection:
            if event.type == "response.output_audio.delta":
                # event.delta is base64-encoded audio in the session's output format.
                ...
            elif event.type == "response.output_audio_transcript.delta":
                print(event.delta, end="", flush=True)
            elif event.type == "response.done":
                break

Browser (ephemeral client secret)¶

Mint the secret from your backend — never expose the deployment's own API key to the browser — then connect directly from client-side JavaScript using the openai-insecure-api-key.<secret> subprotocol, since a browser cannot set custom WebSocket headers:

// Fetched from your own backend, which called POST /v1/realtime/client_secrets
const { value: ephemeralSecret } = await fetch("/api/realtime-secret").then((r) => r.json());

const ws = new WebSocket(
  "wss://your-deployment.example.com/v1/realtime?model=amazon.nova-2-sonic-v1:0",
  ["realtime", `openai-insecure-api-key.${ephemeralSecret}`],
);

ws.addEventListener("open", async () => {
  const stream = await navigator.mediaDevices.getUserMedia({ audio: true });

  // Your own capture code: resample the microphone to the session's input
  // format (24 kHz, 16-bit, mono PCM by default) and hand over small chunks.
  captureAudioChunks(stream, (pcmChunk) => {
    ws.send(
      JSON.stringify({
        type: "input_audio_buffer.append",
        audio: btoa(String.fromCharCode(...new Uint8Array(pcmChunk))),
      }),
    );
  });
  // Nothing else is needed: server voice activity detection ends each turn and
  // starts the answer. Under `turn_detection: null`, send
  // `input_audio_buffer.commit` yourself instead.
});

ws.addEventListener("message", (event) => {
  const serverEvent = JSON.parse(event.data);
  if (serverEvent.type === "response.output_audio.delta") {
    // serverEvent.delta is base64-encoded audio in the session's output format.
  }
});

Ready to add voice to your application? Find compatible models with the Search Models API, or explore the full audio suite for turn-based speech and transcription.