RAG Pipelines Integration¶

Build retrieval-augmented generation and semantic search pipelines on Amazon Bedrock through stdapi.ai's OpenAI-compatible embeddings and Cohere-compatible reranking—one deployment serving every stage of the pipeline.

There are two ways to do it, and they share the same deployment:

• Managed — attach your files to a vector store and search it, or let the model search it for itself. Chunking, embedding, indexing and retrieval are the gateway's job; you write no pipeline.
• Assembled — keep your own framework and vector database, and use stdapi.ai for the embedding, reranking and generation calls.

Start managed, and move to the assembled pipeline when you need a retrieval strategy of your own.

At a glance¶

%%{init: {'flowchart': {'htmlLabels': true}} }%%
flowchart LR
  rag["Your RAG Framework<br/>(Haystack, LlamaIndex, ...)"] --> stdapi["<img src='../styles/logo.svg' style='height:64px;width:auto;vertical-align:middle;' /> stdapi.ai"]
  stdapi --> bedrock["<img src='../styles/logo_amazon_bedrock.svg' style='height:64px;width:auto;vertical-align:middle;' /> Amazon Bedrock"]

About Retrieval-Augmented Generation¶

A RAG pipeline grounds a model's answer in your own documents instead of its training data: a retriever finds candidate passages by vector similarity, an optional reranker reorders them by relevance to the actual question, and a chat model answers from the reordered context.

What a RAG pipeline needs from its backend:

• Embeddings - Vectorize documents and queries into the same space
• Reranking - Reorder retrieved candidates by relevance before they reach the model
• Generation - Answer from the retrieved context with a chat-capable model

stdapi.ai serves all three from Amazon Bedrock through standard, unmodified client libraries.

Connect Your Own Instance¶

Both retrieval modes work against any stdapi.ai deployment, wherever it runs. Nothing below requires the AWS sample in Part 2.

Managed Retrieval with Vector Stores¶

Upload your files, attach them to a vector store, and search it. Nothing else runs.

import time

from openai import OpenAI

client = OpenAI(base_url="https://YOUR_STDAPI_URL/v1", api_key="YOUR_API_KEY")

uploaded = client.files.create(file=open("handbook.txt", "rb"), purpose="assistants")
store = client.vector_stores.create(name="handbook", file_ids=[uploaded.id])

# Indexing is asynchronous: wait until the store reports it finished.
while client.vector_stores.retrieve(store.id).status == "in_progress":
    time.sleep(2)

for result in client.vector_stores.search(
    store.id, query="How much parental leave do I get?"
):
    print(result.score, result.filename, result.content[0].text)

Only text files can be indexed — convert PDFs and office documents first, with the document parsing stage below. See the Vector Stores API for chunking, filters, expiration and the storage it needs in your account.

Let the Model Do the Retrieving¶

Naming a store as a file_search tool on the Responses API moves the whole loop into a single request: the model decides when to search and with which query, answers from the passages it gets back, and annotates the answer with a citation per file it drew on.

response = client.responses.create(
    model="amazon.nova-2-lite-v1:0",
    input="How much parental leave do I get?",
    tools=[{"type": "file_search", "vector_store_ids": [store.id]}],
)
print(response.output_text)

Searching a Knowledge Base You Already Run¶

If your documents are already in an Amazon Bedrock knowledge base, keep it where it is. Allowlist it in AWS_BEDROCK_KNOWLEDGE_BASE_IDS and it is addressed as the vector store vs_kb_<knowledgeBaseId> — searched, listed, read and (on a custom data source) extended with new documents, through the same client code and the same file_search tool as above.

The knowledge base stays yours: it is never created and never deleted here, and a request that would reshape it — renaming, an expiry, a chunking strategy — is refused with the reason rather than half-applied. Its retrieval scores are reported as the backend states them, so a score_threshold against such a store is refused instead of meaning something else. See Knowledge Base Stores.

Assembling Your Own Pipeline¶

The rest of this part wires stdapi.ai into a pipeline you build yourself, with your own vector database and retrieval strategy.

Prerequisites¶

Configuration¶

Every RAG framework that speaks the OpenAI and Cohere SDKs follows the same pattern: the embedder and the generator take the OpenAI-compatible /v1 base URL, and the reranker takes the Cohere-compatible /cohere base URL.

Document Parsing¶

Before embedding, a RAG pipeline needs plain text out of PDFs and office documents. Docling Serve converts them to Markdown/JSON over an HTTP API — deploy it as the ingestion stage in front of the embedder below, not as a standalone gateway showcase.

Docling's default pipeline is classical layout/OCR/table-structure extraction and never calls an LLM. Its optional VLM pipeline additionally routes page images through stdapi.ai to a vision-capable Bedrock model, for documents that benefit from model-assisted layout understanding.

curl -s -X POST "$DOCLING_URL/v1/convert/source" \
  -H 'Content-Type: application/json' \
  -d '{
    "options": {"to_formats": ["md"]},
    "sources": [{"kind": "http", "url": "https://example.com/document.pdf"}]
  }' | jq -r '.document.md_content'

See Part 2 for a Terraform-deployed Docling Serve on AWS.

Embeddings¶

Point your framework's OpenAI-compatible embedder at /v1 with an embeddings-capable model.

from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
from haystack.utils import Secret

document_embedder = OpenAIDocumentEmbedder(
    api_key=Secret.from_env_var("STDAPI_API_KEY"),
    model="amazon.titan-embed-text-v2:0",
    api_base_url="https://YOUR_STDAPI_URL/v1",
)
text_embedder = OpenAITextEmbedder(
    api_key=Secret.from_env_var("STDAPI_API_KEY"),
    model="amazon.titan-embed-text-v2:0",
    api_base_url="https://YOUR_STDAPI_URL/v1",
)

Reranking¶

Point your framework's Cohere-compatible reranker at /cohere, not the full rerank path—the Cohere client appends the operation itself.

from haystack_integrations.components.rankers.cohere import CohereRanker
from haystack.utils import Secret

ranker = CohereRanker(
    api_key=Secret.from_env_var("STDAPI_API_KEY"),
    model="cohere.rerank-v3-5:0",
    api_base_url="https://YOUR_STDAPI_URL/cohere",
    top_k=3,
)

Generation¶

Point your framework's OpenAI-compatible chat generator at /v1, answering from the reranked context.

from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.utils import Secret

generator = OpenAIChatGenerator(
    api_key=Secret.from_env_var("STDAPI_API_KEY"),
    model="anthropic.claude-haiku-4-5-20251001-v1:0",
    api_base_url="https://YOUR_STDAPI_URL/v1",
)

Other Frameworks¶

The same two-route pattern—OpenAI-compatible /v1 for embedding and generation, Cohere-compatible /cohere for reranking—applies to any framework built on those SDKs, including LlamaIndex, RAGFlow, and LightRAG. Set the base URL and API key on the framework's OpenAI and Cohere client configuration; the vector store itself (pgvector, Qdrant, or an in-memory store) is unaffected and stores whatever the embedder returns.

Frameworks that drive the Responses API can take the managed half instead of a store of their own: given a file and a file_search tool, Agno uploads it, creates the vector store, waits for the indexing and names the store on the turn — all against the base URL it already chats with.

Deploy the Full Stack on AWS¶

The architecture below is one worked example of a credible AWS deployment for this pipeline, not the only one that works: the gateway is a normal HTTPS service, and the document-parsing and vector-store pieces it depicts can run wherever you already operate workloads.

Architecture¶

The two modes differ in who runs the pipeline and where its state lives. Managed retrieval puts chunking, embedding, indexing and search inside the gateway, backed by a vector bucket and a durable indexing queue in your account; the assembled pipeline keeps your framework in charge of chunking, indexing and search, and calls stdapi.ai only for the embedding, reranking and generation steps in between.

%%{init: {'flowchart': {'htmlLabels': true, 'nodeSpacing': 20, 'rankSpacing': 40, 'subGraphTitleMargin': {'top': 8, 'bottom': 10}}} }%%
flowchart TB
  app["Your application or RAG framework<br/>(Haystack, LlamaIndex, ...)"]
  ownvdb["Your vector database<br/>(pgvector, Qdrant, ...)<br/>assembled pipeline only"]

  subgraph public["Your VPC · public subnets"]
    alb["<img src='../styles/logo_amazon_load_balancing.svg' style='height:40px;width:auto;vertical-align:middle;' /> Application Load Balancer<br/>document parsing only"]
  end

  subgraph private["Your VPC · private app subnets — no inbound route from the internet"]
    docling["<img src='../styles/logo_docling.svg' style='height:40px;width:auto;vertical-align:middle;' /> Docling Serve<br/>ECS Fargate · document parsing"]
    stdapi["<img src='../styles/logo.svg' style='height:40px;width:auto;vertical-align:middle;' /> stdapi.ai<br/>ECS Fargate"]
    egress["<img src='../styles/logo_amazon_vpc.svg' style='height:40px;width:auto;vertical-align:middle;' /> NAT gateways<br/>one per Availability Zone"]
  end

  subgraph endpoints["AWS service endpoints · your account, the regions you configure"]
    bedrock["<img src='../styles/logo_amazon_bedrock.svg' style='height:40px;width:auto;vertical-align:middle;' /> Amazon Bedrock"]
    s3["<img src='../styles/logo_amazon_s3.svg' style='height:40px;width:auto;vertical-align:middle;' /> Amazon S3<br/>files and vector store records"]
    s3vectors["<img src='../styles/logo_amazon_s3.svg' style='height:40px;width:auto;vertical-align:middle;' /> Amazon S3 Vectors<br/>vector indexes<br/>managed retrieval only"]
    sqs["<img src='../styles/logo_amazon_sqs.svg' style='height:40px;width:auto;vertical-align:middle;' /> Amazon SQS<br/>durable indexing queue + DLQ<br/>managed retrieval only"]
    bedrock ~~~ s3vectors
    s3 ~~~ sqs
  end

  app -->|"HTTPS · API key<br/>parse PDFs/office documents"| alb
  alb -->|"HTTP · private subnet"| docling
  docling -->|"OpenAI API · API key<br/>optional VLM pipeline"| stdapi

  app -->|"managed: upload, attach,<br/>search a store, then<br/>generate the answer"| stdapi
  app -.->|"assembled: embed passages,<br/>embed the query, rerank,<br/>then generate the answer"| stdapi
  app -.->|"assembled: index and<br/>search vectors yourself"| ownvdb

  stdapi --> egress
  egress -->|"HTTPS · SigV4<br/>embed · rerank · generate"| bedrock
  egress -->|"HTTPS · SigV4<br/>vector store records"| s3
  egress -->|"HTTPS · SigV4<br/>index and search vectors"| s3vectors
  egress -.->|"SendMessage / ReceiveMessage<br/>any task may resume a job"| sqs

Two things are worth reading off the picture. In managed retrieval, your documents and their vectors come to rest inside your account — the general-purpose bucket holds the vector store's records, the S3 Vectors bucket holds the indexed embeddings, and the SQS queue makes an indexing job outlive the task that accepted it: this is the one deployment shape on this page where the gateway is not stateless. In the assembled pipeline the gateway stays exactly as stateless as everywhere else on this site — it holds no vectors at all, which land wherever your own framework's vector database puts them, never in this picture.

What Each AWS Service Does Here¶

Security Measures in This Flow¶

• Authentication — every call your application or framework makes to the gateway carries a stdapi.ai API key; Docling's optional VLM pipeline authenticates the same way when it calls the gateway for vision inference.
• Encryption in transit — HTTPS from wherever your application or framework runs to the gateway, and HTTPS with SigV4 from the gateway to Amazon Bedrock, Amazon S3, Amazon S3 Vectors and Amazon SQS.
• Encryption at rest — SSE-KMS on the S3 bucket holding vector store records, and on the S3 Vectors bucket and the SQS queue, each behind its own key: aws_s3_vectors_kms_key_arn and aws_sqs_vector_store_queue_kms_key_arn when you bring your own bucket or queue, a dedicated key created for you otherwise.
• Least privilege — the gateway's task role is granted the Vector Stores, durable indexing and knowledge base actions only when the corresponding feature is configured, scoped to the ARNs of the bucket, the queue and the knowledge bases you name.
• Content policy — a Bedrock guardrail applies to embeddings and reranking through the ApplyGuardrail API rather than Bedrock's native integration, checking each text input before it is embedded or reranked, and its consumed units are reported — unlike the native integration chat routes use.
• Data handling — uploaded files and their S3 storage stay in your account, and an Amazon Bedrock knowledge base you already operate is addressed as a vector store — searched and extended, never recreated.

What's Included¶

📦 stdapi-ai/samples/getting_started_docling is one worked example of the document-parsing stage on AWS — Docling Serve on ECS Fargate, CPU-only, with the VLM pipeline pre-wired to a Bedrock vision model through stdapi.ai. See the sample's README for deployment steps.

Want the platform instead of the pipeline? This page covers wiring stdapi.ai into a pipeline you assemble yourself. If you would rather run a finished product—a web UI, document parsing, knowledge bases, and grounded chat, with no pipeline code to write—see the RAGFlow Integration guide, which deploys RAGFlow with all three stages already bound to Amazon Bedrock.

What It Costs to Run¶

Read a model's price before you send anything to it with GET /model_pricing. Setting COST_TRACKING=true additionally puts a per-request cost on each usage entry — estimated from published AWS prices, not read back from your invoice — though the vector bucket's own storage and request charges never appear in it; read those from AWS Cost Explorer instead.

What to Watch¶

The gateway logs one request event per call and, for managed indexing, a separate background event named vector_store_indexing correlated to it by id — group on that field to see how long a file's indexing actually took after the attach call returned. The dead-letter queue behind AWS_SQS_VECTOR_STORE_QUEUE_URL is the signal that a document failed to index: a file still failing after its queue's redrive policy exhausts its deliveries lands there instead of being silently dropped, so a non-empty dead-letter queue means a document needs attention. Turning on CLOUDWATCH_METRICS republishes the same billed quantities as EMF metrics in the stdapi namespace, dimensioned by Model.

fields @timestamp, event, execution_time_ms, id
| filter type = "background" and event = "vector_store_indexing"
| sort @timestamp desc
| limit 100

Watch the dead-letter queue depth alongside this query — a job that never appears here again after being sent once, with the primary queue's ApproximateNumberOfMessagesVisible back at zero, finished; one that keeps reappearing is heading for the dead-letter queue instead.

Next Steps¶