Retrieval-Augmented Generation on Azure: An Architecture That Survives Production

Every RAG demo works. Almost no RAG demo survives contact with real users, real documents, and a real budget. The gap is architectural, and this article maps it.

The reference architecture

User → App Service (API) → Azure AI Search (hybrid: vector + BM25 + semantic ranker)
                          ↘ Azure OpenAI (GPT-4o) ← retrieved chunks
Ingestion: Blob Storage → Functions (chunk + embed) → AI Search index
Telemetry: App Insights ← every stage, with correlation IDs

Three decisions matter more than everything else combined: how you chunk, how you retrieve, and how you evaluate.

Chunking is a product decision, not a preprocessing step

Fixed 512-token chunks are why your bot answers questions about section 3.2 with the header of section 3.1. Chunk along document structure — headings, tables, list boundaries — and carry metadata:

chunk = {
    "content": section_text,
    "title": doc_title,
    "heading_path": "Networking > Private Endpoints > DNS",
    "source_url": blob_url,
    "last_modified": modified_utc,
}

The heading_path field alone typically improves answer groundedness more than swapping embedding models, because the LLM sees where a chunk lives in the document's argument.

Hybrid retrieval or nothing

Pure vector search fails on exact identifiers — SKU names, error codes, cmdlet names — precisely the queries technical users ask. Azure AI Search's hybrid mode runs BM25 and vector search together and fuses results with Reciprocal Rank Fusion; the semantic ranker then reorders the top 50.

If you don't measure it, it doesn't work

Build the evaluation harness before tuning anything. A minimal loop:

1. Collect 100–200 real user questions (not questions you invented).
2. Label the documents that should ground each answer.
3. Score retrieval (recall@k) and generation (groundedness, relevance) on every change.

Azure AI Foundry's evaluators cover groundedness and relevance out of the box; wire them into CI so a prompt "improvement" that tanks groundedness fails the build like any other regression.

Cost control that actually works

• Cache embeddings by content hash — re-ingestion of unchanged documents should cost nothing.
• Route easy queries to a small model; reserve GPT-4o for questions the router scores as hard.
• Cap context: past ~8 chunks, answer quality plateaus while cost climbs linearly.

RAG in production is an information-retrieval system with an LLM attached — not the other way around. Teams that staff and measure it that way ship systems people keep using.