Designing AI Systems: RAG, Agents & Inference at Scale

The systems engineering underneath the magic trick: how a request flows from prompt to answer, RAG vs fine-tuning, agent loops, vector databases, the economics of inference (batching, KV-cache, quantization, caching), and why evals replace dashboards.

Advanced · 22 min read

Why this matters

You can build a chatbot in an afternoon. Ship it to ten thousand users and it stops being a demo and starts being a systems problem: every answer costs real money and real milliseconds, the model confidently invents facts it never knew, and your monitoring dashboard stays green while users get nonsense. Interviews at AI companies have started asking exactly this — not "can you prompt a model" but "how would you serve a million requests a day without going broke, and how would you know the answers are any good?" This lesson is the engineering underneath the magic trick: the pipelines, the caches, and the measurement that turn a clever model call into a product you can actually run.

Video: Building the Right AI Architecture — Practical AI
Rackspace's chief AI officer explains why AI success needs architecture beyond the model: compute, data, inference, orchestration, governance.

The restaurant you're about to run

This lesson's running analogy: your AI product is a restaurant. One analogy, stretched across the whole lesson, with the mapping stated plainly:

Keep this kitchen in your head. Every section is one part of running it well.

Video: LLM Ops Architecture: Implementing Output Validation and Structured AI Responses — Analytics Vidhya
Analytics Vidhya walkthrough of a complete LLM app: FastAPI endpoints, RAG request pipeline, guardrails, fallbacks, structured responses.

Anatomy of one request

Strip an LLM product to its skeleton and every request walks the same path: a prompt comes in, context gets assembled around it, the model does its one expensive trick, and then someone checks the work. Here's the whole machine in one picture:

flowchart LR
    U[User prompt]:::client --> G1[Input guardrails<br/>PII, injection screen]:::security
    G1 --> R[Context assembly<br/>retrieval, tools, history]:::data
    R --> M[Inference<br/>the model call]:::service
    M --> G2[Output guardrails<br/>filters, citation check]:::security
    G2 --> E[Trace + eval log<br/>what happened, what it cost]:::cloud

The unglamorous truth this diagram hides: the model is the most expensive and least reliable part of your stack. Everything around it exists for two reasons — feed it better context, and check its work. The front of house takes the order (the prompt), checks the reservation list (input guardrails: strip PII, screen for prompt injection), pulls the right recipe cards (retrieval), the chef cooks (inference), the dish gets tasted before it leaves the kitchen (output guardrails), and the ticket is filed with its cost (tracing). Get this pipeline right and you have a product. Get it wrong and you have a very expensive random answer generator.

The rest of this lesson is a tour of the hard parts, left to right.

Video: The Anatomy of AI Agents - Part 1: How LLMs Actually Work — BlueRider.Software
41-minute tech talk tracing a single inference: tokenization, next-token prediction, temperature, autoregressive generation, structured output.

RAG: give the chef a library card

The chef doesn't know your company. You could teach it — retraining is possible, and we'll get to it — but there's a cheaper move first: hand it the recipe cards at order time. That's retrieval-augmented generation (RAG): instead of stuffing knowledge into the model's weights, you keep knowledge in files and fetch the relevant pages into the prompt for each question. The original paper's framing is worth keeping: the model has parametric memory (what's baked into the weights) and non-parametric memory (the index you can swap without retraining).

The pipeline has five stages, and each one is a place quality can quietly die:

  1. Ingest and chunk. Documents go in; passages come out. You chunk because models have fixed context windows and because retrieval works better on focused passages — a whole 200-page manual is a terrible search result. The dial: chunks too big and each one is diluted soup; too small and fragments lose their meaning. The old default — fixed-size chunks with a slight overlap so a sentence split at a boundary still survives — has real competition now. Anthropic's contextual retrieval (Sept 2024) has the model write a one-paragraph summary of each chunk's document and prepends it, so the chunk carries its context into the index. Semantic chunking splits on topic boundaries instead of character counts. Late chunking embeds whole documents first and pools token vectors into chunks afterward, so no chunk is ever embedded in isolation. All three attack the same failure: a chunk that made sense on the page and means nothing alone. Which one wins depends on your corpus — measure on your evals.
  2. Embed. Each chunk becomes a vector — a long list of numbers that captures meaning. Sentences that mean similar things land near each other in this space, even with completely different words. This is the card catalog sorted by meaning instead of alphabet.
  3. Index. The vectors go into a vector database (more on those shortly), alongside the original text and metadata like source URLs and timestamps.
  4. Retrieve and rerank. At query time you embed the question, find the nearest chunks, and — the step beginners skip — rerank. A cheap, fast retriever grabs fifty candidates; a slower, sharper model re-scores them down to the best five — the sous-chef re-checking the stack of pulled cards and keeping only the ones actually worth cooking from. Retrieval is a funnel: wide and cheap at the top, narrow and careful at the bottom.
  5. Generate. The question plus the winning chunks go into the prompt, and the model answers with the evidence in front of it. Ask it to cite which card each claim came from and you get provenance for free.

Watch a query fan out across the index:

Interactive diagram: PacketFlow (loads in the app)

When RAG wins, and when it doesn't

RAG wins when the knowledge changes, is private, or needs receipts. Company docs that update weekly, a codebase the model never saw in training, any answer where "because the model said so" isn't good enough — library card beats memory every time. Swapping the index is a file operation; retraining is a project.

RAG loses when the knowledge is really a skill: writing in your brand's voice, following a tricky format, exercising judgment on ambiguous cases. No recipe card teaches a chef timing. It also loses on tiny corpora that fit in the prompt anyway (just paste them in), and on questions where retrieval adds 300 milliseconds of latency for zero gain — not every question needs the library.

That "just paste them in" line got a lot more literal in 2025. GPT-4.1 takes a million tokens, Gemini takes two million, Llama 4 Scout takes ten — so the honest 2025 architecture question is: why not skip the library and paste the whole corpus in? Three reasons it doesn't always win. First, cost: every one of those million tokens rides every query, and the KV-cache math below is linear in context — retrieval keeps per-query cost roughly flat while long-context prefill grows with the corpus. Second, latency: prefilling a million tokens takes real seconds; a vector lookup stays in the hundreds of milliseconds. Third, attention: models famously lose instructions buried in the middle of very long prompts, so a needle-in-a-haystack corpus can answer worse than a sharp retriever plus a short prompt. The working rule: long context wins when the corpus is medium-sized and the query needs the model to see all of it at once; RAG wins when the corpus is large, changes often, or answers need receipts. Where that line sits is the RAG architecture interview of 2025 — have an answer.

And the honest failure mode: garbage in, gospel out. RAG doesn't fix a bad retriever; it amplifies one, because the model now answers with great confidence about the wrong cards. If your answers are wrong, debug retrieval before you blame the model: log which chunks were fetched, read them yourself, and ask whether you could have answered from them. The usual fixes are better chunking, hybrid search (keyword plus vector, catching what each misses alone), and the reranking step from above.

So when do you skip the library card and teach the chef by heart instead?

Interactive diagram: VsToggle (loads in the app)

One more thing worth knowing: they're not rivals. Plenty of production systems do both — fine-tune for the house style, retrieve for the facts. The chef learns the restaurant's standards by heart and still looks up the recipes.

Video: Local Retrieval Augmented Generation (RAG) from Scratch (step by step tutorial) — Daniel Bourke
Daniel Bourke explains why LLMs need retrieval, then builds a working RAG pipeline from scratch: chunking, embeddings, search, generation.

Agents: the chef starts running errands

A chatbot is a chef who never leaves the kitchen: one question in, one answer out, all from memory. An agent is a chef who runs errands — it plans, takes an action in the world, looks at what happened, and adjusts. The canonical shape of the loop comes from the ReAct paper: the model interleaves thoughts (reasoning traces: "I need the user's order history first") with actions (tool calls: query the orders API) and observations (the tool's result), round after round, until it can answer or finish the task.

Three ingredients separate an agent from a chatbot:

Walk one loop yourself:

Interactive diagram: StepThrough (loads in the app)

One 2025 update to the "tools" ingredient: the industry converged on MCP (Model Context Protocol) — an open standard for exposing tools to models, so one server definition works across assistants and agents instead of every vendor getting a bespoke function schema. Describing tools purely as ad-hoc JSON schemas is 2023-era advice now. And agents started calling each other: A2A (Agent-to-Agent) is the protocol for agent-to-agent handoffs — one intern formally delegating to another, task and context passed along.

Anthropic's field guide to this draws the line you've probably felt: workflows (predefined code paths where the model fills in steps) versus agents (the model dynamically directs its own process). Start with workflows. Graduate to agents when the number of steps genuinely can't be predicted up front. Most "agents" in production are workflows wearing a trench coat, and that's fine — simple and predictable beats clever and surprising when you're on call for it.

Now the failure modes, because agents fail in ways chatbots don't:

Video: AI Agents in Practice • Henrik Kniberg • GOTO 2025 — GOTO Conferences
GOTO talk defines AI agents, covers tools and agent architecture with practical design tips.

Vector databases: the card catalog

A vector database does one thing: it stores embeddings and answers "what's near this point?" fast, at a scale where a linear scan would take forever. Scanning ten thousand vectors per query is fine. Scanning ten million is a non-starter — so vector DBs use approximate nearest neighbor (ANN) search: data structures that find almost the nearest vectors while looking at only a fraction of the index.

The entire field is one triangle — recall vs. latency vs. cost — and every index is a point on it:

flowchart LR
    Q[Query vector]:::client --> I[ANN index]:::data
    I -->|check fewer candidates| F[Fast, cheap<br/>may miss the best chunk]:::service
    I -->|check more candidates| A[Slower, pricier<br/>finds better chunks]:::service

Graph-based indexes (the HNSW family) walk a neighborhood graph: very fast queries, bought with RAM — the whole graph likes to live in memory. Partition-based indexes (the IVF family) split the space into clusters and search only the nearest ones: cheaper to host, but you tune how many clusters to visit, and that tuning is the recall-latency dial. There's no free lunch, only the point on the triangle your workload can afford.

Three practical notes from people who've run these in production:

  1. Metadata filtering comes first. "Nearest chunk from the 2024 docs" — filter by source, date, or permissions before or alongside the vector search. RAG over documents the user isn't allowed to see is a security incident, not a feature.
  2. Changing embedding models is a migration. Vectors from one model aren't comparable to another's, so a model upgrade means re-embedding the whole corpus. Version your embeddings like you'd version a schema.
  3. The index is not the product. Nobody ever said "our vector DB has great recall" about a system with bad chunking. The cards matter more than the catalog.

Video: [AI Stack 15] Vector Databases vs. Traditional Search: What You Need to Know — The AI Stack
Explains how meaning becomes geometry: embeddings, cosine similarity, ANN indexes like HNSW, and tools like FAISS, pgvector, Pinecone.

Inference at scale: the systems-engineering side

Here's where this lesson earns its "advanced" badge. Serving a model to real traffic is a resource-management problem with real numbers, and the numbers are unforgiving.

Two phases, two bottlenecks. Every request has a prefill phase (read the whole prompt at once, build up internal state — compute-bound, fast) and a decode phase (generate one token at a time, each depending on all previous tokens — memory-bandwidth bound, slow). Decoding dominates both latency and cost, and everything in this section is about making it cheaper.

The KV cache. To avoid recomputing attention over earlier tokens at every step, the server caches their key and value tensors — the KV cache. It's a straightforward trade: memory for speed. How much memory? Do the napkin math for a typical 8-billion-parameter dense transformer (32 layers, 4096 hidden size) in FP16:

That math assumes every attention head keeps its own keys and values — full multi-head attention. Real 8B-class models don't: every serious small model since ~2023 ships with grouped-query attention (Llama 3's 8B keeps 8 key-value heads instead of 32), which shrinks the cache 2–4×. The honest number for that 128k conversation on a real 8B model is closer to ~16 GB, not 64. Keep the full-MHA arithmetic as the baseline — the scaling law (linear in layers × hidden × context) is what matters — and remember the headline number moves with the architecture.

That is why long contexts are expensive, and why the cache — not the model weights — is usually what caps your concurrency. The vLLM team attacked exactly this with PagedAttention: manage the KV cache in fixed-size blocks like an OS manages virtual memory, killing fragmentation and letting requests share prefix blocks. Their paper reports 2–4× serving throughput at the same latency. When a systems paper borrows its core idea from operating systems, pay attention.

Batching. Decode is memory-bandwidth-bound: the GPU spends most of its time shuttling weights in, not computing. So you batch — run many requests' decode steps together and amortize one weight read across all of them. Modern servers do this continuously, slotting new requests in the moment old ones finish rather than waiting for a whole batch. Throughput goes up; per-request latency barely moves. It's the dinner rush done right: the kitchen doesn't wait for every table's order to finish before firing the next — a free burner gets the next ticket the instant it opens up.

Quantization. Weights at FP16 are 2 bytes per parameter; an 8B model is ~16 GB before the cache. Drop to INT8 and you halve it; INT4 quarters it — smaller model, bigger batches, cheaper GPUs, at some cost in answer quality. Since 2024 there's a middle path: FP8, the default serving format on Hopper and Blackwell GPUs with native vLLM support — roughly half the memory of FP16 with quality much closer to it than INT8 manages. The tradeoff is workload specific: code generation notices the difference more than summarization does. Measure on your evals, not a benchmark leaderboard.

Caching, three flavors. First, prompt caching: multi-turn conversations and agent loops re-send the same prefix (system prompt, tool definitions, retrieved docs) every call. Providers can reuse the KV state for the matching prefix instead of recomputing it — OpenAI's docs report up to ~80% faster time-to-first-token and about half-price cached input tokens for prompts over 1024 tokens (Anthropic's prompt caching goes further on price, discounting cache reads up to 90%, and charges a small premium to write the cache), and the trick is prompt order: stable content first, the changing bits last. Second, semantic caching: if someone asks a question close enough in meaning to yesterday's — matched by embedding similarity, not exact text — skip the model entirely and serve yesterday's answer. Not the regular who orders the exact same dish; the regular whose order you recognize however they phrase it. Third, plain old response caching for anything deterministic. Every cache is the same bet: repeated work shouldn't be paid for twice.

Here's the serving path in one picture:

flowchart LR
    Q[Waiting requests]:::client --> B[Continuous batcher]:::service
    B --> P[Prefill<br/>build KV cache]:::cloud
    P --> D[Decode<br/>one token at a time]:::cloud
    D --> K[(Paged KV blocks)]:::data
    K -->|prefix shared| C[Prompt cache hit<br/>skip recompute]:::data

Video: Optimize LLM inference with vLLM — Red Hat
Red Hat engineer and vLLM contributor explains PagedAttention, continuous batching, and prefix caching for efficient LLM serving.

Evals & observability: the mystery diners

Your dashboards will lie to you. Latency, error rate, GPU utilization, uptime — they tell you the kitchen is standing and the stoves are hot. They say nothing about whether the food is any good. A model can answer in 200 milliseconds, with zero errors, and be completely, confidently wrong. Traditional monitoring measures the system; AI products need measurement of the answers.

Eval harnesses. An eval is a test suite for behavior: a dataset of questions with expected answers, run automatically whenever the model, the prompt, or the retrieval pipeline changes. Your mystery diners, with scorecards. Build them from real user traffic (anonymized), cover your failure modes on purpose, and score three ways: exact-match where answers are deterministic, human grading on samples where taste matters, and model-as-judge for the vast middle — with the caveat that your judge needs its own evals, because a biased judge just launders your blind spots. The discipline that matters: evals run in CI, on every change, with tracked scores over time. A prompt tweak that "felt better" but dropped factuality scores 4 points is a regression, and now you can prove it.

Tracing. When a user reports a bad answer, you should be able to replay the whole ticket: the exact prompt version, which chunks were retrieved (and their scores), every tool call and its result, token counts, and what it cost. Log all of it, keyed by request ID. Debugging an agent without traces is like debugging a distributed system without logs — technically possible, spiritually crushing.

Guardrails. Deterministic code wrapped around a non-deterministic core. Input side: strip PII before it reaches the model, screen for prompt-injection patterns, enforce length and rate limits. Output side: content filters, citation checks for RAG answers, refusals for out-of-scope requests, and — the one everyone forgets — a final deterministic validation when the model emits structured data (parse the JSON, check the schema, reject and retry rather than shipping garbage downstream). That's the single-request shape; the full guardrail taxonomy for agents — action allowlists, idempotent tools, human-in-the-loop checkpoints — is the agentic-patterns lesson's home turf, and it builds directly on this one.

The mental model to keep: evals are how you know the product is good, tracing is how you find out why it wasn't, and guardrails are how you limit the blast radius while you fix it. Dashboards watch the restaurant. Mystery diners taste the food.

Video: Build an Eval Loop for More Reliable Agents (Evals 101) — Mastra
Mastra team workshop on eval loops: scorers, LLM judges, regression tests, dataset experiments, and monitoring production traces.

Takeaways

  1. Every LLM request walks one pipeline: prompt → guardrails → context assembly → inference → guardrails → trace. The model is the most expensive, least reliable part; everything around it feeds it better context and checks its work.
  2. RAG puts knowledge in files, not weights. Chunk, embed, index, retrieve, rerank, generate. It wins when facts change, data is private, or answers need receipts; the failure mode is garbage in, gospel out — debug retrieval before blaming the model.
  3. Fine-tuning teaches behavior; RAG teaches facts. Use both: fine-tune for house style, retrieve for knowledge. The comparison above is the decision in one screen.
  4. An agent is a loop, not a call. Plan → act (tool use) → observe, with the model choosing each step. Cap iterations and dollars, treat tool output as untrusted, and start with workflows until the steps genuinely can't be fixed in advance.
  5. Vector DBs sell one triangle: recall vs. latency vs. cost. Approximate search is the only option at scale; metadata filtering and chunk quality matter more than which index you pick.
  6. Inference economics are KV-cache economics. Decode is memory-bandwidth-bound, so batch continuously, page the cache (PagedAttention: 2–4× throughput), quantize deliberately, and cache repeated prefixes — stable content first.
  7. Dashboards can't taste the food. Evals (behavioral test suites in CI), tracing (replay any bad answer), and guardrails (deterministic checks around a non-deterministic core) are the observability stack for AI.

Check your understanding

  1. Your company's refund policy changes every quarter. Why is RAG a better fit than fine-tuning for answering policy questions?

    • Fine-tuning bakes the whole policy into the weights, so each quarter's update applies itself automatically
    • RAG keeps the policy in swappable files, with citations; fine-tuning would need retraining on new facts
    • RAG skips the retrieval step, so its answers arrive faster than any fine-tuned model's could
    • Fine-tuning lets the model cite the exact policy section, so every answer carries receipts

  2. In the restaurant analogy, what does the reranking step of a RAG pipeline correspond to, and why does it exist?

    • The chef tasting the finished dish before it leaves the kitchen, as a final quality check
    • The front of house rewriting the customer's question into terms the kitchen understands
    • The expediter carefully matching every finished plate against the customer's ticket before it goes out
    • The sous-chef re-checking pulled cards and keeping the best five; the first pass is wide by design

  3. What is the key difference between a chatbot and an agent, in this lesson's terms?

    • A chatbot answers from memory in one shot; an agent runs plan → act → observe, choosing each step itself
    • An agent uses a bigger model than a chatbot, so it can plan further ahead before it acts
    • An agent follows a fixed script of tool calls, while a chatbot improvises every response
    • They are really the same thing hiding under different marketing names; the underlying model call is identical

  4. You switch your vector index to check fewer candidates per query. What moves on the recall–latency–cost triangle, and what do you accept?

    • Recall goes up, because the index spends more care on each candidate it does check
    • Nothing changes — approximate search returns the same results at any candidate count
    • Latency and cost fall, but recall falls too — the best chunk might never get checked
    • Cost goes up, because the index must shard the smaller query across more partitions

  5. A 128k-token conversation on a real 8B-class model holds roughly 16 GB of KV cache per request — grouped-query attention shrinks the naive 64 GB estimate about 4×. What serving conclusion does the lesson draw?

    • The cache is now small enough that model weights, not the KV cache, cap concurrency instead
    • The KV cache — not the weights — still caps concurrency, which is why PagedAttention exists
    • Prefill becomes the bottleneck, so continuous batching stops helping throughput at all
    • Quantization can no longer reduce memory, since GQA already shrank the cache away

  6. Your AI product's dashboard shows low latency and zero errors, but users complain the answers are wrong. What is the lesson's prescribed fix?

    • Add more GPUs, since lower latency will pull answer quality up along with it
    • Switch to a bigger model, since undersized models are the usual cause of wrong answers
    • Disable the guardrails, since they're the most likely source of the bad answers
    • Dashboards measure the system, not the answers — add evals, tracing, and guardrails

Go deeper

Want to keep pulling this thread? These talks and tutorials go further than we did here:

Sources & further reading