AI 101 GUIDE
Original
Chapter 3 • 9 min read • Last reviewed: June 2026

RAG & Context Windows

An AI model has two kinds of memory. First: Parametric Memory, information baked into model weights during training. Second: Working Memory, the space available in the current prompt, also called the Context Window. Reliable systems use both together through RAG and long-context architecture instead of letting the model freestyle.

Why parametric memory gets exposed

Only trusting what the model memorized has three huge problems:

  1. Knowledge Cutoff: The model only knows what existed before training finished.
  2. Hallucinations: For obscure facts, models can confidently guess and produce plausible-sounding nonsense.
  3. No Access to Private Data: Models cannot see your local PDFs, company emails, or secure databases unless you provide them.

Retrieval-Augmented Generation (RAG)

RAG solves this by making the model an open-book test taker. Instead of answering from memory, the system searches an external database, inserts the relevant docs into the context window, and asks the model to answer from that evidence.

How the RAG pipeline actually works

Docs PDFs, pages Chunks Embed Vectors Vector DB query retrieves nearest chunks LLM
  1. Chunking: Large docs, like a 100-page manual, get split into small chunks.
  2. Embeddings: An Embedding Model turns each chunk into a vector: numbers that represent meaning.
  3. Vector Database: Those vectors go into a specialized database like Pinecone, Chroma, or pgvector.
  4. Retrieval (Semantic Search): When the user asks a question, the system vectorizes it and finds the chunks closest in meaning.
  5. Augmentation & Generation: The system fetches those chunks, puts them beside the user question, and sends the package to the LLM: "Here is the context: [chunks]. Answer this question based on that context: [query]."

Context windows leveled up

If RAG is that useful, why not dump the whole database into the model? Historically, attention made that impossible.

Standard self-attention memory and compute scale quadratically ($O(N^2)$) with input length. Double the input, and you need four times the compute and memory. Early models were capped around 2,048 tokens, roughly 1,500 words.

Recent architecture and serving breakthroughs broke that wall. Frontier systems now commonly offer hundreds of thousands to millions of tokens of working memory, making it possible to analyze long PDFs, codebases, transcripts, and multi-file projects in one request. The big levers are:

1. FlashAttention

Introduced by Tri Dao, FlashAttention is a software optimization. It does not change attention math; it changes GPU memory movement. Standard attention writes huge intermediate tables between slower HBM and fast on-chip SRAM. FlashAttention computes in blocks and keeps data in SRAM as much as possible, reducing memory traffic by up to 20x and letting context windows scale.

2. Rotary Position Embeddings (RoPE)

Older absolute position systems struggled with contexts longer than training length. RoPE represents positions by rotating word vectors in a multi-dimensional space. Because the rotation is relative, the model can track word distances even when the total text is much longer than training length. That lets teams extend context windows after training with minimal fine-tuning.

The "Needle in a Haystack" test

Just because a model can accept a million tokens does not mean it is actually reading them. Researchers test this with the Needle in a Haystack (NIAH) test.

A random fact, the "needle," gets hidden inside a massive document dump, the "haystack." The model must answer a question that depends on that exact fact. Modern models need near-perfect accuracy no matter where the needle appears.

Long context still is not a free RAG replacement. Million-token prompts can be slower, pricier, and harder to audit than a clean RAG pipeline. In real systems, engineers often combine both: retrieval grabs the best evidence, then long context handles cross-document synthesis, codebase-wide reasoning, or comparisons across many artifacts.

RAG vs. Long Context: How to Choose

Use RAG when the corpus is large, frequently changing, permissioned, or needs precise citations. Retrieval keeps prompts smaller, makes source selection auditable, and lets the application enforce access control before the model sees anything.

Use long context when the task needs comparing many pieces at once: reviewing a pull request across files, reconciling a contract with its exhibits, summarizing a full transcript, or finding contradictions across a small document set.

The most reliable pattern is often hybrid: retrieve the best candidates first, rerank them, then give a long-context model enough surrounding material to synthesize rather than quote isolated snippets. The eval should check both steps: did retrieval find the right evidence, and did generation stay faithful to it?

Sources