RAG

RAG (Retrieval-Augmented Generation) combines information retrieval with text generation — it pulls in an external knowledge base to make an LLM’s answers better.

Foundational papers:

• Lewis, P., et al. (2020). “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” NeurIPS. [paper] — the paper that started RAG
• Guu, K., et al. (2020). “REALM: Retrieval-Augmented Language Model Pre-Training.” ICML. [paper]
• Izacard, G., & Grave, É. (2021). “Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering.” EACL. [paper] — a different take on RAG

1. Context Window

What it is

The context window is the maximum amount of text an LLM can read and reason over in a single pass.

Foundational paper:

• Dai, Z., et al. (2019). “Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context.” ACL. [paper] — early work on breaking past a fixed context length

It usually has to fit:

• the user’s question
• conversation history
• the system prompt
• retrieved documents
• whatever the model has generated so far

All of that together has to stay under the model’s token limit.

Key things to know

• Anything over the limit gets truncated
• The model can only “see” what’s inside the window
• Window size directly affects:
  • how long a document the model can handle
  • how much material a RAG system can stuff in at once

Context windows across mainstream models:

2. Token

What it is

A token is the basic unit a model works with when processing text.

Foundational papers:

• Sennrich, R., Haddow, B., & Birch, A. (2016). “Neural Machine Translation of Rare Words with Subword Units.” ACL. [paper] — BPE, the basis for most modern tokenizers
• Kudo, T., & Richardson, J. (2018). “SentencePiece: A simple and language independent subword tokenizer.” EMNLP. [paper]

A token can be:

• a whole word
• a subword
• a fragment of a character

Key things to know

• Model limits are almost always counted in tokens
• Cost scales directly with token count
• Different languages tokenize at different efficiencies (Chinese is usually more token-efficient than English)

Rough rule of thumb:

• English: about 4 characters = 1 token
• Chinese: about 1-2 characters = 1 token

3. Embedding

What it is

An embedding turns text into a numeric vector that represents its meaning.

Foundational papers:

• Mikolov, T., et al. (2013). “Efficient Estimation of Word Representations in Vector Space.” ICLR. [paper] — the original Word2Vec paper
• Pennington, J., Socher, R., & Manning, C. D. (2014). “GloVe: Global Vectors for Word Representation.” EMNLP. [paper]
• Devlin, J., et al. (2018). “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” NAACL. [paper] — pioneered context-dependent embeddings
• Reimers, N., & Gurevych, I. (2019). “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” EMNLP. [paper] — Sentence-Transformers

Sentences with similar meaning end up closer together in vector space.

Key things to know

Embeddings show up in:

• semantic search
• document retrieval
• recommendation systems
• clustering

Common embedding models:

• OpenAI text-embedding-ada-002 / text-embedding-3
• Cohere Embed
• BGE (Beijing Academy of Artificial Intelligence)
• E5 (Microsoft)

4. Vector Search

What it is

Vector search finds relevant content by comparing vector similarity instead of matching keywords.

Foundational papers:

• Johnson, J., Douze, M., & Jégou, H. (2019). “Billion-scale similarity search with GPUs.” IEEE Transactions on Big Data. [paper] — the Faiss vector database
• Malkov, Y. A., & Yashunin, D. A. (2020). “Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs.” TPAMI. [paper] — the HNSW algorithm

Common similarity measures:

• Cosine similarity: $\cos(\theta) = \frac{A \cdot B}{|A| |B|}$
• Dot product: $A \cdot B$
• Euclidean distance: $|A - B|$

How it compares to traditional search

5. RAG (Retrieval Augmented Generation)

What it is

RAG is an AI system design pattern: retrieve external knowledge first, then generate the answer.

Foundational papers:

• Lewis, P., et al. (2020). “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” NeurIPS. [paper]
• Gao, Y., et al. (2023). “Retrieval-Augmented Generation for Large Language Models: A Survey.” arXiv. [paper] — a survey of the field

The basic flow:

User question → vector retrieval → relevant documents found → LLM generates the answer

Two flavors of RAG architecture

Foundational paper:

• Lewis, P., et al. (2020) laid out two variants:
  • RAG-Sequence: uses the same set of retrieved documents for the whole generation
  • RAG-Token: lets each generated token draw on a different document

Why it’s useful

• Stays current without retraining the model
• Cuts down on hallucination, since answers are grounded in retrieved facts
• Plugs straight into a company’s own knowledge base
• Answers can be traced back to their source

6. Chunking

What it is

Chunking means breaking a long document into smaller pieces so it can be embedded and retrieved.

Foundational paper:

• Lyu, C., et al. (2024). “Chunking Strategies for RAG.” arXiv. [paper]

Why it’s necessary:

• documents are usually too long to process as a whole
• embeddings work best when the content stays focused on one topic
• the context window is finite

7. Chunk Size

What it is

Chunk size is how many tokens go into each piece.

Typical ranges:

• 300–800 tokens for general use
• 100–300 tokens for Q&A
• 1000+ tokens when you need to understand a long document as a whole

The trade-off

8. Overlap

What it is

Overlap means adjacent chunks share some content with each other.

The point: avoid cutting a sentence or a key piece of information right down the middle.

Typical setting: 10%–20%

Example:

Chunk 1: [0-500 tokens]
Chunk 2: [400-900 tokens]  # 100 tokens overlap
Chunk 3: [800-1300 tokens] # 100 tokens overlap

9. Sliding Window

What it is

A sliding window is a chunking approach where neighboring chunks overlap partially with each other.

Foundational paper:

• Liu, J., et al. (2022). “Lost in the Middle: How Language Models Use Long Contexts.” arXiv. [paper] — looks at how the position of information inside a long context affects the model

Advantages:

• context isn’t lost
• less chance of a hard information break

10. Semantic Chunking

What it is

Semantic chunking splits text based on where the meaning shifts, rather than at a fixed token count.

Reference:

• Loman, T., & Tekman, B. (2023). “Semantic Chunking for RAG.” [reference]

Basic method:

1. Split by sentence
2. Embed each sentence
3. Compute similarity between neighboring sentences
4. Start a new chunk once similarity drops below a threshold

Why it helps:

• keeps semantic units intact
• improves retrieval precision

11. Hybrid Chunking

What it is

What real systems tend to use: structural splitting + a token cap

Flow:

Split by headings/paragraphs → if over the length limit → split by sentence → add overlap

Why it’s a good default:

• stable and predictable
• respects natural semantic boundaries
• balances structure against length

12. Vector Database

What it is

A vector database stores embeddings and searches them for similarity efficiently.

Foundational papers and systems:

• Johnson, J., Douze, M., & Jégou, H. (2019). “Faiss: A library for efficient similarity search.” [paper]
• Pinecone: a commercial vector database
• Weaviate: an open-source vector database
• Milvus: an open-source vector database [GitHub]
• Chroma: a lightweight vector database

What it does: finds the closest matches out of a huge pool of vectors, fast.

Core techniques:

• HNSW (Hierarchical Navigable Small World)
• IVF (Inverted File Index)
• PQ (Product Quantization)

13. Retrieval

What it is

Retrieval is the step that finds the most relevant document chunks from the vector database.

Flow:

User question → embedding → vector search → Top-K chunks

Common choices: Top-3 / Top-5 / Top-10

Retrieval strategies:

• Dense retrieval: based on vector similarity
• Sparse retrieval: based on keywords (e.g. BM25)
• Hybrid retrieval: combines both

Foundational papers:

• Robertson, S., & Zaragoza, H. (2009). “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in IR. — the BM25 algorithm
• Karpukhin, V., et al. (2020). “Dense Passage Retrieval for Open-Domain Question Answering.” EMNLP. [paper] — DPR

14. Re-ranking

What it is

Re-ranking takes the retrieved results and sorts them again to push the more relevant ones to the top.

Foundational papers:

• Nogueira, R., & Cho, K. (2019). “Passage Re-ranking with BERT.” arXiv. [paper]
• Nogueira, R., et al. (2020). “Document Ranking with a Pretrained Sequence-to-Sequence Model.” EMNLP. [paper]

Flow:

Vector retrieval Top-20 → re-ranker model → Top-3 → LLM answers

What it buys you: better retrieval precision, less irrelevant content getting through.

Common re-ranker models:

• BGE Re-ranker
• Cohere Re-rank
• Cross-Encoder models

15. Hallucination

What it is

Hallucination is when a model generates something that sounds plausible but is factually wrong.

Foundational papers:

• Maynez, J., et al. (2020). “On Faithfulness and Factuality in Abstractive Summarization.” ACL. [paper]
• Zhang, Y., et al. (2023). “Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models.” arXiv. [paper]
• Ji, Z., et al. (2023). “Survey of Hallucination in Natural Language Generation.” ACM Computing Surveys. [paper]

Common causes:

• the right document just isn’t retrieved
• not enough context
• an ambiguous prompt
• the model’s knowledge is out of date

Ways to reduce it:

• ground the model with RAG
• ask the model to cite its sources
• give it an explicit “I don’t know” option
• verify across multiple passes

16. Debugging a RAG System

The usual order to check things in:

1. The user’s query (is it actually clear?)
2. The Top-K chunks retrieved (did it find anything relevant?)
3. The re-ranking output (is the ordering right?)
4. The prompt (are the instructions clear?)
5. The LLM itself (is it actually capable enough for the task?)

Where problems tend to come from, in practice:

• ~80% from retrieval
• ~15% from the prompt
• ~5% from the model’s own capability

Debugging tools:

• LangSmith
• Arize Phoenix
• Weights & Biases

17. The Full RAG Pipeline

Documents
  ↓
Chunking
  ↓
Embedding
  ↓
Vector Database (storage)
  ↓
Retrieval (Top-K)
  ↓
Re-ranking
  ↓
LLM generates the answer

Wrapping Up

The whole point of RAG, in one line:

Get the LLM the most relevant knowledge it can fit into a limited context window, so it can give an accurate answer.

Where the real optimization work happens:

• chunking strategy
• embedding quality
• retrieval accuracy
• re-ranking
• prompt design