Performing Retrieval

Welcome back. In this lesson we cover performing retrieval—an essential capability for building reliable LLM applications. By the end of this lesson you will understand Retrieval-Augmented Generation (RAG), the typical RAG pipeline, and how external context is retrieved and injected into prompts so an LLM can respond accurately with fewer hallucinations. Why retrieval matters

Context is the most important building block of an LLM application. Context means external information (documents, databases, APIs, or the web) retrieved and added to the prompt so the model has knowledge-rich background to generate factually correct responses.
When you ground an LLM’s output in retrieved evidence, you reduce hallucinations and make answers explainable and auditable.

Where does context come from?

Relational databases (SQL queries)
Full-text search indexes (e.g., Elasticsearch)
Semantic search over vector embeddings stored in a vector database
Files and documents (PDF, Word, HTML), object storage (S3), or any external API
Real-time web search for time-sensitive queries

The image displays logos of various database systems (MySQL, SQLite, PostgreSQL, MongoDB) alongside icons for different file types and storage formats (PDF, document, HTML, bucket, DOC, API).

In many applications the most common sources are documents—PDFs, web pages, or unstructured files. For time-sensitive answers, you may also pull context directly from a web search.

Retrieve only what’s relevant The primary idea is to retrieve only the relevant passages (chunks) and place them into the prompt so the model has the right facts. This:

Keeps answers accurate and grounded
Reduces token usage and cost
Limits exposure of private data to external models

RAG explained Retrieval-Augmented Generation (RAG) is the established pattern that augments LLM responses with facts retrieved from external sources so the model can answer accurately and cite sources when needed.

The image illustrates the concept of Retrieval Augmented Generation (RAG), showing the interaction between external data sources and a large language model (LLM) to enhance information with additional facts.

Why not send an entire document? Two practical reasons:

Context window limits: Every model has a finite context window—the combined size of input (prompt + context) and output. Sending a full 100-page PDF is usually impossible or inefficient.
Relevance and privacy: Sending only the relevant subset minimizes exposure of sensitive data and makes it easier to trace the source of a fact.

Instead of sending full documents, RAG retrieves the most relevant chunks (paragraphs or passages) that fit in the model’s context window. This reduces hallucinations and enables transparency: you can cite the exact source and passage used to generate an answer.

The image displays the concept of Retrieval Augmented Generation (RAG) with icons representing four aspects: Explainability, Transparency, Access Control, and Data Privacy.

Typical RAG workflow (end-to-end)

User submits a question or prompt to a chatbot or application.
The system encodes the prompt and queries a search layer (semantic or full-text) over your corpus.
Relevant context chunks are retrieved.
The retrieved context is injected into the prompt and sent to the LLM.
The LLM returns a grounded answer that can include citations, which is delivered to the user.

The image illustrates a RAG (Retrieval-Augmented Generation) workflow, showing the interaction between a user, a chatbot, a large language model (LLM), and databases/documents for search and retrieval of context.

This workflow splits naturally into two phases: indexing and retrieval. Indexing (Phase 1)

Load unstructured data (HTML, PDF, DOCX, JSON, images, etc.).
Split the data into chunks (sentences, paragraphs, or fixed-size windows of tokens). Smaller chunks improve granularity; larger chunks preserve more local context.
Convert each chunk into an embedding vector using an embeddings model (which may be distinct from your generative LLM).
Store the vectors and metadata (source, document id, chunk id, offsets) in a vector database (e.g., Chroma, Milvus, Weaviate, Qdrant).

The image illustrates "RAG – Phase 1," showing a process flow from loading to splitting and embedding, with numerical representations as output.

Note on embeddings An embeddings model encodes text into fixed-length numeric vectors suitable for semantic comparison. Use the same embeddings model for indexing and for queries to ensure consistent similarity scores. Retrieval (Phase 2)

Encode the user query with the same embeddings model used for indexing.
Perform a similarity search across your vector database to find the closest vectors (semantic matches).
Retrieve the corresponding text chunks and metadata, then assemble the context.
Inject the retrieved context into the prompt and send it to the LLM.
The LLM uses the question + retrieved evidence to generate a grounded answer.

The image illustrates the RAG (Retrieval-Augmented Generation) Phase 2 process, depicting a flow from a question through a retrieval mechanism, context creation, prompting, and processing by a large language model (LLM).

Putting the pieces together A RAG pipeline typically includes these components. Below is a concise breakdown to guide implementation.

Component	Purpose	Example / Link
Document loaders	Ingest files, web pages, or data into memory	`PDFLoader`, `HTMLLoader` (LangChain)
Text splitters	Break documents into chunks for indexing	Sentence/paragraph splitters or token-window splitters
Embeddings	Encode text chunks and queries as vectors	See provider docs for embedding models
Vector database	Store and search vectors by similarity	Chroma, Milvus, Weaviate, Qdrant
Retriever	Performs semantic search and returns top matching chunks	Retriever abstraction in LangChain or custom search layer

The image depicts a diagram related to LangChain, featuring elements like document loaders and external data sources, including icons for a web search and PDF.

Minimal RAG pseudocode Use the following high-level pseudocode as a checklist when implementing a RAG Q&A service:

# Indexing
documents = load_documents(source_paths)
chunks = text_splitter.split(documents)
embeddings = embed_model.encode(chunks)
vector_db.upsert(chunks, embeddings, metadata)

# Query time
query_embedding = embed_model.encode(query_text)
matches = vector_db.similarity_search(query_embedding, top_k=5)
context = assemble_context(matches)
prompt = build_prompt(query_text, context)
answer = llm.generate(prompt)
return answer, matches.metadata

Best practices

Use the same embeddings model for indexing and querying.
Tune chunk size to balance context preservation vs retrieval granularity.
Store rich metadata (source URL, document id, chunk offsets) so answers can include citations.
For highly sensitive data, use access controls or on-premise vector stores to minimize exposure.

References and tooling

LangChain: an ecosystem of components for building RAG pipelines and LLM apps.
Vector DBs: Chroma, Milvus, Weaviate, Qdrant

RAG combines semantic search with generation: the search finds evidence and the LLM composes the answer. For reliable semantic matching, always use the same embeddings model for both indexing and query encoding.

Introduction

Overview of LangChain

Key Components of LangChain

Interacting with LLMs

Tips, Tricks, and Resources

Introduction to LCEL

Adding Memory to LLM Apps

Performing Retrieval

Implementing Chains

Using Tools

Building Agents

Conclusion

Building Blocks of LLM Apps

Performing Retrieval

Watch Video