Using Retrieval Chain

In this lesson we’ll build a simple retrieval-augmented generation (RAG) pipeline that constrains a language model to answer using only retrieved documents. We’ll use two TechCrunch articles as the source documents:

“Anthropic claims its new models beat GPT-4” (March)
“AI21 Labs’ new text-generating AI model is more efficient than most” (March)

Workflow overview: load the articles from the web, split them into smaller chunks, create embeddings for those chunks, index them in an in-memory FAISS vector store, and then run a retrieval chain that supplies retrieved context to an LLM with instructions to answer only from that context. Key benefits of this pattern:

Keeps LLM input within the model context window.
Reduces hallucination by restricting answers to retrieved text.
Makes long-document QA, summarization, and fact-checking more reliable.

Below is a consolidated, corrected, and cleaned-up implementation that follows the same sequence as the original lesson.

# Imports
from langchain_core.prompts import PromptTemplate
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.document_loaders import WebBaseLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain.chains import create_retrieval_chain
from langchain_community.vectorstores import FAISS

# URLs to load
URL1 = "https://techcrunch.com/2024/03/04/anthropic-claims-its-new-models-beat-gpt-4/"
URL2 = "https://techcrunch.com/2024/03/28/ai21-labs-new-text-generating-ai-model-is-more-efficient-than-most/"

# 1) Load documents from the web
loader = WebBaseLoader([URL1, URL2])
data = loader.load()

# 2) Split documents into chunks (tune chunk_size/overlap for your documents)
text_splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50)
chunks = text_splitter.split_documents(data)

# 3) Create embeddings for the chunks
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")

# 4) Build an in-memory FAISS vector store and get a retriever
# Note: FAISS is in-memory by default (non-persistent).
vector = FAISS.from_documents(chunks, embeddings)
retriever = vector.as_retriever()

# 5) Define a prompt template that forces the LLM to answer only from context
prompt_template = """Answer the question {input} based solely on the context below:

<context>
{context}
</context>

If you can't find an answer, say "I don't know."
"""
prompt = PromptTemplate.from_template(prompt_template)

# 6) Initialize the LLM with low temperature to reduce creativity/hallucination
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0.0)

# 7) Create the documents combiner chain and the retrieval chain
combine_docs_chain = create_stuff_documents_chain(llm, prompt)
chain = create_retrieval_chain(retriever, combine_docs_chain)

# 8) Invoke the chain with a question that restricts answers to Anthropic and Meta only
result = chain.invoke({
    "input": "List the models and their token size of models only from Anthropic and Meta"
})

# 9) Print the returned answer
print(result["answer"])

FAISS (Facebook AI Similarity Search) is an in-memory vector index that provides fast nearest-neighbor search. It is non-persistent by default—use a disk-backed store (for example, Chroma) or persist FAISS manually for production workflows that need durability.

What the retrieval chain does

The retriever issues a nearest-neighbor search over the vector store to find the most relevant chunks for a query.
The combiner (here, a “stuff” chain) concatenates the retrieved chunks into the context variable passed to the PromptTemplate.
The LLM then answers the user query constrained by that context and the template instruction.

Example expected output (from the provided TechCrunch articles):

- Anthropic's Claude 3: 200,000-token context window
- Meta's Llama 2: 32,000-token context window

The chain returns the model’s answer together with metadata (for example, which chunks were retrieved), which helps with debugging and traceability. This mirrors the standard RAG pattern: retrieve relevant context, then synthesize the response using the LLM.

The image shows a JupyterLab interface displaying code in a Python notebook. The code appears to be related to AI models and their specifications, mentioning Anthropic and Meta's Llama 2.

Best practices and tuning

Chunk sizing:
- Default example: chunk_size=200, chunk_overlap=50.
- For long, dense documents, increase chunk_size. For short or highly topical documents, use smaller chunks.
Temperature:
- Use temperature=0.0 for deterministic answers constrained to retrieved context.
- Increase temperature only for creative tasks where you are not strictly relying on retrieved facts.
Persistence and scaling:
- For experiments and prototypes, in-memory FAISS is convenient.
- For production or multi-run persistence, use a disk-backed or managed vector database.
Prompt flexibility:
- Construct prompt_template programmatically when you need to inject dynamic instructions or system messages before creating the PromptTemplate and chain.

Tuning quick reference

Component	Purpose	Suggested settings
Text splitter	Create searchable chunks	`chunk_size=200`, `chunk_overlap=50` (tune per-doc)
Embeddings model	Semantic vectorization	`text-embedding-3-large` (example)
Vector store	Nearest-neighbor search	`FAISS` (in-memory) or disk-backed `Chroma` for persistence
LLM	Answer synthesis	`ChatOpenAI(model="gpt-3.5-turbo", temperature=0.0)` for deterministic replies
Prompt	Constrain answers to context	Use a template with explicit instructions and `<context>` injection

Links and references

Notes

The example focuses on a minimal, reproducible retrieval chain using LangChain-style primitives. Adapt components (loader, splitter, embeddings, vector store, combiner) to your infrastructure and scale requirements.
Always test with your target documents and queries to find the best chunk size, overlap, and retriever configuration for accuracy and latency.

Introduction

Overview of LangChain

Key Components of LangChain

Interacting with LLMs

Tips, Tricks, and Resources

Introduction to LCEL

Adding Memory to LLM Apps

Performing Retrieval

Implementing Chains

Using Tools

Building Agents

Conclusion

Building Blocks of LLM Apps

Using Retrieval Chain

Watch Video

Practice Lab