> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# RAG with Webpages
> Explains RAG pipeline using webpages instead of PDFs, swapping only the document loader while keeping embeddings and retrieval unchanged.
This lesson demonstrates that the Retrieval-Augmented Generation (RAG) workflow is identical whether your source is a PDF or a webpage. The only change required is swapping the document loader; everything else in the pipeline—splitting, embedding, vector store, retriever, prompt, and LLM—remains the same.
Below is a clean, corrected example that uses a webpage as the source (a Verge article in this case). The code is organized to highlight each step of the RAG pipeline.
```python theme={null}
from langchain.document_loaders import PyPDFLoader, WebBaseLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.chat_models import ChatOpenAI
from langchain.vectorstores import Chroma
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
# --- Source loader ---
# To use a PDF instead, replace the loader below with PyPDFLoader:
URL = "https://www.theverge.com/2024/4/18/24133808/meta-ai-assistant-llama-3-chatgpt-openai-rival"
loader = WebBaseLoader(URL)
# --- Load and split into documents/pages ---
pages = loader.load_and_split()
# --- Split into smaller chunks to improve embedding quality ---
text_splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50)
chunks = text_splitter.split_documents(pages)
# --- Create embeddings and build a vector store ---
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
vectorstore = Chroma.from_documents(documents=chunks, embedding=embeddings)
# --- Use the vectorstore as a retriever ---
retriever = vectorstore.as_retriever()
# --- Helper: format retrieved documents into one context string ---
def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)
# --- LLM and prompt configuration ---
llm = ChatOpenAI() # choose model/temperature as needed
template = """SYSTEM: You are a question-answering bot.
Be factual in your response.
Answer the following question: {question}
Use ONLY the context provided below: {context}
If the answer is not in the context, say you don't know."""
prompt = PromptTemplate.from_template(template)
# --- Compose the chain: combine LLM and prompt ---
chain = LLMChain(llm=llm, prompt=prompt)
# --- Run a query against the retrieved context ---
question = "What's the size of the largest Llama 3 model?"
docs = retriever.get_relevant_documents(question)
context = format_docs(docs)
result = chain.run(context=context, question=question)
print(result)
# Expected output:
# 'The largest Llama 3 model will have over 400 billion parameters.'
```
The key idea: switch the loader (for example, `PyPDFLoader` -> `WebBaseLoader`) to change your source from PDFs to webpages. The rest of the RAG pipeline—splitting, embeddings, vector store, retriever, prompt, and LLM—remains the same.
Workflow summary
| Step | Purpose | Example / Notes |
| ----------------- | ----------------------------------------------------- | ------------------------------------------------------------------ |
| Load documents | Read source content (PDF or webpage) | `WebBaseLoader(URL)` or `PyPDFLoader("file.pdf")` |
| Split into chunks | Create smaller passages for embeddings | `RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50)` |
| Create embeddings | Convert text chunks to vectors | `OpenAIEmbeddings(model="text-embedding-3-large")` |
| Store embeddings | Persist vectors for efficient retrieval | `Chroma.from_documents(...)` |
| Create retriever | Provide similarity search API over embeddings | `vectorstore.as_retriever()` |
| Format context | Aggregate retrieved docs into a single context string | `format_docs(docs)` |
| Query LLM | Send `context` + `question` to the prompt/LLM | `LLMChain(llm=..., prompt=...)` |
Best practices and tips
* Use chunk sizes that balance context fidelity and embedding cost. Typical ranges: 200–1000 tokens depending on use case.
* Persist your vector store (Chroma or other) between runs to avoid re-embedding the same documents.
* Add a retrieval filter or metadata to narrow results if you’re working with many documents.
* When querying the LLM, instruct it clearly to rely only on the provided context if factual accuracy is critical.
Links and references
* LangChain documentation: [https://langchain.readthedocs.io/](https://langchain.readthedocs.io/)
* Chroma vector database: [https://www.trychroma.com/](https://www.trychroma.com/)
* OpenAI embeddings: [https://platform.openai.com/docs/guides/embeddings](https://platform.openai.com/docs/guides/embeddings)
* The Verge article used in this example: [https://www.theverge.com/2024/4/18/24133808/meta-ai-assistant-llama-3-chatgpt-openai-rival](https://www.theverge.com/2024/4/18/24133808/meta-ai-assistant-llama-3-chatgpt-openai-rival)
Try replacing the `URL` with your own webpages or switching to `PyPDFLoader` to use PDFs. You can also plug this retrieval step into higher-level prebuilt chains for summarization, QA, or citation-aware responses.