Skip to main content
This guide shows how to load a web page into LangChain so you can use its content as context for chatbots, retrieval-augmented generation (RAG), or other retrieval tasks. We’ll demonstrate using an article from The Verge about Meta’s AI assistant and Llama 3. The process covers:
  1. Loading the page with a web loader
  2. Inspecting the returned Document(s)
  3. Splitting (chunking) the text for embedding or indexing
The image is an article about Meta's competition with ChatGPT, discussing the introduction of Meta's AI assistant across platforms like Instagram, WhatsApp, and Facebook, along with the release of their AI model, Llama 3. It includes an event photo showing a presentation with large mobile UI mockups in the background.

​
1) Load the page using WebBaseLoader

WebBaseLoader fetches and parses a page, returning a list of LangChain Document objects. It often captures metadata such as the source URL and title. Example:
Common immediate check:

​
2) Inspect the loaded Document

A single Document usually contains full page content in page_content and any available metadata in metadata.
Web pages are usually loaded as a single Document (so len(data) is often 1). Each Document has page_content (the full text) and metadata (title, source URL, etc., when available). Use these fields when adding provenance to your index or when building prompts that reference sources.
Respect website terms of service and robots.txt when scraping or ingesting web content. For production ingestion, consider rate limits, caching, and error handling for transient network issues.

​
3) Split (chunk) the text for embedding or indexing

Large documents should be split into smaller overlapping chunks before embedding or indexing. The RecursiveCharacterTextSplitter is a good default for general-purpose chunking. Example:
Each element in chunks is a Document representing an excerpt of the original page. These chunks are ready to be passed to an embedding model or a vector store.

​
Typical workflow after chunking

  1. Create embeddings for each chunk.
  2. Store embeddings and chunk metadata in a vector database.
  3. Retrieve relevant chunks at query time and use them as context in a prompt to an LLM.
Links and references:

​
Quick reference table

This completes loading a webpage and preparing it for chunking and embedding. From here, you can proceed to create embeddings, insert into a vector store (e.g., Pinecone, FAISS, Milvus), and build retrieval-augmented prompts for downstream applications.

Watch Video