Skip to main content
Retrieval is the next essential building block for production-ready applications that use large language models (LLMs). Why it matters: LLMs are trained on data up to a fixed cutoff and therefore can’t natively answer questions about documents or events created after that checkpoint. Retrieval bridges this gap by bringing relevant, external context into prompts so an LLM can reason over up-to-date, domain-specific content (PDFs, web pages, APIs, knowledge bases, etc.). At a high level, a retrieval pipeline separates expensive, offline work from fast, online lookups. This separation reduces latency and cost while improving accuracy and relevance. A typical retrieval pipeline has two phases:
  1. Ingestion (offline / pre-processing)
    • Acquire data from sources: PDFs, XML, APIs, web pages, databases, and more.
    • Clean and transform documents, extract metadata, and split text into passages or chunks that are retrieval-friendly.
    • Convert each passage into a vector embedding using an embedding model (e.g., OpenAI embeddings, Hugging Face models).
    • Store embeddings and metadata in a vector database (e.g., FAISS, Pinecone, Weaviate) that supports efficient similarity search and filters.
  2. Retrieval (online, per-request)
    • Encode the user query into an embedding.
    • Perform a nearest-neighbor search against the vector database to find the most relevant passages.
    • Retrieve and rank passages, then inject that context into the prompt sent to the LLM.
The image is a flowchart illustrating a data retrieval process from sources like PDF, XML, and APIs, which are loaded, transformed, embedded as numerical vectors, and stored in a vector database.
Because ingestion is decoupled from runtime retrieval, you avoid re-querying raw sources for every prompt. This yields lower latency, reduced cost, and more consistent, reproducible results. Invest most engineering effort in the ingestion and indexing steps (document splitting, embedding selection, vector DB configuration, metadata design, and update strategies)—these determine retrieval relevance and downstream LLM quality. Table: Retrieval pipeline phases at a glance Best practices
  • Document splitting: Use semantic-aware chunking (sliding windows, section-aware splits) so passages preserve complete facts and context.
  • Embedding selection: Match the embedding model’s capabilities to your domain and query types. Re-embed only when necessary (e.g., model upgrades, significant content changes).
  • Metadata design: Store document identifiers, section headings, source timestamps, and any domain-specific tags to enable filtering and provenance.
  • Vector DB configuration: Tune index type, distance metric, and recall/latency trade-offs for your expected query patterns.
  • Update strategy: Combine batch re-indexing with incremental updates for new or changed documents to keep latency and cost manageable.
Practical tips for better retrieval and LLM responses
  • Use passage-level retrieval rather than whole-document retrieval to keep the injected context concise and focused.
  • Apply simple reranking (BM25 or an LLM-based reranker) after vector similarity to improve precision.
  • Include provenance and source links in the final LLM response so users can verify facts.
  • Monitor retrieval quality via relevance metrics and user feedback loops, then iterate on splitting and embedding choices.
Design the ingestion and update processes (batch updates, incremental indexes, and re-embedding strategies) carefully. Good document splitting, metadata, and embedding selection significantly improve retrieval relevance and downstream LLM responses.
Links and references

Watch Video