Skip to main content
This lesson explains how to convert messy source files into clean, machine-readable artifacts suitable for retrieval-augmented generation (RAG). You’ll learn why raw files often “poison” downstream retrieval, what successful ingestion looks like, and the key parsing challenges across common file types. Why this matters: large language models and vector search expect clean, sequential, and semantically coherent text. When documents contain layout noise (headers repeated as body text, tables flattened, multi-column misordering), embeddings become noisy and retrieval quality and answer accuracy drop.
The image illustrates the problem of raw data being "poisonous" for models due to issues like formatting noise and misinterpretation of headers and tables. It features a robot icon, an input box showing different document types, and a list of problems caused by raw data.

​
What successful ingestion produces

The ingestion pipeline’s output should be consistent, searchable, and enriched with metadata so chunking and indexing produce semantically meaningful embeddings. Effective ingestion typically:
  • Extracts readable text with preserved sections and paragraphs.
  • Preserves logical structure: headings, sections, lists, and hierarchy.
  • Captures relationships: table columns and figure-caption pairings.
  • Records document metadata: source, page numbers, authors, timestamps, document type.
These outputs allow chunking to generate coherent, semantically dense units that retrieval systems can match reliably.
Design ingestion to produce clean textual units plus metadata. Chunking and indexing rely as much on structure and metadata as on raw text.

​
Three pillars of RAG data and ingestion implications

Understanding the data category helps select the right parsing strategy and chunking approach.
  • Unstructured: plain text, Markdown, simple logs. Minimal parsing required—text is ready for chunking.
  • Semi-structured: PDFs, DOCX, HTML, and other layout-aware documents. Require layout-aware parsing to recover reading order, headings, and tables.
  • Structured: databases, CSVs, spreadsheets (XLSX). Focus is on preserving relationships—columns, keys, and row semantics—so query-time reasoning remains accurate.
The image illustrates the three pillars of RAG data: unstructured (plain text, markdown), semi-structured (layout-aware documents like PDFs and DOCX), and structured (databases, spreadsheets like CSVs and XLSX).
Format categoryCommon ingestion problemsRecommended tools / approaches
Unstructured (txt, md, logs)Little to no structure; noisy tokensPython I/O, simple token cleaning
Semi-structured (PDF, DOCX, HTML)Reading-order loss, columns, detached captionspython-docx, pdfplumber, PyMuPDF, pdfminer.six, layout-parser, Apache Tika
Structured (CSV, XLSX, DB)Column relationships, type conversionpandas, openpyxl, DB connectors, schema-aware loaders
Note: wrap any example objects or placeholders as code (e.g., [{ "object": "person" }]) to avoid MDX parsing issues.

​
Why PDFs are often the hardest case

PDFs are presentation formats: content is placed by coordinates instead of logical sequence. Naive text extraction can produce:
  • Column-order collisions (left column mixed with right column).
  • Flattened or scrambled tables (cell order lost).
  • Headers, footers, and page numbers mixed into body text.
  • Captions or figure labels detached from visuals.
The image highlights the challenge with PDFs, emphasizing layout significance with a design meant for viewing rather than reading, and content stored in coordinates. A PDF cover titled "Powering the Future of Automation" is also shown.
Recovering the intended reading order typically requires layout analysis and heuristics—or ML-based document understanding—to group lines into paragraphs, separate columns, and reconstruct tables and captions.
Beware of naive PDF text extraction—without layout-aware parsing you’ll get noisy text that degrades embeddings and retrieval quality.

​
Typical ingestion pipeline pattern

A resilient ingestion pipeline usually mixes multiple specialized parsers and a normalization stage:
  1. Raw extraction: choose a parser appropriate for the file type (e.g., pdfplumber for PDFs, python-docx for DOCX, pandas for CSVs).
  2. Layout and structure recovery: use layout parsers or heuristics to reconstruct sections, columns, tables, and captions.
  3. Chunking: split content into semantically coherent chunks (by paragraph, heading, or table row), keeping chunk size aligned to your embedding model’s context window.
  4. Metadata enrichment: attach source, page range, section heading, and other helpful attributes to every chunk.
  5. Indexing: calculate embeddings and store in a vector store with metadata for filtering and retrieval.

​
Tools and libraries

  • DOCX and rich text: python-docx — preserves paragraphs and headings.
  • PDFs: pdfplumber, PyMuPDF (fitz), pdfminer.six — combine with layout-parser or Apache Tika for layout analysis.
  • Tables & spreadsheets: pandas, tabula-py, openpyxl — preserve columns and data types.
  • RAG and loader frameworks: LangChain, LlamaIndex, Haystack — provide document loaders, chunkers, and connectors for common storage backends.
Resources:

​
Practical tips for robust ingestion

  • Always retain source metadata. It enables filtering and provenance in retrieval.
  • Chunk semantically (by heading/section) rather than blindly by token count whenever possible.
  • Normalize repeated headers/footers and remove page artifacts early in the pipeline.
  • Treat tables as first-class objects: keep columns and types instead of flattening to plain text.
  • Validate with small end-to-end tests: ingest a representative sample, compute embeddings, and run retrieval queries to check quality.

​
Conclusion

Ingestion is the foundation of reliable RAG systems. Converting presentation-oriented or noisy documents into structured, metadata-rich text enables meaningful chunking and high-quality embeddings. Combine layout-aware parsing, format-specific tools, and metadata-first chunking to maintain retrieval accuracy and trustworthy responses downstream. In this course we’ll combine the tools and techniques above to build ingestion pipelines that produce clean text chunks enriched with metadata—improving embeddings and retrieval performance for RAG workloads.

Watch Video