RAG Essentials

You’re asked to connect an AI assistant (e.g., ChatGPT) to a company repository containing 500 GB of documents so the assistant can answer questions about that data. Typical chat UIs can only accept a few files, so you need a solution that lets the LLM search, read, and reason over the entire dataset without rescanning 500 GB on every query. A naive approach — scanning raw files or running fresh summarization on every query — is too slow or loses important context. The recommended pattern is Retrieval-Augmented Generation (RAG): convert documents and queries into semantic vectors (embeddings), store them in a vector database, and at query time retrieve the most relevant chunks to augment the LLM prompt.

A dark slide showing two document snippets labeled "service_agreement.pdf" and "employee_handbook.docx" with handwritten annotations and arrows illustrating preprocessing, summarization, and how key context (service terms, policies, benefits) can be lost.

What RAG does (high-level)

RAG splits the work into three clear stages:

Retrieval
Convert documents and the incoming query into embeddings. Use semantic search against stored vectors to find the most relevant chunks (meaning-based matches, not keyword-only).
Augmentation
Inject the retrieved chunks into the LLM prompt at runtime so answers are grounded in current, private data. No need to fine-tune the base LLM.
Generation
The LLM generates the final answer using the augmented prompt. Ideally the response cites source chunks to improve traceability.

This approach dramatically improves the assistant’s recall on private content, but it requires careful tuning: chunking strategy, embedding model choice, retrieval thresholds, and filtering all influence result quality. Chunking strategies vary by document type:

Legal/contract documents → larger chunks preserving structure.
Conversational transcripts → finer-grained chunks with more overlap to preserve context.

For a quick architecture overview:

A hand-drawn system diagram of a retrieval-augmented chatbot architecture showing an LLM, training data, document storage (~500GB files), embeddings and a vector database. It also includes a chatbot UI sketch, arrows and annotations like chunking/embedding/retrieval strategies and an example query about a service agreement.

Practical walk-through: from repo to demo UI

This section provides an end-to-end example demonstrating the core components: environment setup, vector DB initialization, chunking, embedding, ingestion, semantic search, and a minimal web UI (Flask) that ties retrieval → augmentation → generation together.

Environment setup (example commands)

Create a project directory, virtual environment, install packages, and write a ready marker:

cd /root && mkdir -p rag-project && cd rag-project
python3 -m venv venv && source venv/bin/activate
pip install uvicorn chromadb sentence-transformers openai flask
echo "READY" > /root/rag-setup-complete.txt

Example repository layout (simulated TechCorp)

TechCorp Knowledge Repository

employee-handbook/
  pet-policy.md (CEO's dog!)
  remote-work-policy.md
  benefits-overview.md
product-specs/
  cloudsync-pro.md ($1M product)
  datavault.md
meeting-notes/
  q3-planning-meeting.md
  product-launch-review.md
customer-faqs/
  general-faqs.md

Total: 500GB simulated as focused docs

Inspect and count Markdown files:

cd /root/techcorp-docs
ls -la
find . -name "*.md" | wc -l
find . -name "*.md" | wc -l > /root/doc-count.txt

Initialize the vector database (ChromaDB)

Use a persistent ChromaDB collection to store embeddings, documents, and metadata:

# init_vectordb.py
import chromadb
from chromadb.config import Settings

print("Initializing AI Brain...")
client = chromadb.PersistentClient(
    path="./chroma_db",
    settings=Settings(anonymized_telemetry=False)
)

collection = client.get_or_create_collection(
    name="techcorp_docs",
    metadata={"hnsw:space": "cosine"}
)

print(f"Brain Created: {collection.name}")
print(f"Memories: {collection.count()}")
print("AI Brain Ready!")

Run:

cd /root/rag-project && python init_vectordb.py

Chunking documents (keeps context via overlap)

A robust chunker uses overlapping windows to preserve context between chunks:

# chunk_documents.py
import os

def chunk_text(text, size=500, overlap=100):
    """Split `text` into chunks of up to `size` characters with `overlap` to preserve context."""
    if size <= 0:
        raise ValueError("size must be > 0")
    if overlap < 0 or overlap >= size:
        raise ValueError("overlap must be >=0 and < size")

    chunks = []
    start = 0
    text_len = len(text)

    while start < text_len:
        end = min(start + size, text_len)
        chunks.append(text[start:end])
        if end >= text_len:
            break
        start += size - overlap

    return chunks

if __name__ == "__main__":
    repo_dir = "/root/techcorp-docs"
    total_chunks = 0
    for root, _, files in os.walk(repo_dir):
        for fname in files:
            if fname.endswith(".md"):
                path = os.path.join(root, fname)
                with open(path, "r", encoding="utf-8") as f:
                    text = f.read()
                chunks = chunk_text(text, size=500, overlap=100)
                total_chunks += len(chunks)
    with open("/root/chunk-count.txt", "w") as out:
        out.write(str(total_chunks) + "\n")
    print(f"Document chunking complete. Total chunks: {total_chunks}")

Embedding basics (semantic vectors)

Use the SentenceTransformers family (example: all-MiniLM-L6-v2) to convert text into vectors and compute cosine similarity:

# test_embeddings.py
from sentence_transformers import SentenceTransformer
import numpy as np

print("Loading embedding model (all-MiniLM-L6-v2)...")
model = SentenceTransformer('all-MiniLM-L6-v2')
print("Model loaded!")

sentences = [
    "Dogs are allowed in the office on Fridays",
    "Pets can come to work on Furry Fridays",
    "Remote work policy allows 3 days from home"
]

print("Converting text to vectors...")
embeddings = model.encode(sentences)

def cosine_sim(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

sim_1_2 = cosine_sim(embeddings[0], embeddings[1])
sim_1_3 = cosine_sim(embeddings[0], embeddings[2])

print("Semantic Similarity Analysis:")
print(f"'Dogs allowed' ↔ 'Pets permitted' -> {sim_1_2:.3f} (Related)")
print(f"'Dogs allowed' ↔ 'Remote work'     -> {sim_1_3:.3f} (Not related)")

with open('/root/embedding-test.txt', 'w') as f:
    f.write(f"SIM_PET:{sim_1_2:.3f},SIM_REMOTE:{sim_1_3:.3f}\n")

Takeaway: both documents and queries become vectors; similarity measures semantic relatedness, not just token overlap.

Ingesting documents into ChromaDB

Read files, chunk, embed each chunk, and store IDs, documents, embeddings, and metadata:

# ingest_documents.py
import os
import uuid
from sentence_transformers import SentenceTransformer
import chromadb
from chromadb.config import Settings
from chunk_documents import chunk_text

# Connect to the same persistent client/collection created earlier
client = chromadb.PersistentClient(
    path="./chroma_db",
    settings=Settings(anonymized_telemetry=False)
)
collection = client.get_or_create_collection(name="techcorp_docs")

model = SentenceTransformer('all-MiniLM-L6-v2')

repo_dir = "/root/techcorp-docs"
all_ids, all_embeddings, all_documents, all_metadatas = [], [], [], []

for root, _, files in os.walk(repo_dir):
    for fname in files:
        if not fname.endswith(".md"):
            continue
        path = os.path.join(root, fname)
        with open(path, "r", encoding="utf-8") as f:
            text = f.read()
        chunks = chunk_text(text, size=500, overlap=100)
        if not chunks:
            continue

        embeddings = model.encode(chunks)
        for i, (chunk, emb) in enumerate(zip(chunks, embeddings)):
            uid = str(uuid.uuid4())
            all_ids.append(uid)
            all_embeddings.append(emb.tolist())
            all_documents.append(chunk)
            all_metadatas.append({
                "file": fname,
                "path": path,
                "chunk_index": i,
                "category": os.path.basename(root)
            })

# Bulk add to collection
collection.add(
    ids=all_ids,
    documents=all_documents,
    metadatas=all_metadatas,
    embeddings=all_embeddings
)

print("Ingestion complete. Chunks added:", len(all_ids))

A full ingestion run typically logs progress and prints totals (documents processed, chunks ingested).

Semantic search example

Encode the query and retrieve the top N most similar chunks:

# search_test.py
from sentence_transformers import SentenceTransformer
import chromadb
from chromadb.config import Settings

model = SentenceTransformer('all-MiniLM-L6-v2')

client = chromadb.PersistentClient(
    path="./chroma_db",
    settings=Settings(anonymized_telemetry=False)
)
collection = client.get_collection("techcorp_docs")

queries = [
    "What is the pet policy at TechCorp?",
    "Tell me about CloudSync Pro features",
    "How many days of remote work are allowed?"
]

with open('/root/search-results.txt', 'w') as results_file:
    for query in queries:
        print(f"Query: '{query}'")
        results_file.write(f"QUERY:{query}\n")

        query_embedding = model.encode(query).tolist()
        results = collection.query(
            query_embeddings=[query_embedding],
            n_results=3
        )

        print("Top Results (by semantic similarity):")
        for i, (doc, meta) in enumerate(zip(results['documents'][0], results['metadatas'][0]), start=1):
            print(f"{i}. [{meta['category']}] {meta['file']}")
            print(f"   Preview: '{doc[:100].replace('\\n',' ')}...'")
            results_file.write(f"RESULT:{meta['category']}/{meta['file']}\n")

        print("="*50 + "\n")

A search for “bring my dog” should surface the pet policy chunk even if wording differs—this is the power of semantic retrieval.

Minimal UI: Flask app for retrieval → augmentation → generation

A lightweight Flask app receives a question, runs semantic search, assembles a prompt with retrieved chunks, and calls an LLM (e.g., OpenAI) to generate a grounded answer. Include provenance (metadata + similarity scores) in responses to reduce hallucination. High-level notes:

Embedding model: all-MiniLM-L6-v2 (compact, performant).
Chunking: size 500 chars; overlap 100–400 depending on content density.
Storage: ChromaDB persistent collection techcorp_docs.
Web: Flask app for quick prototyping (port 5000).
Safety: apply similarity thresholds, metadata filters, and return provenance links to mitigate hallucinations.

A screenshot of a Q&A panel titled "Are Pets allowed to Office" showing a reply that TechCorp allows pets under specific conditions (approval, registration, vaccination proof, allowed/restricted areas). A source link labeled "TechCorp Pet Policy" with a cursor is visible at the bottom.

Operational advice and tuning knobs

Key tuning variables to monitor and iterate on:

Chunk size and overlap (balance context vs. prompt cost).
Embedding model (trade-offs: cost, latency, semantic quality).
Retrieval n_results and similarity threshold to reduce noise.
Provenance reporting in responses to help debugging and reduce hallucinations.

Best practices:

Legal/technical docs → larger chunks, higher overlap to keep structure.
Conversational logs → smaller chunks with more overlap to preserve turn-level context.
Run embedding and retrieval tests (e.g., the test_embeddings script) to validate semantic distances for your corpus.

Table: quick component summary

Component	Purpose	Example
Embedding model	Convert text → vectors	`all-MiniLM-L6-v2`
Vector DB	Persist vectors + metadata	ChromaDB (persistent collection)
Chunker	Split files while preserving context	`size=500`, `overlap=100` (tune per-corpus)
Retriever	Find top-k similar chunks	`n_results=3` (increase for recall)
App	Orchestrate RAG and serve UI	Flask on port 5000
Safety	Reduce hallucinations	similarity thresholds, provenance

Be careful with sensitive data. Embeddings and vector indexes may leak information if not properly secured. Ensure access controls, encryption at rest, and a clear data policy before indexing PII.

A vertical diagram of an information-retrieval pipeline listing components and settings. It shows Model (all-MiniLM-L6-v2), Chunking (size 500, overlap 400, stride 400), Storage (ChromaDB), App (Flask port 5000), Safety (similarity threshold filters), and Result (reduces hallucinations before web output).

Final checklist before production

Tune chunk size and overlap on a representative sample of your corpus.
Choose an embedding model that balances cost, latency, and semantic fidelity.
Implement retrieval filtering (similarity thresholds, metadata) to reduce noise.
Log provenance and similarity scores to help debug and explain answers.
Secure your vector DB and restrict access to sensitive data.

That’s it — with a vector DB, embeddings, a robust chunker, and a small web app you can move from zero to a working RAG system that returns grounded, explainable answers over large private corpora. Further reading and references:

Day 1 - AI-Powered DevOps

Day 2 - RAG Debugging

Day 3 - MCP Servers

Day 4 - Subagents

Day 5 - Production Infrastructure Crisis Management

What RAG does (high-level)

Practical walk-through: from repo to demo UI

Environment setup (example commands)

Example repository layout (simulated TechCorp)

Initialize the vector database (ChromaDB)

Chunking documents (keeps context via overlap)

Embedding basics (semantic vectors)

Ingesting documents into ChromaDB

Semantic search example

Minimal UI: Flask app for retrieval → augmentation → generation

Operational advice and tuning knobs

Final checklist before production

Watch Video

Practice Lab

​What RAG does (high-level)

​Practical walk-through: from repo to demo UI

​Environment setup (example commands)

​Example repository layout (simulated TechCorp)

​Initialize the vector database (ChromaDB)

​Chunking documents (keeps context via overlap)

​Embedding basics (semantic vectors)

​Ingesting documents into ChromaDB

​Semantic search example

​Minimal UI: Flask app for retrieval → augmentation → generation

​Operational advice and tuning knobs

​Final checklist before production

Watch Video

Practice Lab

What RAG does (high-level)

Practical walk-through: from repo to demo UI

Environment setup (example commands)

Example repository layout (simulated TechCorp)

Initialize the vector database (ChromaDB)

Chunking documents (keeps context via overlap)

Embedding basics (semantic vectors)

Ingesting documents into ChromaDB

Semantic search example

Minimal UI: Flask app for retrieval → augmentation → generation

Operational advice and tuning knobs

Final checklist before production