Demo Testing your RAG System

Now that the Retrieval-Augmented Generation (RAG) pipeline is running, the next step is systematic testing: send a variety of questions, inspect retrieved context, identify failure modes (hallucination, poor recall, or ambiguous answers), and then tune ingestion and retrieval parameters.

A slide titled "Testing your RAG System" on a light background with a large dark curved shape on the right featuring the word "Demo" in blue. Small copyright "KodeKloud" appears in the bottom-left corner.

This demo used a small mixed corpus containing two classic books: The Adventures of Sherlock Holmes and Frankenstein. Below are the queries we used, how we interpreted the outputs, and which knobs we adjusted to improve results.

Quick overview of core code (abbreviated)

These helper functions are used throughout the examples below.

# app.py - core imports and helper placeholders
import argparse
import os
import re
import json
import pickle
from pathlib import Path
from typing import List, Dict, Tuple
from tqdm import tqdm
import chromadb
from rank_bm25 import BM25Okapi
import ollama

def read_text_files(root: Path) -> Dict[str, str]:
    ...

def chunk_text(text: str, chunk_size: int = 800, overlap: int = 150) -> List[str]:
    ...

def tokenize(s: str) -> List[str]:
    ...

def rrf_merge(list_a: List[str], list_b: List[str], k: int = 60, topn: int = 5) -> List[str]:
    ...

Simple interactive test — mixed corpus

Run the interactive test against the hybrid retriever (vectors + BM25):

(.venv) jeremy@MACSTUDIO BookSearch % python hybrid_rag.py ask --query "Who is the narrator of this document?"

Example answer when both books were indexed:

=== Answer ===

The narrator of "The Adventures of Sherlock Holmes" appears to be someone who is friends with Sherlock Holmes, likely Dr. John Watson, as indicated by text in chunk 324 where it says "I remember" and "you on one occasion, in the early days of our friendship". However, the name "Dr. John Watson" is not explicitly mentioned in the provided context.

In contrast, the narrator of "Frankenstein" is not clearly identified in the given chunk.

----- Sources -----
adventuresofsherlockholmes.txt   (chunk 324)
frankenstein.txt                 (chunk 48)
adventuresofsherlockholmes.txt   (chunk 467)
adventuresofsherlockholmes.txt   (chunk 1)
adventuresofsherlockholmes.txt   (chunk 447)

Because both books were present, the RAG system returned mixed evidence from multiple documents. To focus queries on Sherlock Holmes alone, we reset the Chroma DB and re-ingested only the Sherlock Holmes texts.

Re-ingest only Sherlock Holmes

Reset the vector DB and re-ingest a single book to narrow retrieval scope:

# Reset Chroma DB (example)
python app.py reset-chroma

# Re-ingest the data folder with just Sherlock Holmes text
python hybrid_rag.py ingest --dir data/

Rerunning the narrator question after re-ingestion produced a clearer answer:

(.venv) jeremy@MACSTUDIO BookSearch % python hybrid_rag.py ask --query "Who is the narrator of this document?"

Example improved answer:

=== Answer ===

The narrator appears to be Dr. John Watson, as indicated by references to the narrator as "I" and by passages associating the narrator with being a friend of Sherlock Holmes (see chunk 77).

---- Sources ----
adventuresofsherlockholmes.txt   (chunk 77)
adventuresofsherlockholmes.txt   (chunk 1)
adventuresofsherlockholmes.txt   (chunk 324)

Testing character and setting questions

Some direct fact questions work well (e.g., identifying Scotland Yard inspectors).
Other queries are sensitive to phrasing. Example: the query What is Holmes' address? initially returned “I don’t know.” Rephrasing produced the expected result.

Failing then succeeding queries:

# This initially returned "I don't know."
python hybrid_rag.py ask --query "What is Holmes' address?"

# Rephrased queries that produced the target answer
python hybrid_rag.py ask --query "Who lives on Baker Street?"
python hybrid_rag.py ask --query "Where does Sherlock Holmes live?"

Successful response example:

=== Answer ===

Sherlock Holmes lives on Baker Street, at 221B.

--- Sources ---
adventuresofsherlockholmes.txt  (chunk 63)
adventuresofsherlockholmes.txt  (chunk 703)
adventuresofsherlockholmes.txt  (chunk 445)

Key knobs to tune (what they do and why they matter)

Use these controls to influence retrieval and final LLM prompt composition.

Parameter	Where it’s set	What it controls
`chunk_size` / `overlap`	ingestion (`chunk_text()`)	How much context each retrieved unit contains; affects recall vs. precision.
`k_each`	query-time (`--k-each`)	Number of top candidates to pull from each retriever (vectors and BM25) before merging.
`final_k`	query-time (`--final-k`)	Number of merged candidates included in the prompt sent to the LLM.
Reranking / fusion	code (`rrf_merge` or other)	How to combine rank lists from multiple retrievers (BM25, vector).
Embedding model	ingestion / query	Affects vector similarity quality (e.g., `nomic-embed-text`).
LLM model	query (`--llm`)	The model that consumes the retrieved context and produces the final answer.

Tips for tuning:

Start with small k_each and final_k for fast iteration; increase if you need more candidate evidence.
Adjust chunk_size and overlap per document type (narrative vs. technical).
Always re-ingest after changing chunking parameters.

Reciprocal Rank Fusion (RRf) — merging ranked lists

We used RRf to merge BM25 and vector ranking lists. This is a simple but effective fusion technique:

def rrf_merge(list_a: List[str], list_b: List[str], k: int = 60, topn: int = 5) -> List[str]:
    """Reciprocal Rank Fusion for two ranked lists of IDs."""
    from collections import defaultdict
    scores = defaultdict(float)
    for lst in [list_a, list_b]:
        for rank, _id in enumerate(lst):
            scores[_id] += 1.0 / (k + rank + 1)
    # return topn ids sorted by score (descending)
    return [x for x, _ in sorted(scores.items(), key=lambda kv: kv[1], reverse=True)][:topn]

Chunking implementation

A simple chunker used when ingesting plain text documents:

def chunk_text(text: str, chunk_size: int = 800, overlap: int = 150) -> List[str]:
    """Split text into chunks of approximately chunk_size with an overlap."""
    tokens = text.split()
    chunks = []
    i = 0
    step = chunk_size - overlap
    while i < len(tokens):
        chunk_tokens = tokens[i:i + chunk_size]
        chunks.append(" ".join(chunk_tokens))
        i += step
    return chunks

Remember: if you change chunk_size or overlap, you must re-ingest the corpus for changes to take effect.

CLI options for runtime tuning

The script exposes flags to tune retrieval at query time.

sp = p.add_subparsers(dest="cmd", required=True)

p_ing = sp.add_parser("ingest")
p_ing.add_argument("--dir", required=True, help="Folder or .txt file")
p_ing.add_argument("--embed-model", default="nomic-embed-text")

p_ask = sp.add_parser("ask")
p_ask.add_argument("--query", required=True)
p_ask.add_argument("--llm", default="llama3.3:latest")
p_ask.add_argument("--embed-model", default="nomic-embed-text")
p_ask.add_argument("--k-each", type=int, default=6)
p_ask.add_argument("--final-k", type=int, default=5)

args = p.parse_args()
if args.cmd == "ingest":
    ingest(args.dir, embedding_model=args.embed_model)
else:
    ask(args.query, llm_model=args.llm, embedding_model=args.embed_model,
        k_each=args.k_each, final_k=args.final_k)

Examples of adjusting retrieval budgets:

# Increase retrieval candidate budget and final context budget
python hybrid_rag.py ask --query "What is Holmes' address?" --k-each=10 --final-k=10

Note: Increasing k_each and final_k provides the LLM with more context but does not guarantee a correct answer. Often phrasing or chunking changes are necessary.

Testing methodology — query categories to run

Use a consistent test suite that includes:

Known-answer queries: verify correct retrieval and supporting chunks.
Paraphrase/rephrasing: test semantic robustness.
Out-of-corpus queries: check that the system abstains instead of hallucinating.
Ambiguous/referential questions: ensure the system either clarifies, anchors, or abstains.

Examples that should return abstention (not in corpus):

# Should return "I don't know"
python hybrid_rag.py ask --query "What is Holmes' mother's maiden name?"

# Also should be "I don't know" — out of period invention
python hybrid_rag.py ask --query "Which smartphone did Holmes prefer?"

Abstention response example:

=== Answer ===

I don't know. The provided context does not contain information about Holmes' mother's maiden name.

--- Sources ---
adventuresofsherlockholmes.txt (chunk 228)
adventuresofsherlockholmes.txt (chunk 451)

Dealing with ambiguity and partial support

If the LLM returns partial or uncertain answers:

Add anchors in the query (story title, character name).
Increase k_each and final_k to add more evidence.
Tune chunking at ingestion to ensure relevant facts are grouped.
Use synonyms and paraphrases in tests to probe retrieval sensitivity.

Examples:

# Ambiguous question that may need anchoring and larger budgets
python hybrid_rag.py ask --query "Where does he go afterward?" --k-each=12 --final-k=10

# Anchored query referencing Baker Street:
python hybrid_rag.py ask --query "After leaving Baker Street, where does Watson go next?" --k-each=12 --final-k=10

A focused fact-check example (Irene Adler):

python hybrid_rag.py ask --query "Who does Irene Adler marry?"

Example result:

=== Answer ===

Irene Adler marries Godfrey Norton, an English lawyer.

--- Sources ---
adventuresofsherlockholmes.txt (chunk 67)

Ingestion tuning — practical tip

Chunk size and overlap are applied at ingestion time. To experiment:

# Modify chunk_text parameters to chunk_size=1024, overlap=200 and then:
python hybrid_rag.py ingest --dir data/

Changing chunking parameters requires re-ingesting the data and resetting the vector store (e.g., Chroma). Always backup or version your embeddings/database before resetting.

Testing summary and next steps

Create a test suite with known answers, paraphrases, ambiguous, and out-of-corpus queries.
Use that suite to evaluate whether the system retrieves relevant chunks, abstains appropriately, or hallucinates.
Iterate on ingestion (chunking), retrieval budgets (k_each, final_k), reranking/fusion strategy (e.g., RRf), and prompt templates.
Compare results across embedding models and LLMs to determine the best combination for your corpus and use case.

Testing approach summary:

Create a suite of queries that includes known facts, paraphrases, ambiguous questions, and clearly out-of-corpus questions.
Use these to evaluate whether the system retrieves relevant context, abstains properly, or hallucinates.
Tune ingestion (chunk_size, overlap) and retrieval (k_each, final_k) and compare results.

Final notes and references

RAG systems reduce hallucination risk by grounding answers in retrieved context, but they remain sensitive to retrieval quality, chunking strategy, and prompt engineering.
Use the testing methodology above to find failure modes, then iterate on ingestion, retrieval budgets, reranking, and prompts.
The full example code and repository for this lesson are included with the course materials — reproduce, tune, and extend the pipeline with your own data.

Useful references:

Retrieval-Augmented Generation (RAG) overview
Chroma DB: https://www.trychroma.com/
BM25 / rank_bm25: https://pypi.org/project/rank-bm25/
Reciprocal Rank Fusion (RRf): original literature and implementations

Thanks for following this lesson — experiment with the knobs, and reach out to the course community if you want help troubleshooting specific retrieval failures.

Introduction

Document Processing and Chunking

Keyword Search & Retrieval

Semantic Search & Embeddings

Vector Databases

Building the RAG Pipeline

Demo Testing your RAG System

Quick overview of core code (abbreviated)

Simple interactive test — mixed corpus

Re-ingest only Sherlock Holmes

Testing character and setting questions

Key knobs to tune (what they do and why they matter)

Reciprocal Rank Fusion (RRf) — merging ranked lists

Chunking implementation

CLI options for runtime tuning

Testing methodology — query categories to run

Dealing with ambiguity and partial support

Ingestion tuning — practical tip

Testing summary and next steps

Final notes and references

Watch Video

​Quick overview of core code (abbreviated)

​Simple interactive test — mixed corpus

​Re-ingest only Sherlock Holmes

​Testing character and setting questions

​Key knobs to tune (what they do and why they matter)

​Reciprocal Rank Fusion (RRf) — merging ranked lists

​Chunking implementation

​CLI options for runtime tuning

​Testing methodology — query categories to run

​Dealing with ambiguity and partial support

​Ingestion tuning — practical tip

​Testing summary and next steps

​Final notes and references

Watch Video

Quick overview of core code (abbreviated)

Simple interactive test — mixed corpus

Re-ingest only Sherlock Holmes

Testing character and setting questions

Key knobs to tune (what they do and why they matter)

Reciprocal Rank Fusion (RRf) — merging ranked lists

Chunking implementation

CLI options for runtime tuning

Testing methodology — query categories to run

Dealing with ambiguity and partial support

Ingestion tuning — practical tip

Testing summary and next steps

Final notes and references