Effective Chunking Strategy for RAG Systems

​

Common chunking strategies

• Splitting text by fixed character count
• Splitting text by fixed token count
• Splitting text by semantic units with overlap (recommended)
• Random chunking to increase diversity

​

Recommended approach: semantic units with overlap

​

Why other strategies fall short

​

Additional notes on token-based splitting

• Token-aware splitting is preferable to raw character counts because embeddings and LLMs operate on tokens. However, tokenization must match the embedding/LLM tokenizer to avoid mismatches.
• Use the embedding model’s tokenizer (for example, tiktoken) to compute token lengths before embedding or truncation.

​

Practical recommendations

• Chunk unit
  • Prefer semantic boundaries: sentences, paragraphs, or logical sections.
  • Preserve special content (code blocks, tables, lists) as single chunks when possible to keep structure intact.
• Chunk size
  • Tune to your embedding model and LLM context window. A common target is a few hundred tokens per chunk (e.g., 200–500 tokens), adjusted for your model and document types.
• Overlap
  • Use overlap to cover boundary-spanning concepts. Typical overlap options:
    • Sentence-based: 1–3 sentences
    • Proportional: 10–20% of the chunk
    • Token-based: 50–100 tokens as a pragmatic starting point
• Token accounting
  • Use the embedding model’s tokenizer to measure tokens. Example: tiktoken on GitHub.
• Metadata
  • Store source identifiers, position offsets, and structural metadata so retrieved chunks can be traced and reassembled.
• Deduplication and normalization
  • Remove near-duplicates to avoid redundant retrievals, but keep intended overlapping chunks since overlaps are deliberate for context continuity.

​

Quick reference: chunking guidelines by document type

​

Example: sentence-based chunking with overlap (Python)

from typing import List

def chunk_sentences(sentences: List[str], chunk_size: int, overlap: int) -> List[str]:
    """
    Split a list of sentences into overlapping chunks.
    - chunk_size and overlap are in number of sentences.
    - Returns a list of chunk strings where sentences are joined with a space.
    """
    if chunk_size <= 0:
        raise ValueError("chunk_size must be > 0")
    if overlap < 0:
        raise ValueError("overlap must be >= 0")

    chunks: List[str] = []
    i = 0
    step = max(1, chunk_size - overlap)
    while i < len(sentences):
        chunk = sentences[i : i + chunk_size]
        chunks.append(" ".join(chunk))
        if i + chunk_size >= len(sentences):
            break
        i += step
    return chunks

# Example usage:
sentences = [
    "First sentence.", "Second sentence.", "Third sentence.",
    "Fourth sentence.", "Fifth sentence."
]
chunks = chunk_sentences(sentences, chunk_size=2, overlap=1)
# chunks -> [
#  "First sentence. Second sentence.",
#  "Second sentence. Third sentence.",
#  "Third sentence. Fourth sentence.",
#  "Fourth sentence. Fifth sentence."
# ]

​

When to consider alternatives

• Very short documents: Semantic chunking still applies; overlap may be unnecessary.
• Extremely long documents: Use hierarchical chunking—first split into sections, then paragraphs with overlap—to preserve high-level structure.
• Real-time or streaming ingestion: Use sliding windows or rolling buffers. Aim to respect semantic boundaries (e.g., sentence/paragraph fences) when possible to retain coherence.

​

Validation and tuning

• Measure token counts with the embedding tokenizer to ensure chunks fit model limits.
• Sample retrieval results and run relevance/recall evaluations to confirm that chosen chunk size and overlap improve retrieval quality.
• Iterate: different corpora (legal text, scientific papers, code) will require different heuristics.

​

References and further reading

• tiktoken tokenizer: https://github.com/openai/tiktoken
• Retrieval-Augmented Generation concepts: search for RAG architectures and embedding-based retrieval papers and blog posts for deeper guidance.