an experiment

Chunk Lab

Same document. Same question. Different chunking — different answer.

Real embeddings (Voyage), real similarity search, real LLM answer (Claude). Switch the strategy to see retrieval change.

Pick a document

Retrieval-Augmented Generation (RAG) combines large language models with external knowledge bases. Unlike traditional LLMs that rely solely on training data, RAG systems retrieve relevant documents at query time. This approach reduces hallucinations and allows models to access up-to-date information.

The core architecture has three components: a retriever, a knowledge store, and a generator. The retriever encodes user queries into vector representations. The knowledge store contains pre-embedded documents. The generator produces responses conditioned on retrieved context.

Vector databases play a critical role in RAG systems. They store document embeddings and support fast similarity search. Common implementations include Pinecone, Weaviate, ChromaDB, and Qdrant. Each makes different tradeoffs between latency, scalability, and cost.

Chunking strategy significantly impacts retrieval quality. Documents must be split into segments before embedding. Fixed-size chunking is simplest but ignores semantic boundaries. Recursive chunking respects paragraph and sentence structure. Semantic chunking groups related ideas together using model-based methods.

Evaluation of RAG systems remains challenging. Standard metrics include retrieval precision and answer relevance. However, end-to-end evaluation often requires human judgment. Automated evaluation using LLMs as judges has emerged as a practical compromise.

Production RAG systems typically include reranking. After initial retrieval, a more expensive cross-encoder rescores the top results. This two-stage pipeline balances speed and accuracy. Without reranking, retrieval quality often plateaus regardless of embedding quality.

Chunk Lab

Pick a document

Pick a chunking strategy

Ask a question