Why RAG

Large language models are impressive right up until you ask them something they were never trained on. They have a knowledge cutoff, know nothing about your private documents, and will answer confidently even when they are wrong. Retrieval-Augmented Generation fixes this by handing the model relevant context at query time instead of hoping it memorized the right facts during training.

The idea is straightforward. Instead of fine-tuning a model on your documents, which is slow, expensive, and needs to be redone every time the data changes, you retrieve the most relevant chunks at runtime and drop them into the prompt. The model answers from what you give it, not from what it recalls.

I built my first real RAG system for DocuLume, a document assistant that lets you upload PDFs and ask questions about them, with answers cited back to the exact page. Here is what actually happened building it, not the sanitized version.

The shape of it

RAG has two phases. Indexing: load documents, split into chunks, embed each chunk, store the vectors somewhere searchable. Retrieval and generation: embed the incoming question, pull the closest chunks, hand them to the model as context.

User Query
    |
    v
[Embed Query] --> [Vector Store] --> [Top-k Chunks]
                                           |
                                           v
                                  [LLM + Context] --> Answer

That diagram is the easy part. Almost everything that mattered turned out to be in the details underneath it.

Chunking, and why my first attempt was wrong

You cannot feed a two hundred page PDF into a prompt. It has to be split, and only the relevant pieces sent through.

from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = PyPDFLoader("document.pdf")
pages = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_documents(pages)

My first version used a chunk size of 2000, because bigger felt like it should mean more context and better answers. It did the opposite. Big chunks bury the actually relevant sentence in a wall of surrounding text, and the embedding ends up representing the average of everything in the chunk rather than anything precise. Dropping to 800 to 1000 characters with 200 characters of overlap was the single change that improved retrieval the most, more than anything I did later to the model or the prompt.

The overlap matters too. Without it, a sentence that happens to fall across a chunk boundary gets cut in half, and neither half retrieves well on its own.

Embeddings without an API key

I wanted DocuLume to run without depending on a paid embeddings API, partly for cost and partly because I did not want an outage somewhere else to take down document search. I used a local sentence transformer model instead:

from langchain.embeddings import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
)

It is small enough to run on CPU without noticeable lag for single documents, and good enough for general document QA. The one rule that actually matters here: whatever model embeds your documents at indexing time has to be the exact same model used at query time. Swap models between the two and retrieval silently breaks, because the two sets of vectors no longer live in comparable space. It does not error, it just returns nonsense with total confidence, which took me longer to debug than I would like to admit.

ChromaDB, and the sqlite surprise

For the vector store I went with Chroma. It is embedded, persists to disk as a set of files next to the project, and does not need a separate service running, which fit a solo project much better than standing up Pinecone or a hosted vector database for something with a handful of users.

import chromadb
from langchain.vectorstores import Chroma

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",
)
vectorstore.persist()

The one thing worth knowing before you hit it yourself: Chroma needs a newer SQLite than what ships with the Python on a lot of base Docker images and some Linux distros. The first deploy failed with a version error that had nothing obviously to do with my code. The fix is a couple of lines swapping in pysqlite3-binary before Chroma imports its own sqlite3, and then it is fine, but it is exactly the kind of environment quirk that eats an afternoon if you do not already know it is coming.

Wiring retrieval to the model

from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    return_source_documents=True,
)

result = qa_chain({"query": "What are the main findings?"})
print(result["result"])
print(result["source_documents"])

k=4 controls how many chunks get retrieved. More chunks means more context and generally better answers, but also more tokens and a higher chance of the model getting distracted by something only loosely relevant. Four to six worked best across the documents I tested with, which were mostly technical PDFs in the twenty to eighty page range. Your mileage will vary with document type.

What broke, and what fixed it

No memory across turns. The first version treated every question in isolation. Ask "who wrote this," then "what did they conclude," and the second question had nothing to connect it to the first. ConversationalRetrievalChain with a memory buffer fixed this:

from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory

memory = ConversationBufferMemory(
    memory_key="chat_history",
    return_messages=True,
)

chain = ConversationalRetrievalChain.from_llm(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    memory=memory,
)

Confident wrong answers. Sometimes nothing relevant gets retrieved, and instead of saying so, the model fills the gap with something plausible sounding. A blunt instruction in the system prompt fixed most of it:

system_prompt = """You are a helpful assistant answering questions about a document.
Use ONLY the provided context to answer. If the context does not contain enough
information to answer the question, say "I don't have enough information in the
document to answer that." Do not make up information."""

It is a small change and it made a bigger difference than most of the retrieval tuning combined.

Messy source text. Raw PDF extraction drags along headers, footers, and page numbers into the chunks. A pass to strip them before splitting cleaned up retrieval noticeably:

import re

def clean_text(text: str) -> str:
    text = re.sub(r"\n{3,}", "\n\n", text)
    text = re.sub(r"Page \d+ of \d+", "", text)
    return text.strip()

Where it stands

DocuLume handles multi-turn questions over PDFs reliably now, and the citations back to page numbers make it easy to check whether an answer is actually grounded. Where it still struggles is questions that require pulling together information scattered across widely separated sections of a document, since each chunk is judged for relevance mostly on its own. That is a known limitation of chunk-based retrieval, and techniques like re-ranking or hypothetical document embeddings help but do not fully solve it.

What I would tell myself before starting: the LLM and the prompt are the least interesting part of the whole system. Chunking strategy, embedding consistency, and cleaning the source text upfront decide whether retrieval works at all, and no amount of prompt engineering on top fixes a bad retrieval layer underneath it.