How to Build a RAG Application That Actually Works in Production

Most RAG (Retrieval-Augmented Generation) demos work beautifully. Most RAG applications in production fail within 60 days. The gap between demo and production is where engineering discipline matters most.

Why RAG Demos Mislead

A RAG demo typically uses a small, curated dataset with clean formatting and obvious answers. Production data is messy: PDFs with tables, scanned documents, multi-language content, contradictory information, and edge cases that were never anticipated.

The Production RAG Architecture

A production-grade RAG system has five layers, each with specific engineering requirements:

Layer 1: Document Processing

• Parse multiple file formats (PDF, DOCX, HTML, CSV, images with OCR)
• Handle tables, figures, and structured data within documents
• Extract metadata (author, date, source, section)
• Normalize text encoding and formatting

Layer 2: Chunking Strategy

Chunking is the most underappreciated decision in RAG architecture. Choose wrong, and retrieval quality collapses.

# Bad: Fixed-size chunking ignores document structure
chunks = [text[i:i+500] for i in range(0, len(text), 500)]

# Better: Structure-aware chunking
def semantic_chunking(document):
    sections = split_by_headings(document)
    chunks = []
    for section in sections:
        if len(section) > MAX_CHUNK_SIZE:
            sub_chunks = split_by_paragraphs(section)
            chunks.extend(sub_chunks)
        else:
            chunks.append(section)
    return chunks

Key principles:

• Chunk boundaries should respect document structure (sections, paragraphs)
• Include overlap between adjacent chunks (100 to 200 tokens)
• Preserve metadata with each chunk (source document, section, page number)

Layer 3: Embedding and Indexing

• Choose an embedding model appropriate for your domain
• Test embedding quality with your actual data, not benchmark datasets
• Index with a vector database that supports filtering on metadata
• Re-index on a scheduled cadence as source data updates

Layer 4: Retrieval and Reranking

• Retrieve more candidates than you need (top 20) and rerank to top 5
• Use hybrid search: combine vector similarity with keyword matching
• Filter by metadata before ranking (date ranges, document types, access permissions)
• Implement a minimum relevance threshold; do not force irrelevant context

Layer 5: Generation and Grounding

• Include explicit instructions to only answer from provided context
• Ask the model to cite specific sources for each claim
• Implement hallucination detection on the output
• Set a confidence threshold below which the system says "I don't know"

Common Failure Modes

Failure 1: Retrieval misses the right chunk Cause: Poor chunking, wrong embedding model, or inadequate retrieval strategy. Fix: Evaluate retrieval separately from generation. Measure recall at top 5 and top 10.

Failure 2: The right chunk is retrieved but the model ignores it Cause: Too much context overwhelming the prompt, or the answer requires synthesis across multiple chunks. Fix: Reduce context window size, prioritize chunks, add explicit citation instructions.

Failure 3: The model hallucinates despite having correct context Cause: Model tendency to generate plausible-sounding text rather than strictly quoting sources. Fix: Use grounding prompts, implement output verification against retrieved sources.

Failure 4: Stale information served as current Cause: Source documents updated but embeddings not re-indexed. Fix: Implement a re-indexing pipeline triggered by document updates.

Evaluation Framework

Build an automated evaluation pipeline:

1. Retrieval evaluation: Does the system retrieve the correct documents for each query?
2. Answer quality: Is the generated answer factually correct and complete?
3. Faithfulness: Does the answer stay grounded in the retrieved context?
4. Relevance: Is the answer addressing the actual user intent?

Maintain a golden dataset of 100+ queries with known correct answers and source documents. Run this evaluation weekly.

Conclusion

Production RAG requires engineering discipline at every layer. The most common mistake is treating RAG as a simple "embed, retrieve, generate" pipeline. In reality, each layer needs careful tuning, monitoring, and maintenance.

Start with structure-aware chunking and hybrid retrieval. Add hallucination detection and automated evaluation early. And assume that your first architecture will need significant iteration once real users and real data hit the system.