RAG-Based Procurement Chatbot: Building Accurate Financial Document Search with Vector Embeddings

The Problem

An organization needed instant answers to complex questions about procurement policies, procedures, and financial documents. Manual document search was slow, and traditional keyword-based search failed to understand context and semantic relationships.

The challenge:

• Users asking questions in natural language about dense financial documents
• Need for accurate context retrieval to avoid hallucination
• Support for multiple LLM backends (Claude via AWS Bedrock and local Ollama)
• Conversational context to handle follow-up questions
• Real-time indexing as new documents are added

The goal: build a RAG (Retrieval-Augmented Generation) chatbot that understands questions, retrieves relevant document chunks, and generates accurate answers grounded in source material.

Architecture Overview

The system follows a classic RAG pipeline with enhancements for accuracy and performance:

User Query
    ↓
[Hybrid Retriever] ← vectors + BM25 ranking
    ↓
[Reranking] ← FlashRank for precision
    ↓
[LLM with Context] ← Claude or Ollama
    ↓
Answer with source citations

Key components:

• Indexing: PDF ingestion, chunking, and vector embedding
• Retrieval: Hybrid search combining vector similarity and BM25 ranking
• Reranking: Context-aware reranking to surface only the most relevant chunks
• Generation: Conversational LLM chain with memory and glossary expansion
• Deployment: Streamlit UI with real-time document sync

Tech Stack

The Journey

Stage 1 — Document Ingestion & Smart Chunking

Naively chunking PDFs at fixed sizes loses context and creates fragments that are hard to answer questions about.

I implemented:

• RecursiveCharacterTextSplitter — Chunks recursively (sentences → paragraphs → fixed size), preserving semantic boundaries
• Configurable chunk size/overlap — Default 1000 chars with 200-char overlap to maintain context across chunks
• Metadata preservation — Each chunk retains source file, page number, and section info for traceability
• Table extraction — img2table + Tesseract/Textract OCR for structured data, preserving tables as markdown
• Incremental indexing — Only re-chunk new documents; track processed files in chunked_docs.pkl

One challenge: oversized chunks (tables, lists) that exceeded embedding token limits. Solution: split tokens exceeding the limit via token_exceed_split(), then merge sub-chunks with overlap.

Stage 2 — Hybrid Retrieval: Vectors + BM25

Vector similarity alone can miss exact terminology matches. Keyword search alone lacks semantic understanding. The fix is hybrid search.

Implemented a HybridRetriever class combining:

• Vector search — FAISS similarity search returns top-k semantically similar chunks
• BM25 ranking — Preprocessed texts (stopword removal, tokenization) ranked by TF-IDF
• Fusion — Reciprocal rank fusion (RRF) combines both rankings: score = 1 / (k + vector_rank) + 1 / (k + bm25_rank)
• Deduplication — Merges duplicate results from both methods
• Top-N filtering — Returns only top-k after fusion to reduce noise

Example: query "What is the maximum budget for procurement?"

• Vector search finds semantically similar chunks about budget policies
• BM25 finds exact keyword matches in procedures
• RRF fusion prioritizes chunks ranking high in both methods

Stage 3 — Context-Aware Reranking

Retrieving top-k chunks is good, but the LLM still sees noise. The hybrid retriever returns 20-30 candidates; the reranker narrows it to the most relevant 5.

Added ContextualCompressionRetriever with FlashRank:

• Input — User query + retrieved chunks
• Ranking — FlashRank scores each chunk's relevance to the query (0-1 confidence)
• Filtering — Keep only chunks with score > threshold (configurable, default 0.5)
• Output — Top 5 most relevant chunks guaranteed

This step dramatically reduced hallucination by ensuring the LLM only sees truly relevant context.

Stage 4 — Glossary Expansion & Abbreviation Handling

Financial documents use many abbreviations (LOA, RICA, SMP, etc.) that confuse embeddings.

I added glossary-aware preprocessing:

• CSV glossary — Map abbreviations to full definitions
• Regex compilation — Build a single compiled pattern for all abbreviations (word boundaries, case-insensitive)
• Query expansion — When user types "LOA", expand it to "LOA (Letter of Authority)" before embedding
• Context injection — Abbreviations are explained in the prompt context, so Claude can resolve ambiguities

Example: user asks "What does LOA mean?" → glossary substitutes "What does LOA (Letter of Authority) mean?" → embedding captures semantics better.

Stage 5 — Conversational Memory & Multi-Turn Context

A RAG chatbot must handle follow-up questions. "What's the budget?" followed by "Can you break it down by category?" needs conversation memory.

Implemented with LangGraph:

• Message buffer — Store all user + assistant messages in session state
• Deduplication — Avoid replaying the same message twice
• Trimming strategy — Keep last 3 QA pairs (6 messages + system prompt = 13 messages max)
• LangGraph integration — MemorySaver checkpoints conversation state
• System prompt — Injected as SystemMessage, providing consistent persona and constraints

Example flow:

1. User: "What's our procurement budget?"
2. Assistant: "Your budget is $X million" (memory stores this)
3. User: "Break it down by category"
4. Assistant: Uses prior message to contextualize breakdown, doesn't repeat "Your budget is $X million"

Stage 6 — Dual LLM Support: Bedrock & Ollama

Supporting both Claude (via AWS Bedrock) and Ollama enabled:

• Production use with Bedrock's structured output and reliability
• Local testing with Ollama for rapid iteration (no API costs, no latency)

Architecture:

if api_type == 'bedrock':
    llm = BedrockChat(
        model_id="anthropic-claude-...",
        client=client,
        streaming=True
    )
elif api_type == 'ollama':
    llm = OllamaChat(
        model_name=llm_model,
        streaming=True
    )

Both support streaming, which displays incremental responses to users in real-time.

Stage 7 — Live Document Sync & Rebuild

Documents change frequently. The system needed to:

1. Auto-detect new files in the PDF directory
2. Update vector store when new documents arrive
3. Remove orphaned documents if a file is deleted
4. Preserve existing vectors to avoid rebuild on every startup

Implemented incremental indexing:

current_files = set([f for f in os.listdir(PDF_PATH) if f.endswith('.pdf')])
chunked_files = set([doc.metadata.get('filename') for doc in docs])
 
missing_files = current_files - chunked_files  # New files to process
orphaned_files = chunked_files - current_files  # Deleted files to remove
 
# Process only missing files
if missing_files:
    new_docs = data_ingestion(..., specific_files=list(missing_files), append_mode=True)
    docs.extend(new_docs)
 
# Remove orphaned docs
if orphaned_files:
    docs = [doc for doc in docs if doc.metadata.get('filename') not in orphaned_files]
 
# Save and rebuild vector store only if needed
if needs_update:
    rebuild_vector_store(docs)

Benefit: A 10-file system can add a new document and re-index in seconds, not minutes.

Stage 8 — Caching & Performance Optimization

Repeated queries with identical parameters don't need re-fetching. Added two layers of caching:

• LRU cache for similarity search — @lru_cache(maxsize=100) on the vector store query function
• Document cache in retriever — _doc_cache stores already-retrieved documents to avoid duplicate processing

Result: Second and subsequent calls to the same query are ~10x faster.

Why This Approach Worked

1. Hybrid search reduces hallucination — By combining vector + BM25, the system finds both semantic matches and exact keyword hits, reducing the chance of LLM inventing answers
2. Reranking adds precision — FlashRank ensures only top-relevance chunks reach the LLM, cutting noise
3. Glossary expansion bridges embeddings and domain language — Abbreviations are a weak point for embeddings; explicit expansion fixes this
4. Incremental indexing scales — New documents don't trigger full rebuild; vector store stays efficient
5. Dual LLM support enables fast iteration — Ollama for local dev, Bedrock for production
6. Conversational memory feels natural — Users can ask follow-ups without repeating context

What Could Work Better

1. Adaptive Chunking

Current: Fixed chunk size (1000 chars) with overlap.

Better: Chunk based on semantic boundaries detected via embedding similarity. If two sentences are semantically distant, they shouldn't be in the same chunk. This would preserve context while reducing noise.

How: Use sentence transformers to detect breaks in semantic continuity, then chunk at those boundaries.

2. Sub-Graph Retrieval (Knowledge Graphs)

Current: Flat document chunks with BM25 + vectors.

Better: Extract entities (vendor names, budget lines, approval steps) and build a knowledge graph. Then retrieve by traversing semantic relationships.

Example: "Who approves budgets over $100k?" → entity query for approval rules + traversal to budget thresholds.

How: Use Claude to extract entities during ingestion, build a graph structure (Neo4j or similar), then retrieve by graph traversal + vector hybrid search.

3. Multi-Query Expansion

Current: User's single query → vector + BM25 search.

Better: Expand a single query into 3-5 semantically equivalent phrasings (via LLM), search each, then deduplicate and fuse results. This catches documents using different terminology.

Example: "What's the procurement budget?" expands to:

• "What is the total procurement budget?"
• "How much money do we allocate for procurement?"
• "Procurement budget allocation amount"

Then search all three and combine results.

4. Streaming Reranking

Current: Retrieve all chunks, rerank all, then pass to LLM.

Better: Rerank on-the-fly as chunks come in from vector search. Stop reranking early (after top-5 high-confidence matches) to reduce latency.

Benefit: Answers appear faster; user sees streaming responses sooner.

Performance Optimization Tips

1. Batch Embedding

Current: Embed chunks one-by-one during ingestion.

Better: Batch embed 32 or 64 chunks at once. The Titan embed API supports batching; chunking requests reduces API overhead.

# Before: 1000 API calls for 1000 chunks
embeddings = [embed_fn(chunk) for chunk in chunks]
 
# After: ~32 API calls for 1000 chunks (32 chunks per batch)
embeddings = []
for i in range(0, len(chunks), batch_size=32):
    batch = chunks[i:i+batch_size]
    batch_embeddings = embed_fn(batch)  # Single API call
    embeddings.extend(batch_embeddings)

Speedup: 10-20x faster embedding for large document sets.

2. Approximate Nearest Neighbor Search

Current: FAISS with exact L2 distance.

Better: Use FAISS IVF (Inverted File) index for approximate search. Trade tiny accuracy loss for massive speed gains.

# Before: Exact search (slow for 100k+ vectors)
vector_store = FAISS.from_documents(docs, embeddings)
 
# After: Approximate search with IVF
quantizer = faiss.IndexFlatL2(d)  # Dimension of embeddings
index = faiss.IndexIVFFlat(quantizer, d, nlist=100)
index.train(np.array([d.embedding for d in docs]))
index.add(np.array([d.embedding for d in docs]))

Speedup: 100x faster search; 99% accuracy retention for top-5 results.

3. Prefix Caching for LLM Calls

Current: Every LLM call includes full system prompt + context.

Better: Use Claude's prompt caching (available in recent API versions). Repeated system prompts and glossary sections are cached; only new chunks are charged full rate.

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": system_prompt + glossary,
                "cache_control": {"type": "ephemeral"}  # Cache this
            },
            {
                "type": "text",
                "text": retrieved_context  # Only this varies per call
            },
            {
                "type": "text",
                "text": user_query
            }
        ]
    }
]

Savings: 90% reduction in token costs for repeated patterns.

4. Parallel Chunk Retrieval

Current: Retrieve chunks sequentially (vector search, then BM25, then rerank).

Better: Fetch vectors and BM25 in parallel using ThreadPoolExecutor.

with ThreadPoolExecutor(max_workers=2) as executor:
    vector_future = executor.submit(vectorstore.similarity_search, query, k)
    bm25_future = executor.submit(bm25_retriever.retrieve, query)
    
    vector_results = vector_future.result()
    bm25_results = bm25_future.result()
    
    fused_results = rrf_fusion(vector_results, bm25_results)

Speedup: 1.5x faster retrieval (retrieval time dominated by I/O, not CPU).

5. Lazy Loading with Streaming

Current: Load all chunks into memory upfront.

Better: Stream chunks from FAISS on-demand. Only load top-k matches into memory.

Benefit: Scales to 10M+ vectors without OOM (out-of-memory) errors.

6. Quantized Embeddings

Current: Full-precision embeddings (768 dimensions × 4 bytes float32).

Better: Quantize embeddings to int8 (768 dims × 1 byte). Reduces memory by 4x, speeds up search.

embeddings_fp32 = embedding_model.encode(texts)  # shape: (n, 768), dtype float32
embeddings_int8 = np.round(embeddings_fp32 * 127).astype(np.int8)  # quantized

Trade-off: 1-2% accuracy loss; 4x memory savings.

Deployment Considerations

Current Setup:

• Streamlit app running on EC2 / local server
• FAISS index and documents cached locally
• Config-driven (TOML for settings, .env for secrets)

For Production Scale:

1. Vector Store — Migrate from FAISS to Pinecone / Weaviate for managed scaling
2. LLM — Keep Bedrock; add retry logic and exponential backoff
3. Caching — Redis for distributed caching (embeddings, reranking scores)
4. Monitoring — Log retrieval quality (did reranked chunks contain the answer?), LLM latency, embedding staleness
5. A/B Testing — Compare retrieval strategies (BM25 only vs. hybrid vs. knowledge graph) on real queries

Summary

The RAG-Based Procurement Chatbot demonstrates that RAG quality hinges on retrieval precision, not just LLM power. By combining:

• Hybrid search (vectors + BM25)
• Context-aware reranking
• Glossary expansion
• Incremental indexing
• Conversational memory

The system delivers accurate, grounded answers to complex document questions. The next frontier is knowledge graphs and multi-query expansion for even higher precision.

Key Takeaway: RAG is only as good as the chunks you feed it. Invest in retrieval quality, and the LLM will excel.