RAG-Based Procurement Chatbot: Building Accurate Financial Document Search with Vector Embeddings

The Problem

An organization needed instant answers to complex questions about procurement policies, procedures, and financial documents. Manual document search was slow, and plain keyword search kept missing the point — it has no idea that "budget cap" and "spending limit" mean roughly the same thing.

The requirements stacked up quickly: users wanted to ask questions in plain language about dense financial documents, answers needed to be grounded well enough that the model wasn't just making things up, multiple LLM backends had to work (Claude via AWS Bedrock for production, local Ollama for testing), follow-up questions needed to actually carry context forward, and the index had to stay current as new documents landed.

So the goal was a RAG chatbot — retrieval-augmented generation — that understands the question, pulls the right document chunks, and answers from what's actually in those chunks rather than from vibes.

Architecture Overview

The system follows a fairly standard RAG pipeline, with a few extra steps bolted on for accuracy and speed:

User Query
    ↓
[Hybrid Retriever] ← vectors + BM25 ranking
    ↓
[Reranking] ← FlashRank for precision
    ↓
[LLM with Context] ← Claude or Ollama
    ↓
Answer with source citations

The pieces that make it up: indexing (PDF ingestion, chunking, vector embedding), retrieval (hybrid search combining vector similarity and BM25), reranking (context-aware filtering down to the chunks that actually matter), generation (a conversational LLM chain with memory and glossary expansion), and deployment (a Streamlit UI that syncs with documents in real time).

Tech Stack

The Journey

Chunking documents without losing what makes them useful

Naive fixed-size chunking loses context and produces fragments that are genuinely hard to answer questions about — cut a paragraph in half and you've got two pieces that individually mean nothing. I used RecursiveCharacterTextSplitter instead, which chunks recursively (sentences, then paragraphs, then fixed size as a last resort) so it preserves semantic boundaries rather than slicing blindly. The default is 1000 characters with a 200-character overlap, which keeps context bridging across chunk edges. Every chunk keeps its source file, page number, and section info for traceability, tables get extracted via img2table plus Tesseract/Textract OCR and preserved as markdown, and indexing is incremental — only new documents get re-chunked, with processed files tracked in chunked_docs.pkl.

The one snag: oversized chunks, usually tables or long lists, blew past the embedding model's token limit. token_exceed_split() splits anything over the limit and merges the sub-chunks back together with overlap so nothing gets orphaned.

Combining vectors and keyword search

Vector similarity alone misses exact terminology matches — ask about a specific clause number and pure semantic search might wander right past it. Keyword search alone has the opposite problem: no sense of meaning at all. Hybrid search fixes both at once.

I built a HybridRetriever class that runs FAISS similarity search for the top-k semantically similar chunks alongside BM25 ranking on preprocessed text (stopwords stripped, tokenized, scored by TF-IDF), then fuses the two rankings with reciprocal rank fusion: score = 1 / (k + vector_rank) + 1 / (k + bm25_rank). Duplicate results across both methods get merged, and only the top-k survives the fusion step.

Take the query "What is the maximum budget for procurement?" Vector search finds chunks that are semantically about budget policy even if they don't use those exact words. BM25 catches the literal keyword matches in the procedures. RRF fusion then favors whatever ranks well in both — which tends to be exactly the chunk you want.

Filtering out noise before it reaches the LLM

Getting the top-k chunks back is only half the battle — the LLM still ends up staring at a pile of noise if you stop there. The hybrid retriever returns 20 to 30 candidates, and the reranker's job is to cut that down to the 5 that actually matter.

ContextualCompressionRetriever with FlashRank does this: it takes the user query plus the retrieved chunks, scores each chunk's relevance on a 0–1 scale, keeps only what clears a configurable threshold (0.5 by default), and guarantees the top 5 relevant chunks come out the other end. This step alone cut hallucination dramatically, simply because the LLM stopped seeing irrelevant context it might otherwise try to make sense of.

Teaching the system to understand abbreviations

Financial documents are thick with abbreviations — LOA, RICA, SMP — and embeddings tend to choke on them; there's no semantic signal in three random capital letters. So I added a glossary-aware preprocessing step: a CSV maps each abbreviation to its full definition, a single compiled regex pattern (word boundaries, case-insensitive) catches all of them at once, and when a user types "LOA" the query gets expanded to "LOA (Letter of Authority)" before it's ever embedded. The abbreviations also get explained directly in the prompt context, so Claude has what it needs to resolve ambiguity on its own.

Ask "What does LOA mean?" and the glossary quietly rewrites it to "What does LOA (Letter of Authority) mean?" before embedding — which gives the embedding model something real to work with instead of three letters it's never seen mean anything.

Making follow-up questions actually work

Any RAG chatbot worth using has to handle follow-ups. "What's the budget?" followed by "Can you break it down by category?" only makes sense if the second question remembers the first.

I built this on LangGraph: a message buffer stores every user and assistant turn in session state, deduplication stops the same message from getting replayed, and a trimming strategy keeps just the last 3 QA pairs — 6 messages plus the system prompt, 13 messages max — so the context window doesn't grow without bound. LangGraph's MemorySaver checkpoints the conversation state, and the system prompt gets injected as a SystemMessage so persona and constraints stay consistent turn after turn.

In practice: the user asks "What's our procurement budget?", the assistant answers "$X million" and that gets stored in memory. Then the user asks "Break it down by category," and the assistant uses the stored context to answer the follow-up directly — it doesn't need to repeat the first answer to make sense of the second.

Supporting both Bedrock and Ollama

Running both Claude via AWS Bedrock and Ollama locally solved two different problems: Bedrock gave production the structured output and reliability it needed, and Ollama gave local development fast iteration without API costs or latency getting in the way.

Here's the architecture:

if api_type == 'bedrock':
    llm = BedrockChat(
        model_id="anthropic-claude-...",
        client=client,
        streaming=True
    )
elif api_type == 'ollama':
    llm = OllamaChat(
        model_name=llm_model,
        streaming=True
    )

Both backends support streaming, so responses show up incrementally instead of all at once.

Keeping the index in sync as documents change

Documents in this system don't sit still — new ones show up, old ones get deleted, and the vector store needs to keep pace without rebuilding from scratch every time. That meant auto-detecting new files in the PDF directory, updating the vector store only for what's new, removing anything orphaned by a deleted file, and leaving existing vectors alone whenever nothing had actually changed.

Here's the incremental indexing logic that handles it:

current_files = set([f for f in os.listdir(PDF_PATH) if f.endswith('.pdf')])
chunked_files = set([doc.metadata.get('filename') for doc in docs])
 
missing_files = current_files - chunked_files  # New files to process
orphaned_files = chunked_files - current_files  # Deleted files to remove
 
# Process only missing files
if missing_files:
    new_docs = data_ingestion(..., specific_files=list(missing_files), append_mode=True)
    docs.extend(new_docs)
 
# Remove orphaned docs
if orphaned_files:
    docs = [doc for doc in docs if doc.metadata.get('filename') not in orphaned_files]
 
# Save and rebuild vector store only if needed
if needs_update:
    rebuild_vector_store(docs)

The practical effect: a small system with 10 files can add a new document and be re-indexed in seconds instead of minutes.

Caching what's already been computed

A query that's already been answered doesn't need to be re-fetched from scratch. I added two layers of caching on top of the retrieval path: an LRU cache (@lru_cache(maxsize=100)) on the vector store query function, and a document cache in the retriever itself (_doc_cache) that keeps already-retrieved documents around to avoid reprocessing them. Together they make the second and later calls to the same query roughly 10x faster than the first.

Why This Approach Worked

A few decisions ended up mattering more than the rest. Hybrid search cut down hallucination by catching both semantic matches and exact keyword hits, so the model had less reason to invent an answer when retrieval came up short. Reranking with FlashRank meant only the genuinely relevant chunks reached the LLM in the first place. Glossary expansion closed the gap between how embeddings work and how domain abbreviations actually get used — a weak spot that's easy to overlook until you hit it. Incremental indexing kept the vector store efficient as documents piled up, since new files never triggered a full rebuild. Running both Ollama and Bedrock made iteration fast without sacrificing production reliability. And conversational memory made the whole thing feel less like a search box and more like a conversation — users could ask follow-ups without re-explaining themselves.

What Could Work Better

Adaptive chunking. Right now chunks are a fixed 1000 characters with overlap, which is simple but arbitrary — it has no idea where one idea actually ends and another begins. Chunking at semantic boundaries instead (using sentence transformers to detect where meaning shifts, then cutting there) would keep related sentences together and unrelated ones apart, instead of splitting purely on character count.

Knowledge graph retrieval. The system currently treats documents as flat chunks scored by BM25 and vectors. Extracting entities during ingestion — vendor names, budget lines, approval steps — and building an actual graph out of them (Claude for extraction, something like Neo4j to hold the structure) would let a question like "Who approves budgets over $100k?" get answered by traversing approval rules to budget thresholds, rather than hoping the right chunk happens to mention both.

Multi-query expansion. A single query goes straight to vector and BM25 search today, but that misses documents phrased differently than the question. Expanding "What's the procurement budget?" into a few semantically equivalent phrasings first — "What is the total procurement budget?", "How much money do we allocate for procurement?", "Procurement budget allocation amount" — then searching all of them and fusing the results, would catch documents that never use the querier's exact words.

Streaming reranking. Right now the pipeline retrieves everything, reranks everything, and only then hands it to the LLM — a strictly sequential path. Reranking on the fly as chunks arrive from vector search, and stopping early once 5 high-confidence matches show up, would shave off latency and let streaming responses start sooner.

Performance Optimization Tips

Batch embedding. Embedding chunks one at a time during ingestion means one API call per chunk — 1000 chunks, 1000 calls. The Titan embed API supports batching, so grouping 32 or 64 chunks per request cuts that down to roughly 32 calls for the same 1000 chunks:

# Before: 1000 API calls for 1000 chunks
embeddings = [embed_fn(chunk) for chunk in chunks]
 
# After: ~32 API calls for 1000 chunks (32 chunks per batch)
embeddings = []
for i in range(0, len(chunks), batch_size=32):
    batch = chunks[i:i+batch_size]
    batch_embeddings = embed_fn(batch)  # Single API call
    embeddings.extend(batch_embeddings)

That's roughly 10-20x faster embedding for large document sets.

Approximate nearest neighbor search. FAISS with exact L2 distance is precise but slow once you're past 100k vectors. Switching to a FAISS IVF (inverted file) index trades a sliver of accuracy for a large speed gain:

# Before: Exact search (slow for 100k+ vectors)
vector_store = FAISS.from_documents(docs, embeddings)
 
# After: Approximate search with IVF
quantizer = faiss.IndexFlatL2(d)  # Dimension of embeddings
index = faiss.IndexIVFFlat(quantizer, d, nlist=100)
index.train(np.array([d.embedding for d in docs]))
index.add(np.array([d.embedding for d in docs]))

That's about 100x faster search while keeping 99% accuracy on top-5 results — a trade worth making for most retrieval workloads.

Prompt caching for LLM calls. Every LLM call right now sends the full system prompt plus context from scratch. Claude's prompt caching changes that: repeated system prompts and glossary sections get cached, and only the chunks that actually vary per call are charged at full rate.

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": system_prompt + glossary,
                "cache_control": {"type": "ephemeral"}  # Cache this
            },
            {
                "type": "text",
                "text": retrieved_context  # Only this varies per call
            },
            {
                "type": "text",
                "text": user_query
            }
        ]
    }
]

For repeated patterns, that's roughly a 90% reduction in token costs.

Parallel chunk retrieval. Retrieval currently runs sequentially — vector search, then BM25, then rerank — when two of those three steps don't actually depend on each other. Running vector and BM25 search in parallel with a ThreadPoolExecutor closes that gap:

with ThreadPoolExecutor(max_workers=2) as executor:
    vector_future = executor.submit(vectorstore.similarity_search, query, k)
    bm25_future = executor.submit(bm25_retriever.retrieve, query)
    
    vector_results = vector_future.result()
    bm25_results = bm25_future.result()
    
    fused_results = rrf_fusion(vector_results, bm25_results)

The gain is modest — about 1.5x faster retrieval — since I/O dominates the time here, not CPU.

Lazy loading with streaming. Loading every chunk into memory upfront works fine at small scale and stops working at large scale. Streaming chunks from FAISS on demand, and only loading the top-k matches into memory, is what actually lets this scale past 10 million vectors without running out of memory.

Quantized embeddings. Full-precision embeddings — 768 dimensions at 4 bytes each in float32 — take up more memory than they need to for most retrieval purposes. Quantizing down to int8 (still 768 dimensions, but 1 byte each) cuts memory by 4x and speeds up search, at the cost of maybe 1-2% accuracy:

embeddings_fp32 = embedding_model.encode(texts)  # shape: (n, 768), dtype float32
embeddings_int8 = np.round(embeddings_fp32 * 127).astype(np.int8)  # quantized

Deployment Considerations

Right now this runs as a Streamlit app on EC2 (or a local server), with the FAISS index and documents cached locally, and everything config-driven through TOML for settings and .env for secrets. That's fine for the current scale, but scaling further would mean a few changes: moving from FAISS to something like Pinecone or Weaviate for managed vector store scaling, keeping Bedrock but adding retry logic and exponential backoff, introducing Redis for distributed caching of embeddings and reranking scores, logging retrieval quality (did the reranked chunks actually contain the answer?) alongside LLM latency and embedding staleness, and running A/B tests comparing BM25-only versus hybrid versus knowledge-graph retrieval on real queries.

Summary

If there's one thing this project made clear, it's that RAG quality comes down to retrieval precision far more than raw LLM power. Hybrid search, context-aware reranking, glossary expansion, incremental indexing, and conversational memory — none of these are individually clever, but together they're what makes the system deliver accurate, grounded answers instead of confident guesses. Knowledge graphs and multi-query expansion feel like the next places to push for even more precision.

The takeaway I keep coming back to: RAG is only as good as the chunks you feed it. Get retrieval right, and the LLM does the rest without much drama.