How Our Multi-step Retrieval-Augmented Generation (RAG) Improves Information Retrieval and Search Accuracy

RAG-based chatbots generally perform well if the document corpus is less complex. However, when it is on a bigger scale, they struggle to provide accurate information. This is why they are not as efficient in the context of a bank where there are several documents in different formats. To tackle this problem, a new bot that can increase response accuracy had to be developed. 

The new RAG process involves a key process that prepares documents before retrieval. This is referred to as pre-processing, which now involves the following:

• Coherent chunking: Documents are separated into meaningful chunks, which retain the context that may otherwise be scattered into different sections, paragraphs, and tables.

• Question generation: Using LLMs, another layer is added, which generates potential user questions from the chunks, which improves the retrievability of content.

• Vector embedding: The document chunks are embedded in the vector store so that it retrieves information based on semantic similarity.

Once the embedding was done, work began on the RAG pipeline. When a query enters the new system, it is processed by an NLU layer, which rephrases the query. This is important because it improves accuracy by boosting BM-25 based keyword search and HNSW-based vector similarity search.

The response generation happens in two stages, which are:

• Stage 1: The top-k chunks are passed to the LLM, which produces the first draft.

• Stage 2: The second agent verifies the draft based on the BM-25 keyword search and provides additional context.

The dual-stage response generation enhances accuracy and performance.

• Lack of contextual awareness: During the chunking process, it is possible for partial information to slide into another chunk without context.

• Documents were not LLM friendly: Documents with different formats, like tables or with different products spread across several pages, could not be linked logically, which again led to missing information.

• Information duplication: The same information would be repeated with slight changes in the tone, language, or nuances.

• Recency and partial updates: This included updated information or policies which must be retrieved instead of the older version. 

While not all these issues are not addressed in the current architecture, the team seeks to come up with solutions.

The banking sector requires accurate information every time since it can influence major financial decisions and customer transactions. Guardrails and strict grounding prompts must be in place to prevent inaccurate and made-up information. This includes:

• Not going beyond the retrieved content

• Avoiding hallucination, speculative or generative content, or partial information

• Using only validated documents to retrieve information. 

• Redirecting to humans in case of uncertainty.

It took continuous prompt engineering and domain expert feedback to solve these challenges.

When accuracy and consistency cannot be compromised, you need a system that can split the task into multiple steps to ensure every piece of information is based on documented evidence. Iterative testing and prompt tuning were necessary at each step. Collaborating with domain experts is an important step to identify relevant terms and topics.