KBSE: Knowledge Base Search Engine
A search engine that answers legal and regulatory questions — and links every answer to its original source.
Context & Constraints
KPMG's Legal and Corporate teams needed to consult legislation and internal legal knowledge. This was 2024: generative models were still weak at precise information retrieval. And in a legal context, a wrong reference can compromise real analyses and decisions.
Conversational models of the time had low precision when searching large document bases. A confident answer with a wrong legal reference is worse than no answer. The teams needed more than answers. They needed the applicable legislation, a contextual interpretation, references to case law when relevant — and a direct link to the original source, so anyone could verify the answer in one click.
- 2024 models: generative AI was still unreliable at precise retrieval. The architecture had to compensate for the model, not lean on it.
- Legal domain: a wrong legal reference can compromise analyses and decisions. Traceability was a requirement, not a feature.
- Every answer had to be verifiable by the user: applicable legislation, interpretation, case-law references, and a direct link to the source.
“The legal team only started trusting the system when every answer showed the exact law it came from.”
Architecture
Four stages, each with one job. Retrieval finds the most relevant documents by vector search — ChromaDB stores the embeddings, generated with Azure OpenAI. Consolidation organizes and merges the evidence retrieved across documents. Validation checks the consistency of that evidence before any answer is written. The answer stage generates the final response with its legal grounding: the applicable legislation, an interpretation in context, case-law and regulatory references when relevant, and a direct link to the original source. User feedback closed the loop. Every rating was stored in Supabase and reviewed during the three-month validation with the Legal and Corporate teams. The pipeline was refined against real usage, not against a benchmark.
- Legal question — input. Leads to: Retrieval.
- Retrieval, vector search — agent. Leads to: Consolidation.
- ChromaDB, Azure OpenAI embeddings — store. Leads to: Retrieval.
- Consolidation, merge evidence — agent. Leads to: Validation.
- Validation, consistency check — agent. Leads to: Answer, then feeds back to Retrieval (re-query).
- Answer, legislation · source links — output. Leads to: feeds back to User feedback (ratings).
- User feedback, Supabase — store.
KBSE — four-stage RAG pipeline with source-linked answers
Decisions & Trade-offs
Validate before generating
A validation stage checks the consistency of the retrieved evidence before the answer is written.
Considered: Generate directly from retrieval — standard RAG
In 2024, the model was the weak link. Letting it write from unchecked excerpts meant confident answers built on wrong evidence. Checking the evidence first cost latency — and paid for it in trust.
Link to the source, don't just cite it
Every answer carries a direct link to the original document, next to the legislation and the interpretation.
Considered: Textual citations only
A citation asks the user to trust the system. A link lets the user check. For a legal team, verifying an answer in one click is what turned the tool from a curiosity into something they actually used.
Measure with users, not benchmarks
About three months of evaluation with the Legal and Corporate teams, rating real answers. Feedback stored in Supabase.
Considered: Offline benchmark evaluation
The teams' real questions were the distribution that mattered — no benchmark reproduced them. Rating live answers for three months produced two honest numbers: 94% validated accuracy and 99% rated relevance. Numbers earned that way hold up.
- The validation stage adds latency to every query. In a legal context, a slower correct answer beats a fast wrong one — the trade was deliberate.
- Built with 2024 models, so the pipeline carries heavy scaffolding. Newer models would need less of it. The traceability layer — sources, links, validation — stays valuable no matter how good the model gets.
My Role
This was a team project. I worked on the RAG architecture, implemented the retrieval, consolidation and answer pipeline, and helped structure the vector base (ChromaDB with Azure OpenAI embeddings). I also sat with the Legal and Corporate teams through the validation rounds.
Results & Validation
After about three months of testing with Legal and Corporate professionals, the system reached 94% validated answer accuracy — and 99% of its answers were rated relevant by the users. The lesson from building RAG in 2024: when the model is weak, the architecture has to carry the precision. Retrieval, consolidation and validation did the work the model couldn't.
Stack
Building AI that has to work every time?
I design, ship and operate systems like this one.
Let's talk