Document Analysis Automation
An $8 billion lawsuit's evidence sat in 50,000 scanned documents. My pipeline read every page — and lawyers validated what it found.
Context & Constraints
An $8 billion lawsuit was running in two countries at once, the US and Brazil. It involved retroactive tax exemptions. The evidence lived in more than 50,000 documents going back to 1958 — each between 3 and 600 pages. Some were handwritten. Some were typed on typewriters. Some were printed and scanned. Only the newest were born digital. The task: find every mention of profit or dividend distributions to shareholders.
Reading everything by hand was off the table. The manual estimate: 5 consultants for 10 days — per batch of 500 documents, and only for the initial questions. At that pace, 50,000 documents were out of reach. Three AI tools had already been tried: an internal LLM chat, Microsoft Copilot, and a batch document tool. All three failed for the same reason: their OCR could not read the old documents. When the text extraction fails, no language model can save the answer. The project then came to the Tax AI team.
- Legal context: the output had to survive scrutiny from lawyers on both sides. 'Mostly right' was not an option — every page's result had to be auditable.
- The documents spanned seven decades of formats: handwritten, typewritten, printed-and-scanned, born-digital. One OCR setup could not read them all.
- Scale: more than 50,000 documents, 3 to 600 pages each. Any manual step in the middle would break the pipeline.
“Three AI tools had already failed on these documents. The problem wasn't reasoning — it was reading.”
Architecture
The pipeline has four stages. Pre-processing: a Python script splits every document into individual pages and tracks each one, so nothing is silently dropped. Reading: standard OCR (AI Builder) handles the clean pages; the difficult ones — handwriting, typewriter text, degraded scans — go to a custom model trained on Azure Document Intelligence. Interpretation: a chain of prompts processes each page. One prompt interprets the content. One looks for evidence of dividend or profit distributions. One structures the finding into the output format. Each prompt has a single job, like a small agent, which makes failures easy to locate. Delivery: Power Automate orchestrates the flow end to end and writes the result — one row per page, with the evidence flag. Why page-level? Because accuracy collapses when a model reads a 600-page document at once. Small inputs made the reading accurate and made every error traceable to a single page.
- 50,000+ docs, 1958–present · 3–600 pages — input. Leads to: Page splitter.
- Page splitter, Python · tracks every page — process. Leads to: Standard OCR (clean pages), then Custom OCR model (hard pages).
- Standard OCR, AI Builder — process. Leads to: Prompt chain.
- Custom OCR model, Azure Document Intelligence — agent. Leads to: Prompt chain.
- Prompt chain, interpret · find · structure — agent. Leads to: Evidence sheet.
- Evidence sheet, one row per page — output.
Document Analysis — page-level pipeline, orchestrated in Power Automate
Decisions & Trade-offs
Split everything into single pages
A Python script breaks every document into one-page units before anything else runs.
Considered: Process whole documents · Chunk by sections
Two reasons. Accuracy: a model reading one page at a time misses far less than a model reading a 600-page file. Auditability: when a lawyer questions a result, the answer points to one page, not to a whole document. Tracking pages individually also guarantees coverage — no page can be silently skipped.
Train a custom OCR model for the hard documents
Standard OCR for clean pages. A custom Azure Document Intelligence model for handwriting, typewriter text and degraded scans.
Considered: One generic OCR for everything — the approach the failed tools used
The three tools that failed before all failed at the same stage: reading. They reasoned fine over text they could extract, but they could not extract text from the old documents. Fixing the reading layer was the whole game. A dedicated model for the difficult formats raised extraction quality enough for the rest of the pipeline to work.
A chain of single-purpose prompts, not one big prompt
Separate prompts to interpret, find evidence and structure the output — orchestrated like a small agent flow.
Considered: One prompt doing everything
In a legal context, you need to know where a mistake happened. With one giant prompt, a wrong answer is a black box. With a chain, each step's output is visible — an error is traceable to the step that produced it, and fixable without touching the rest.
- Power Automate orchestrates the flow instead of a pure-code pipeline. Inside KPMG's Microsoft stack, it was the fastest path to production and kept the flow readable for non-engineers — at the cost of less flexibility than code.
- The custom OCR model took training effort that a generic OCR would not need. The three failed attempts had already proven the generic route did not work on these documents. The effort was the price of accuracy.
My Role
Three tools had failed before I was called in. I designed the full architecture and built every stage: the Python pre-processing, the Power Automate flow, the prompt chain, and the custom OCR model. I also worked directly with KPMG's lawyers to define how the results would be validated.
Results & Validation
The pipeline read every page of more than 50,000 documents with 98% accuracy — validated by KPMG lawyers on a 10% sample. Its output became one of the main pieces of evidence used in the case. Two decisions made the difference: split documents into single pages so nothing gets skipped, and fix the reading layer first — the tools that failed before failed at OCR, not at reasoning.
Stack
Building AI that has to work every time?
I design, ship and operate systems like this one.
Let's talk