Multi-Stage Legal RAG with Azure AI Search

Standard RAG implementations chunk documents by token count and call it done. For general-purpose question answering, this works. For legal documents, it is a hallucination factory. Legal ordinances have specific structure: preamble, operative sections, definitions, and schedules. Schedules contain country lists, fee tables, and rate tables. These are the most frequently queried content and the most likely to be merged into a blob with surrounding pages by a token-count chunker. When the schedule disappears into the blob, the retrieval system cannot surface it independently. The model, finding no ground truth in the retrieved context, draws on training data and confidently gets it wrong. The client needed accurate answers across 3,145 Hong Kong and Singapore legal ordinances, covering questions about specific contract clauses, regulatory requirements, fee schedules, and country-specific enforcement rules. A question like whether France is on the reciprocating countries list under Cap 188B would consistently produce hallucinated answers because the schedule table was buried inside a larger chunk.

The solution has three architectural pillars: structure-aware chunking, a five-layer retrieval pipeline, and a corpus audit system. For chunking, every page is classified into one of four semantic types using regex pattern matching: schedule, definition, preamble, or operative. Schedule detection takes highest priority. Chunks are determined by section-type transitions, not token count. When the page type changes, a new chunk begins. This means schedule tables always get their own standalone, independently retrievable chunk. Each chunk carries 15 metadata fields including cap number, section type, section number, and legal category. Chunk IDs are deterministic MD5 hashes, making re-indexing idempotent through mergeOrUpload. The retrieval pipeline has five layers. Layer one detects jurisdiction using weighted regex patterns for Hong Kong and Singapore terminology. Layer two runs pre-agent interceptors that catch security probes and verify deep subsection references in five milliseconds without touching the LLM. Layer three evaluates ten deterministic injection triggers. If a query matches a known pattern like stamp duty rates or annual leave entitlements, the system directly queries Azure AI Search with OData filters and injects the exact chunk as verified ground truth before the agent runs. Layer four runs the Azure AI Foundry agent with BM25 keyword retrieval followed by semantic reranking across the full index. Layer five applies post-generation guardrails that scrub forbidden URLs and parse citation metadata. Conversation memory runs on Cosmos DB with four containers. Azure AI Foundry maintains state through a previous_response_id mechanism, and GPT-4o summarizes every four turns with delayed injection to prevent the model from citing summaries as factual sources. The entire system deploys on Azure App Service with a Next.js frontend that receives responses via SSE streaming. New documents uploaded to Azure Blob Storage automatically trigger the indexing pipeline.

Built a production legal RAG system for 3,145 Hong Kong and Singapore ordinances with a five-layer retrieval pipeline. The system uses structure-aware page classification to chunk documents by semantic type instead of token count, eliminating schedule-merge hallucinations. Ten deterministic injection triggers bypass semantic search for known query patterns and inject verified ground truth. Azure AI Search handles BM25 retrieval with semantic reranking, GPT-4o on Azure AI Foundry generates grounded answers, and Cosmos DB manages multi-turn conversation memory. The 200-question validation suite completed three consecutive clean runs with zero hallucinations.

The 200-question test suite passed with zero errors, zero hallucinations, and zero URL leaks across three consecutive identical runs. Structure-aware chunking resolved the core retrieval problem where schedule tables were being merged into surrounding text, causing the model to hallucinate country lists from training data. The corpus audit identified and fixed 173 caps with missing schedule chunks and 171 with incomplete coverage across all 3,145 ordinances. Belgium, Norway, and France were all correctly identified as non-reciprocating countries without any per-country hardcoded logic. Fake sections like Section 999 of Cap 486 were correctly refused instead of fabricated.