Most RAG tutorials use Wikipedia or PDF whitepapers as their corpus. Regulatory documents are a different problem. They are dense with cross-references, defined terms that override common meanings, and section hierarchies that carry legal weight. A chunk boundary in the wrong place can return an answer that is technically incorrect under the regulation.
Why Regulatory Text Is Hard
Defined terms. "Occupancy" in IBC Chapter 3 means something specific that differs from plain English. Chunking across the definitions section loses this.
Cross-references are load-bearing. "As required by Section 1604.3.1" is the actual rule. A chunk capturing the obligation without the referenced section returns an incomplete answer.
Hierarchical structure. IBC sections run Chapter to Section to Subsection to Exception. Exceptions frequently override the parent rule. Ignore this hierarchy and you miss exceptions.
Embedding Model Selection
| Model | Recall@5 |
|---|---|
| all-MiniLM-L6-v2 | 0.61 |
| e5-large-v2 | 0.74 |
| bge-large-en-v1.5 | 0.89 |
BGE-Large-EN-v1.5 was the clear choice: at recall@5 of 0.89 it led the field by a wide margin over e5-large-v2 (0.74) and all-MiniLM-L6-v2 (0.61), at an inference cost low enough to run on-premises without bottlenecking a synchronous API.
Chunking Strategy
Structural chunking first: parse the document heading hierarchy and chunk at section boundaries. Then add one paragraph of semantic overlap at boundaries. Every chunk gets tagged with corpus, chapter, section_id, effective_date, and jurisdiction for filtered retrieval.
Vector Store: Qdrant
We evaluated Qdrant against Milvus. Milvus has the higher throughput ceiling and richer distributed-configuration options, but it carries a heavier operational burden: it expects an etcd cluster and a MinIO object store as dependencies — a lot of moving parts to stand up and maintain on the single-node, on-premises hardware our deployments target.
Qdrant won on operational simplicity. It ships as a single binary, deploys cleanly as one container, exposes both gRPC and REST with solid client libraries, and does native payload filtering at the ANN search level — composable without post-retrieval overhead. Its throughput is more than sufficient for our scale.
For a regulated on-premises deployment where the entire stack runs on customer hardware, that simplicity was decisive. We did not want customers operating etcd clusters just to run a vector store. The corpus currently holds approximately 2.1M vectors across all regulatory sources.
What We Are Still Solving
Amendment tracking. Regulations get amended. Our current pipeline treats the corpus as a snapshot. Version-aware retrieval with effective dates is the next major infrastructure piece.