Retrieval evaluation

Held-out slice

The 32 paraphrased queries, which deliberately avoid the target section's vocabulary. These are the numbers that mean something.

The same numbers, as shapes

Two things the table cannot show. On the left, how the configurations converge as k grows — BM25 never catches up, and the reranker's advantage is largest at the top of the list, where it matters. On the right, what that advantage costs: latency is on a log scale because a flat vector scan answers in a millisecond and a model call takes eleven seconds, and on a linear axis every row but one would sit on zero.

Citation lookups

Queries that name a section outright. This slice is here because the headline result is that fusion does not help, and that conclusion is only honest if the query class where the lexical arm is indispensable is on the page too.

Development slice

The 76 direct queries. The fusion weight was chosen here, so read these as a description of the slice rather than an independent result.

Choosing the fusion weight

Equal-weight reciprocal rank fusion is the textbook default, and it is the wrong default here. BM25 is much the weaker arm on this corpus, and giving it an equal vote drags the fused ranking below dense retrieval alone. Swept on the development slice, the best lexical weight is 0 — which is to say the honest answer is that fusion earns nothing here, and the report says so rather than reporting a hybrid number that a tuned weight quietly rescued.

Lexical weight against development nDCG@10. Bars are scaled to the best value, not to zero.

Chunk size

Four full rebuilds of the index across a fourfold range of chunk sizes, each evaluated by the same harness on the same held-out (paraphrased) slice. Dense retrieval, reranker off, because the reranker sits downstream of what the chunking makes retrievable.

The result is a null one, and the shipped configuration is deliberately left alone. nDCG@10 moves across a range smaller than 32 queries can resolve. What does show a shape is a tradeoff rather than a winner: larger chunks put the answer at rank 1 more often, smaller ones cover more by rank 10. The nominally best row is not adopted, because choosing it on held-out numbers would fit the system to its own test set. Chunk size is a knob that gets turned by reflex; on this corpus it is not where the quality is — reranking is.

Configurations

BM25

Okapi BM25 alone (k1 = 1.2, b = 0.75), no fitted parameters

Dense

Cosine over float32 gemini-embedding-001 vectors at 768d

Dense (int8)

The same vectors quantized to int8 with one scale per vector, 4x smaller

Hybrid RRF (equal)

Dense and BM25 fused by Reciprocal Rank Fusion, k = 60, both arms weighted 1

Hybrid RRF (weighted)

The same fusion with the lexical arm weighted 0, chosen on the dev slice

Hybrid + rerank

Top 30 weighted-fusion candidates reranked listwise by gemini-3.6-flash

What this does not prove

• The labels are one person's judgement. Every query was written after reading the corpus and every gold section was read to confirm it answers the query, but a relevant section nobody thought of counts as a miss. That penalises every configuration equally, so the ordering of the rows survives even where the absolute numbers are conservative.
• The held-out slice is 32 queries. One query moves recall by about 3.1 points, so differences smaller than that are noise. Read the table for its ordering, not its decimals.
• Two of the eight frameworks Verity classifies have no corpus at all. SOC 2 and ISO/IEC 27001 are copyrighted and cannot be redistributed here.
• The reranker is a language model. Candidate order is shuffled with a fixed seed so it cannot simply echo the fusion order, and temperature is zero, but reruns still move.

Disagree with it in the playground — the same configurations, on whatever query you like.