Sovereign RAG · Evaluation

Measured per step, not asserted

Anyone can claim a RAG system “works.” This page is the proof for the live demo — and it isolates each processing step so a weakness can be located, not just noticed. Three pillars: the ingestion that builds the chunks, the retrieval that finds them, and the rerank step measured on its own. The gold set and the method are disclosed in full below.

These numbers are for this demo only (588 chunks of the EU AI Act + GDPR) and are unrelated to retrieval figures quoted for other projects on this site, which were measured on different systems and corpora.

Measured directly on the chunks, with no retrieval — does the pipeline turn two regulations into clean, complete, embedding-friendly units?

Extraction is structural and lossless (one chunk per recital; one per article, long articles split on paragraph/point boundaries), so there is no noise-filter stage — willma's noise_ratio and filter_loss have no analogue here and are N/A. Zero-chunk provisions: 0 · oversize-chunk rate 0.0% · max 837 words.

2 · Retrieval

The gold set (40 questions, 16 DE / 24 EN) through the production path — multilingual embedding, Qdrant top-20, rerank to top-10.

3 · Rerank lift

The same metrics computed on the raw vector order (before rerank) and after rerank. The delta is exactly what the rerank step contributes — isolated, not inferred. It moved 11 questions to rank 1 that the vector search alone ranked lower.

Methodology & the full gold set

Three pillars, each isolating one processing step. Pillar 1 (ingestion) is measured directly on the chunk corpus, with no retrieval. Pillars 2 and 3 run a self-labelled gold set (16 DE / 24 EN) through the production path — multilingual embedding, Qdrant top-20, rerank to top-10 — and score it. Pillar 3 compares the raw vector order against the reranked order, isolating the rerank step's lift. A negative set of out-of-domain questions checks the honesty cut-off. Binary relevance; nothing is tuned on the gold set.

All 40 questions, with the rank of the target provision before → after rerank — nothing hidden, including the 1 miss.