The mechanics of evidence-grounded archive search
An archive system is only as trustworthy as the source behind each answer. We separate retrieval from generation. Search runs two methods in parallel: BM25 for exact terms, file numbers and proper names, and a vector search over self-hosted embeddings for semantic proximity. Both result lists are merged through Reciprocal Rank Fusion. A cross-encoder reranking step narrows the top 50 candidates down to the five most relevant passages. Only those reach the answer.
Before output, an NLI sentence-level grounding check tests every generated sentence against the cited sources. If the model finds no support, the system replies not found rather than guessing. This abstention is the point: no invented references, no hallucinated case numbers. An eval_harness runs the RAGAS Faithfulness metric ahead of every release.
To navigate large holdings, PageIndex provides a document tree down to section level. The knowledge graph links entities, relations, paths and communities, which makes cross-comparisons across thousands of files practical. Where audit integrity is mandatory, we implement the framework defined by GoBD, section 147 of the German Fiscal Code and WORM audit logging.
- Hybrid search combining BM25 and vector retrieval with rank fusion
- Reranking from top-50 to top-5 before any answer
- Sentence-level grounding with abstention when evidence is missing
- PageIndex and knowledge graph for structure and cross-references
- Audit integrity aligned with GoBD and section 147 AO
This architecture runs in production, including the AI archive of eine große Regionalzeitung. Built for archives, newsrooms, law firms, public authorities and research. Your data stays on our servers in Germany.