Retrieval quality, mandatory citations and ‘I don’t know’ behaviour: practical design principles for a trustworthy RAG-based assistant.
Large language models are fluent, but fluency is not accuracy. In an enterprise setting a wrong policy answer can create compliance risk or lose a customer. Retrieval-augmented generation (RAG) architecture reduces hallucination markedly by tying the model to the organisation’s own documents, but simply adding a vector database is not enough. In building Ilmek Bilgi, the outcomes of the following seven decisions were decisive.
1. Chunking should follow document structure
Fixed-length chunking breaks the context of headings, tables and bullets. Structure-aware chunking that preserves heading hierarchy and table rows significantly raises the rate of finding the right paragraph.
2. Hybrid search: semantic + keyword
Embedding-only search is weak on exact expressions such as article numbers or product codes. Combining keyword (BM25) and semantic search with reranking is the single strongest improvement.
3. Authorisation must be applied at retrieval
A document the user cannot see should never be given to the model. Filtering permissions after answer generation is a security hole.
4. Citations must be mandatory
Every claim must link to a paragraph. Sentences that cannot be linked are removed from the answer. This rule eases both trust and debugging.
5. ‘I don’t know’ behaviour must be trained
If the retrieved context is irrelevant the model should say it does not know and refer to an expert rather than invent an answer. This behaviour is measured separately in the evaluation set.
6. A continuous evaluation set
Build a set of at least 200 questions from real user questions with known answers and sources; run it automatically on every model, setting and content change. Track correct answers, source accuracy and refusal rate as separate metrics.
7. Content governance
Outdated policy misleads even the best model. Assign owners, validity dates and version information to documents, and remove old versions from the index.
These seven decisions are applied by default in the product; you can measure the results in a three-week pilot with your own document set.
Measurable targets
- Correct-answer rate (by human evaluation): 90% and above
- Citation accuracy: 98% of citations in answers should actually point to the relevant paragraph
- Correct refusal rate: 85% or more ‘I don’t know’ on questions not in the knowledge base
- Average response time: under three seconds
These targets vary with the use case; what matters is to write the target before the pilot and report regularly. In high-risk areas such as legal and compliance, an extra human-approval layer is recommended.