Decide whether RAG addresses the actual problem
If an employee knows which file they need but cannot open it, fix access or document organisation first. RAG becomes worth assessing when users need to combine explanations across documents and reach the relevant evidence quickly. An existing enterprise search product may already meet the requirement.
Limit the first scope to a defined collection and user group. Establish document ownership, current versions and intended use. Uploading an entire unmanaged archive transfers uncertainty from employees to the model rather than resolving it.
Evaluate retrieval and answers separately
Consider the question: “What limits apply when this product is used outdoors?” The system should identify the current documentation for the correct product, retain the conditions in its answer and provide the source. A polished answer based on a similarly named discontinued product is still wrong.
| Test case | Expected behaviour |
|---|---|
| A current document contains the answer | Answer supported by the relevant version and passage |
| Two sources disagree | Surface the conflict rather than hide it |
| The collection has no answer | Explain that supporting information was not found |
| The user lacks access | Do not expose content, titles or quotations |
| A document changes or is deleted | Verify index and cache freshness |
This is an illustrative acceptance framework, not a report of completed customer tests.
Enforce permissions outside the model
Identity and access filtering belong in the application. Only permitted material should reach the model. Microsoft's document-level access guidance describes query-time filtering as one implementation approach; confirm the capabilities and maturity of the particular product version you select.
Test revocation explicitly. When an employee leaves a group, what changes in search results, links from earlier answers and shared caches? Set an acceptable propagation delay as part of acceptance. A prompt telling the model to show information only to authorised people is not a substitute for an access-control mechanism.
Build a pilot scorecard
Ask the business owner to prepare answerable and unanswerable questions, keeping final evaluation cases separate from development examples. Measure correct-source retrieval, answer support, permission failures and response time independently. A confidence score produced by the model is not sufficient evidence of accuracy.
Budget for document preparation, index updates, model usage and human review. A frequently changing collection creates an operating responsibility, not merely an installation task. Assign someone to resolve contradictory documents and to decide when an answer should be escalated to a specialist.
Frequently asked questions
Does RAG eliminate incorrect answers?
No. Retrieval, source selection and answer generation can each fail.
Should every document be uploaded immediately?
No. A representative collection with clear ownership and version history makes a more useful first evaluation.
Explore our AI integration approach. Start with your document types, permission rules and several questions employees actually ask.
Discuss your document search pilot
Related guides
Technical references
Sources checked on 17 November 2026.