Accepting an AI knowledge base: pilot acceptance tests

This guide is for someone reviewing a delivery. For the architecture decision, start with RAG for businesses. For the project stages, see how to build an AI knowledge base.

Write down the business area, included sources and versions, authorized users and tasks the system can support. An internal procedure assistant does not automatically include drafting contracts, changing an ERP or answering questions for every department. Agree who validates the content and who authorizes operational use.

Acceptance scope should match the contracted scope. If a new source or task emerges during the pilot, record it as an extension. Separate that extension from correcting behavior that was already agreed.

We recommend recording each case with its question, user or role, authorized documents, expected evidence, actual answer, outcome and owner of the next step. Define what counts as a pass before running the test. The matrix below proposes situations to include; expected results depend on your organization's rules.

Fictional example: operations staff may read a returns policy, while individual commercial agreements are restricted to a specific group. Test both a question about the returns procedure and one about a restricted agreement. Success on the first does not demonstrate correct access control on the second.

Use questions your team asks at work and associate each with the evidence needed to answer it. Include different phrasings, missing information and relevant exceptions. A person who understands the process should validate expected answers; the supplier should not be the only judge of its delivery.

Separate questions used to tune the system from questions reserved for accepting a version. Record what changed between tests: documents, configuration, model or rules. This lets you repeat the evaluation and verify a correction without relying on an improvised conversation.

Anthropic's evaluation guide recommends specific, measurable criteria tied to the task, including edge cases. Percentages in its examples are not universal targets for your knowledge base; your goals should reflect the agreed use.

An answer can sound correct while citing a document that does not support its claim. Ask to open the source, locate the passage and check its version. Distinguish three failures: required evidence was not retrieved, the answer added something unsupported, or the citation points to the wrong place. Each can require a different correction.

Amazon Bedrock separates faithfulness, citation precision and citation coverage in RAG evaluation. This distinction helps review a delivery; an average quality score does not replace checking a critical case or grant permission to disclose information.

Ask how updates propagate, what interval is expected and who investigates a synchronization failure. Repeat tests after access is revoked or a document is replaced, considering existing conversations and source links too. Recording that someone had access yesterday does not resolve whether they should still have it today.

As a technical example, Microsoft documents query-time access controls and permission-metadata update requirements. That capability is described as preview and has limitations; it is not an automatic permission guarantee for every system. Require the supplier to explain the behavior of the architecture actually delivered.

We recommend treating unauthorized information exposure as a blocker to expanding the pilot until it is corrected and retested. Define other blocking failures with IT rather than hiding them inside an overall score.

The acceptance record should identify the tested version, included sources and roles, executed cases and evidence. List unresolved issues with an owner and a closure condition. Your team can accept the scope, request corrections or restrict use to verified tasks; approving an internal query does not approve actions on external systems.

Agree which changes require retesting and who maintains documents after delivery. If the system fails, the record should distinguish a defect in the original scope from a new request. The evaluation remains useful after the pilot as a reference for reviewing changes.

At Duotach we implement knowledge bases and query agents around specific sources, permissions and workflows. Discovery, test design and implementation are part of the contracted scope. The proposal defines the delivery, exclusions and acceptance criteria according to the available systems and documents.

To discuss scope, describe the business area, available sources and who would approve the answers. Our AI agents service is the entry point for this work. Do not send credentials or sensitive documents through the contact form.