How to pilot an AI assistant for company documents

Choose a clearly defined set of sources

Imagine a support team starting with its current process guide and product help documents. Record each document’s owner, last update date and permitted users. If old and new copies describe the same rule, ask the document owner to confirm which version applies.

Enforce access permissions in the sources used by the assistant. If a support employee cannot access a financial document, the assistant’s search and response must respect that restriction. Ask the same question under different permission profiles, checking both the answer text and source links.

Prepare three types of questions

The first group should cover questions answered explicitly in the documents, such as “What information is required to open a support ticket?” The second should require interpreting two documents together. The third should include questions whose answers are unavailable or inaccessible to the user.

Write the expected answer and correct document section before testing. For each response, the evaluator should record whether the text is correct, whether the source supports the claim and whether access boundaries were respected. When information is missing, directing the user to the right team can also be a success criterion.

Test the pilot in everyday work

Start with a closed group of users. Measure document search time on sample tasks and compare it with the same tasks completed using the assistant. Send ambiguous or outdated answers to the document owner. Use these records to decide which questions will remain in scope after the pilot.

Checklist

• Does every source have an owner and an access rule?
• Can the document section supporting an answer be checked?
• Does the pilot include unanswerable and unauthorised questions?

Related articles

• What should you review before automating a workflow?
• What is GEO, and how do you measure visibility in AI answers?
• AI solutions