In this guide 6 sections
Build a test set from real staff questions. For each one, write the expected answer and the source page. Add questions the documents cannot answer, to see whether the assistant admits it. Score the assistant on the set before launch, then run it again after every content or setting change.
Collect questions from real people, not from the project team
Project teams write tidy questions. Staff ask messy ones. A storekeeper types leave rules for contract staff, and a technician types torque for pump coupling. Your test set should sound like that.
Pull questions from places where people already ask. Try the shared inbox, the HR query sheet, the support desk chat group and the notes from your last onboarding session. Real wording matters more than tidy wording.
Aim for forty to sixty questions across every department that will use the assistant, and spread them out so that no single team dominates the set.
- Shared inbox threads asking where a form or rule lives
- Messages on the support desk chat group
- Questions new joiners ask in their first month
- Queries sent to HR, accounts and the quality team
Write the expected answer and the source page
A question alone is not a test. Beside each one, a subject expert writes the correct answer in a sentence or two and notes the file name and page where it lives.
The source page counts as much as the answer. An assistant can reach a right answer from the wrong file, and that breaks the next time the files change. Check both.
Mark a few questions as critical, such as notice periods, expense limits, safety shutdown steps and warranty terms. A wrong answer there costs real money or injures someone, so those rows carry no tolerance. Keep the sheet plain. Each row needs the question, the expected answer, the source document and page, the department and the verifier’s name.
Include questions the documents cannot answer
Add a block of questions with no answer anywhere in the files. Ask about a policy that does not exist or a product you never sold. The right behaviour is a plain statement that the documents do not cover it.
This is where many assistants fail. They produce a confident, plausible answer stitched together from nearby text, and a clerk then acts on it as if it were policy. Also add trick pairs. Ask the same thing in two wordings, and ask about two similar documents, such as the 2025 and 2026 leave policy, because differences between the answers expose weak spots.
Test the edges of access too. Ask a store assistant about director pay bands and see whether the answer leaks a file that only HR should open. Any leak stops the launch.
- Questions about rules that do not exist
- Questions about a different branch or country
- Paired questions worded differently
- Questions mixing two similar documents
Score each answer on three counts
Run every question through the assistant and mark three things. Is the answer correct? Does it cite the right source? Did it stay inside the documents?
Use a simple scale of pass, partial and fail. Two reviewers should mark a sample independently so you can see how often they disagree, because disagreement usually means the expected answer was vague.
Log the failures by type, too. Wrong file, outdated figure, missing caveat and invented detail each point to a different repair, from content cleanup to retrieval settings. Set a launch bar in advance and write it down. For example, no fails on pay or safety questions, and every unanswerable question refused.
Re-score after every change
The set is not a one off exam. A new file, an edited policy, a change of model or a tweak to retrieval can each move the results. Run the full set every time and keep the scores in a dated table.
Keep the scores in a table with one column per run, so the trend is visible at a glance and a sudden drop after a model upgrade cannot hide. When a real question fails in production, add it. Over months the set becomes a record of what your staff actually need, written in their own words rather than in the project team’s.
This week, ask ten colleagues for the last question they could not answer from the documents. That is your starter set.
Assistant test sheet
A one page scoring sheet. Columns cover question, department, expected answer, source file and page, verifier, assistant answer, cited source, and pass, partial or fail. A second block lists the unanswerable questions, and a footer records the date, version and launch bar.
Open a blank worksheet to printQuestions owners ask
How many test questions do we need?
Forty to sixty is a workable start for a company wide assistant. Smaller teams can use fewer, provided every department and every sensitive topic has several questions.
Who should write the expected answers?
The person who owns the subject, such as the HR lead for leave rules or the quality head for inspection steps. A project engineer cannot judge whether a policy answer is complete.
What if the assistant passes the test but fails in real use?
Add the failed questions to the set and study how real users word things. Your first set was probably too tidy. Keep growing it from live logs.