In this guide 6 sections
Enterprise AI builds run in six stages. Choose one use case, prepare data and permissions, build retrieval or a model, test on real staff questions, add access control, then monitor and hand over. At each stage, ask to inspect something concrete, such as the use case brief, the source inventory, the evaluation results or the access roles.
Stage one: pick a use case you can measure
Choose one task with a named owner, such as answering leave questions for the HR desk or drafting replies to supplier emails about late deliveries. Write down who asks, how often it happens and what a good answer looks like today.
Skip the company wide assistant. A narrow task gives you clear test cases, a clear stop point and a short list of people who can tell you quickly whether the answers are right. Ask to inspect a one page use case brief that names the owner, the inputs, the outputs and the cost of a wrong answer.
Stage two: prepare data and permissions
List every source, from SharePoint folders and scanned PDFs to ticket history and ERP tables, then mark which are current, which are duplicates and who may read each one. An old leave policy left in a shared folder will be quoted back to staff with total confidence, and nobody notices until a payroll query arrives.
Permissions decide the design. If the finance folder is restricted, the assistant must respect that restriction for every user, every time, including on follow up questions. Keep the source file path and the access group with each chunk of text, so a permission check can run before any passage reaches the model.
- Source inventory with owners
- Current versus superseded documents
- Read rights per folder or group
- Personal data that must be excluded
Stage three: build retrieval or a model
Most questions need retrieval, not training. Documents are split into passages and stored as embeddings in a vector store such as pgvector, and the best passages travel to a language model with the question.
Fine tuning is a late option. Ask which chunking approach was used, which model sits behind it and where the data is processed, and ask to see the retrieved passages behind every answer.
Stage four: test against real questions
Collect a few dozen genuine questions from the staff who will use it, with the correct answers and the source page for each. Run them after every change. Record which answers pass, which cite the wrong document and which are invented outright, then sort the failures by cause so the builder sees patterns.
Include trick cases: a question the documents cannot answer, a question about a restricted file and a question phrased badly by someone typing on a phone. The right reply is a plain refusal. Review every failure with the task owner, because only that person can say whether the document or the question is at fault.
- Question set with expected answers
- Citation correct or not
- Refusal on unknown topics
- Restricted file never revealed
Stages five and six: control, monitor, hand over
Add single sign-on, role checks and multi-factor authentication so only approved staff reach it, then log each question, the sources used, the answer given and the response time. Add a thumbs down button. It feeds a review queue.
At handover, ask for the prompt files, the evaluation set, the deployment pipeline and a runbook covering model changes and cost alerts. Name who reviews the feedback queue each week, and write that name into the runbook next to the escalation contact at the builder.
AI build stage gate sheet
Six rows, one per stage. Columns show the deliverable to inspect, the question to ask the builder, the owner on your side and a sign-off date. A final block lists the evaluation set, access roles and runbook needed at handover, with a tick box beside each item for the sign-off meeting.
Open a blank worksheet to printQuestions owners ask
Do we need to train our own model?
Usually not. Retrieval over your documents with an existing language model covers most internal question and answer tasks, and it needs far less data preparation than training would.
Where does our data go?
That depends on the hosting choice. Ask for the region, the provider terms on retention and whether prompts are used for training, before a single document is loaded into the system.
Who checks the answers once it is live?
A named reviewer on your side does, using the logged questions, the sources shown with each answer and the thumbs down queue that staff fill in. Without one, quality drifts unnoticed.
Can staff see documents they should not?
Not if permissions travel with each passage and the system checks them against the signed in user at query time, before any text reaches the language model. Test it with a restricted file.