In this guide 6 sections
Name the task first. Then check completeness, labels and provenance, clean out duplicates, hold back an evaluation sample, mask sensitive fields, and schedule a review so quality does not decay after launch. Reporting teams already practise most of this, so the genuinely new work is access control and testing the system against answers you already know to be correct.
Define the task before touching the data
Consider a bookkeeping firm with several branches that wants an assistant to answer staff questions from its process documents and past client correspondence. Preparation means different things for different tasks. A question-answering assistant needs accurate, well-organised text, a classifier that sorts incoming enquiries needs a labelled set of past examples, and a forecast needs consistent historical numbers. Fix the task first.
Decide what is allowed in. Not every field a business holds belongs in an AI tool, least of all personal details, payroll records and commercially sensitive pricing. The decision sits with the person accountable for the data, such as the finance lead for payroll files, and not with whoever happens to be building the assistant.
Completeness, labels, provenance and permissions
Gaps are costly here. A system grounded on records with large unexplained holes will produce fluent answers that simply reflect those holes, instead of warning you that the evidence is thin. Count how many relevant records actually contain each field the task depends on, such as client type, filing year or matter reference, and do it first.
Labels need care too. Where a task calls for labels, such as enquiry categories, they must be accurate and applied identically, because a model learns inconsistency as readily as the pattern you intended. Provenance is where each record came from. Permissions record who may use it, and both should be written down before the data leaves its source system, especially client correspondence collected for another reason.
Clean, structure and hold out an evaluation sample
Cleaning works as it does for reporting. Standardise formats, merge duplicate clients, validate dates and amounts, and strip email signatures and quoted reply chains that would otherwise be read as fresh content. Structure matters as well. Free text, a labelled table and a time series each need a different preparation route, so match the method to what the task actually consumes.
Hold some data back. Before deployment, set aside a sample of prepared records purely for testing, with known correct answers the system has never seen during setup or grounding. Testing on the material the system was built from flatters it, and the first unusual client enquiry in production then exposes the difference in front of staff.
Protect sensitive fields and document limitations
Personal details, bank and tax information and commercially sensitive terms each need a handling decision before launch: mask it, exclude it, or allow access only under named conditions. Presence in the export is no reason. Treat it as an access-control decision, reviewed with the same seriousness as who may see payroll in the accounting system, and record the approver’s name and the date.
Write down the limits of the dataset. Note thin coverage, months with poor data quality, and categories with few examples, so whoever reviews outputs knows where answers deserve less trust. Not every answer deserves equal confidence.
Maintain quality, freshness and access over time
Preparation is not a one-off task. A dataset prepared well last year drifts as source systems change, new document types appear and old procedures are superseded, and answers degrade without any obvious trigger.
Build a routine review. Each quarter, re-check completeness and label accuracy, confirm that access permissions still match who works there, and update the written limitations along with the evaluation sample. Launch-day quality is not permanent.
AI data-readiness worksheet
A worksheet listing each data source, with the AI task it serves, fields used, completeness score, handling decision for each sensitive field, and test sample size. Complete it before the assistant is built.
Open a blank worksheet to printQuestions owners ask
Do we need labelled data for every AI use case?
Labels are not always required. Classification tasks need correctly tagged past cases to learn from, while an assistant answering questions from existing documents depends mostly on those documents being accurate, current and well organised.
Can poor data produce misleading AI outputs?
Yes, and often with no visible warning. A system built on incomplete, inconsistent or stale records still writes fluent, confident answers, which makes its errors harder to spot than the gaps in an obviously broken report.
How much data is enough?
Enough depends on the task. Test a small prepared sample against known answers first, then look at gaps and label errors before collecting more, because extra volume rarely repairs inconsistent records.