In this guide 6 sections
A good report lists each test case with its result, the browsers and devices covered, open defects by severity, load and data checks, and who retested each fix. A demo shows one happy path. The report shows what still fails. Request it before paying.
A demo is not evidence
In a demo the vendor chooses the path, the data and the machine, so everything works because it was rehearsed on that laptop the night before. That proves the software can work once. It does not prove it works on a busy Monday morning, with five staff logged in to the staging server and a month of invoices waiting behind them.
Request the paper behind it. A report records what was tried, by whom, on which device, with what outcome, and a team that tested properly can export one in an afternoon. Hesitation tells you something.
Ask for test cases with results, line by line
The core is a list of test cases. Each case carries an ID, the steps, the expected and actual result, a date and the tester’s name, so count them against the clauses of your signed specification. Gaps there are gaps in coverage.
Look for negative cases too. Ask what happens when the invoice date is blank, when the uploaded file is far too large, or when a storekeeper opens the accounts screen without permission. Success-only reports were written for approval. They were not written to find faults.
- Test case ID linked to a requirement number
- Steps, expected and actual result
- Pass, fail or blocked, with date
- Tester name, not just the company
- Cases for wrong input and wrong permissions
Check the browsers, devices and data they covered
Ask for the matrix. It should list which browsers, which phones and tablets and which screen sizes were tried, and name the older Android handset or outdated Edge laptop your staff use. New vendor laptops prove little.
Then ask about data. Was the work done on three sample records, or on a copy shaped like your real files, with long customer names, blank tax fields and duplicate accounts? Ask whether a load check ran at your busiest hour, with every staff login active, and how long the slowest page took to open.
Read the open defects and the retest trail
Every honest report has open defects. A zero count looks suspicious. What matters is severity and ownership, so list the defects by band and read the promised fix date for anything severe before deciding which regression risks you can tolerate.
Then check who retested the fixes. If the developer who made the change also marked it closed, nobody independent looked, so ask for the retest date and the build number. A defect closed on build 41 says nothing about build 44, which is the build that actually shipped to your production server after the vendor’s final fixes.
- Open defects sorted by severity
- Fix date and owner for each
- Retest by a person other than the developer
- Build number on every closure
Decide what to sign and what to hold back
Never sign everything at once. Approve what passed, list what failed, and tie the final instalment to those failures being closed, written into a signed email that both sides keep on file.
Run your own short acceptance session. Two or three staff should do a real day’s tasks, such as raising a quote, receiving stock and printing the month-end statement, and log every finding. Sign after their result. Never sign on a summary page.
Vendor test report review sheet
Twelve rows, one per request. The left column names the request, such as test cases with results, the browser and device matrix, the load check, open defects by severity and retest evidence. Right column: status, note, initial. A closing line records approve, approve with conditions or reject, plus the review date and the initials of the person who decided.
Open a blank worksheet to printQuestions owners ask
Is a demo ever enough to approve software?
Only for something tiny. For anything that handles money, customer records or daily operations, ask for the written test report as well and run your own acceptance session with real staff.
What if the vendor says testing is internal?
Internal does not mean invisible. Ask for a redacted report, or a summary showing case counts and open defects by severity, and treat a flat refusal as a reason to hire independent acceptance testing.
How many defects are acceptable?
The number matters less than the severity. None should block the main task or touch money, and each minor one needs an owner’s name and a fix date written beside it.
Who should retest the fixes?
An independent person should do it. That can be the vendor’s tester or your own staff, but the retest must be recorded with a build number and a date so the closure can be checked.