Example deliverable · signed before go-live
Evaluation and guardrail report
How the invoice-reading pilot performed on real past invoices, where it failed, and the controls in place. The client signs this before anything goes live.
1. How we tested
300 past invoices chosen by the AP lead, including 60 poor-quality scans and 40 from the vendors with the most naming variations. Each system output was compared with what the specialist actually posted. The pass bars were agreed with the finance lead before testing.
2. Results
| Field | Correct | Agreed bar | Result |
|---|---|---|---|
| Vendor | 294 of 300 (291 before the vendor fix) | 285 | Pass |
| Invoice number | 297 of 300 | 294 | Pass |
| Invoice date | 296 of 300 | 294 | Pass |
| Total amount | 294 of 300 | 294 | Pass (at the bar) |
| Line items, all invoices | 262 of 300 | 270 | Below bar |
| Line items, digital invoices only | 236 of 240 | 216 | Pass |
3. What it got wrong
- Line items on multi-page scans were sometimes split or merged. Decision: line items stay manual for scans until a later round.
- Six totals were misread on faded scans. All six were flagged as low confidence, so none would have posted unnoticed.
- Three invoices from one vendor were matched to a similarly named vendor. Fixed by adding the vendor's tax ID to the lookup; the retest got all three right. The other six vendor misses were unreadable scans, all flagged as low confidence.
4. Guardrails in place
- Nothing posts to the ledger without a specialist's confirmation.
- Low-confidence fields are highlighted and must be edited or confirmed.
- Access follows existing ERP roles; the system can only read invoices and draft entries.
- Every suggestion and every human change is logged for audit.
- Weekly sample of 25 posted invoices re-checked for the first two months.
5. Recommendation and sign-off
Go live for header fields on all invoices and line items on digital (non-scanned) invoices only. Re-run this evaluation after any model or ERP change.
Signed off by: Finance lead ______________ AP lead ______________ Date ________