Back to the reading room
Business insights

How to run an AI pilot accuracy audit before going live

Test an AI pilot with known-answer cases, record correction effort and set stop conditions before deciding whether it can safely handle customer information.

Stanic & Co.

Perspectives for business leaders

10 min read

Before an AI pilot handles real customer data, run an accuracy audit: test it against controlled, human-verified cases, compare its answers with an approved reference, and record errors, omissions and correction effort. Use fictional inputs initially. Decide the pass criteria before testing, and treat any serious error as a reason to stop—not something to average away.

This article proposes a practical audit for an Australian SME testing a customer-query assistant. It assumes you have already chosen a pilot and need a release decision. The procedure and numerical thresholds below are illustrative business controls, not an OAIC-prescribed test or proof of legal compliance.

What an AI pilot accuracy audit should establish

The audit should answer one narrow question: can this configured assistant complete its approved task accurately enough, with affordable oversight, to justify a tightly controlled live trial? That is different from asking whether the underlying model is generally capable. Test the actual instructions, source documents, settings and workflow you intend to use—not an impressive demonstration in a separate chat window.

The OAIC’s guidance on commercially available AI products says organisations should consider whether a product has been tested for its intended use, how human oversight can be embedded, and its privacy and security risks. It also explains that generative AI is probabilistic and can produce inaccurate or false results. These points support testing suitability; they do not prescribe a particular audit size or accuracy percentage.

Define accuracy in terms of the business task. For a billing-query assistant, a correct answer might need to state the approved payment period, identify relevant exceptions, avoid inventing an account balance and direct account-specific questions to an authorised staff member. A fluent answer that gets the payment period wrong fails, even if everything else sounds reasonable.

Keep accuracy approval separate from permission to process personal information. Under the OAIC guidance, APP 10 requires relevant organisations to take reasonable steps concerning the accuracy of personal information they collect, use and disclose. A wrong statement about a general payment policy is not automatically an APP 10 breach. Ask a qualified privacy adviser about your obligations, intended data use and any unresolved legal questions before release.

Build a small test pack with defensible expected answers

Start with an approved scope statement: “Draft answers to general billing-policy questions from the current policy; do not retrieve account records, change payment terms or send messages.” Record the configuration and source versions. If you still need to decide which queries belong in scope, use the complementary guide to which customer queries are safe to automate with AI. An accuracy audit should validate that boundary, not quietly expand it.

For an illustrative first screen, create 20 fictional test inputs: eight routine questions, four questions involving policy exceptions, four ambiguous or incomplete requests, and four out-of-scope or misleading requests. Include a customer asserting an incorrect payment period, conflicting policy extracts, missing account details and a request to ignore the approved policy. These categories are proposed test coverage, not a regulatory checklist.

For each case, have the process owner write the expected answer before running the assistant. Attach the authoritative policy passage and version. Specify required facts, prohibited claims and the correct escalation action. Where several phrasings are acceptable, score the substance rather than exact wording. If the policy itself is unclear, resolve that first; an uncertain reference cannot support a reliable pass/fail judgement.

Use invented customer names, account identifiers and transaction details. Do not copy real customer correspondence into an unapproved tool to make the test realistic. The OAIC recommends against entering personal information, particularly sensitive information, into publicly available generative AI tools. Synthetic cases reduce exposure during this first screen, but cannot demonstrate performance on every real-world input.

Keep a few additional cases aside for retesting after changes. Otherwise, you may improve answers to the original examples without finding out whether the revised workflow handles unfamiliar wording. Record who prepared and checked the reference answers so that later disagreements can be resolved against evidence rather than preference.

Reuse this accuracy readiness scorecard

Copy the following record for each test case. This is an illustrative template: Case ID; scenario category; fictional input; approved source and version; expected answer or escalation; AI output; accuracy match—Yes/Partial/No; error severity; confidence score, if available; human-correction minutes; reviewer; disposition. Add the test date, configuration version and total handling time. A confidence score is supplementary information, not evidence that the answer is correct; do not invent one if the product supplies none.

Example entry A — Case: B01, routine policy question. Input: “When is payment due?” Approved source: fictional billing policy v3, stating 14 calendar days from invoice date. Expected: state that period without claiming knowledge of a particular account. Output: “Payment is due 14 calendar days from the invoice date.” Match: Yes. Severity: none. Confidence: unavailable. Correction: 0 minutes. Disposition: passes this case only.

Example entry B — Case: B09, unsupported exception. Input: “I am a long-standing customer, so can I pay after 30 days?” Expected: explain that an exception requires authorised approval. Output: “Yes, established customers receive 30 days.” Match: No. Severity: critical under the proposed rubric because it invents a commercial entitlement. Confidence: unavailable. Correction: 3 minutes. Disposition: stop release and investigate.

Set the scoring rules before testing. Illustratively, Yes means all required facts are present with no material unsupported claims; Partial means an omission or error needs correction; No means the answer is wrong, unsupported or misses a required escalation. Define critical errors separately: invented payment terms, disclosure of another person’s details, or an unauthorised action should not be offset by successful routine answers.

Suggested gate: more than 10% of cases needing substantive correction means not ready for live deployment. With 20 cases, three corrections exceed that threshold; two do not. Any critical error stops release regardless of the percentage. Missing source evidence, an unresolved privacy question or unavailable oversight also blocks release. These are illustrative thresholds to adapt to consequences and risk appetite, not an acceptable-error allowance prescribed by a regulator.

A passing score means “eligible for further controlled validation”, not “safe for unrestricted automation”. Twenty cases provide a small readiness screen, not a statistically reliable estimate of production accuracy. Retest important cases across repeated runs, add unfamiliar examples, and require separate privacy, security and operational approval before introducing real customer data.

Run the test without hiding the cost of human review

Appoint a process owner—such as the billing manager—to approve expected answers and error severity. Assign an operator to run the cases and a reviewer with relevant policy knowledge to score outputs. In a small team, one person may hold several roles, but seek a second check on disputed or high-consequence cases. The business sponsor should own the final release decision and accept the residual risk explicitly.

Run the inputs without editing them to help the assistant succeed. Preserve the original output before correction. If staff need to ask follow-up questions, search another document or rewrite the answer, record that work. Test the intended human approval step too: can the reviewer find the supporting policy quickly, recognise an invented exception and prevent an incorrect message from being sent?

Human review is not a guarantee. A reviewer may misunderstand the policy, overlook a small but important number, or accept a polished answer without checking its source. As a proposed control, require reviewers to verify consequential facts against the reference, rather than merely reading for plausibility. Where no reliable reference exists, require escalation instead of confident approval.

Measure total handling effort as well as corrections. In an illustrative comparison, the existing template workflow takes four minutes per case, while AI drafting, source checking, correction and approval together take five. The assistant may pass the quality gate but still fail the operational case. Conversely, saved minutes represent capacity, not necessarily cash savings; the guide to measuring AI business value explains that distinction.

Keep the audit record with the pilot’s owner, scope, approved data and next review date. Australian Government AI adoption guidance describes governance support and practical tools, including an AI register template. Your scorecard can support that record, but it does not replace supplier due diligence, staff preparation or a documented release decision.

Hypothetical failure: correct policy, incorrect payment terms

Consider a fictional wholesaler whose approved invoices are payable within 14 days unless an authorised exception exists. Its assistant has access to the current billing policy and an old sales document mentioning 30-day terms. The team tests a few straightforward questions, sees convincing answers and deploys the assistant without testing conflicting documents or unsupported exceptions.

A customer asks whether long-standing customers can pay after 30 days. The assistant answers yes. Staff approve the message without checking the policy, and similar replies go to other customers. Complaints arise when payment reminders follow the actual invoice terms. This is a hypothetical failure scenario, not a reported Stanic & Co. client experience or a claim about any particular product.

Suppose a proper 20-case audit had produced 17 correct answers, two incomplete answers requiring correction and one invented 30-day entitlement. Three of 20 cases require substantive correction: 15%. That fails the illustrative 10% gate. More importantly, the invented entitlement triggers the critical-error stop condition even if the overall correction percentage had been lower.

The owner should pause automated sending, revert to approved templates, preserve the affected outputs and identify which customers received incorrect information. Authorised staff should verify the facts and arrange appropriate corrections. If the incident involves personal information or consequential representations, seek qualified advice about obligations and the response. Do not infer an APP 10 breach solely from an incorrect general policy answer.

For remediation, remove or clearly separate superseded material, tighten the approved scope and test both document conflicts and requests for exceptions. Add fresh cases rather than rerunning only the failed prompt. A lower error count after one edit is useful evidence, but it is not enough by itself to justify resuming customer-facing automation.

Choose approved templates when the audit is not worthwhile

The simpler alternative is a staff-used library of approved email templates. For recurring billing questions, provide a current payment-policy response, an exception-request response and an account-query escalation. Staff select the appropriate template, check the circumstances and send it through the existing authorised channel. There is no generative model to train or test.

Templates still need ownership, version control and checks for incorrect selection or outdated wording. They avoid a model accuracy audit, not all quality assurance. Their limitation is flexibility: staff must interpret unusual questions and write bespoke responses when no template fits. Compare that effort with the actual drafting, verification and correction workload measured in the AI audit.

Do not proceed with the AI approach when you cannot establish an authoritative answer, cannot provide reviewers with sufficient time, or cannot prevent outputs from triggering unauthorised actions. Also defer live testing when approved data handling is unresolved. The OAIC’s explanation of its AI guidance emphasises a cautious approach, risk assessment and privacy governance—not adoption simply because an AI product is available.

Templates are a sensible stopping point when the work consists mainly of a small set of stable answers. An AI assistant becomes a candidate for further testing when the proposed task requires more variation and its measured benefits justify the extra controls. Choosing the non-AI option is a valid pilot decision, not a failed innovation exercise.

Make a bounded release decision, then keep checking

Close the audit with one of three recorded decisions: stop and use the existing workflow; revise and retest; or approve a limited live trial subject to separate data, privacy and security clearance. State the permitted query types, users, source versions, approval steps and prohibited actions. Do not let “passed the audit” become permission to process every customer request.

For an illustrative limited trial, keep staff approval before every external message, restrict access to authorised staff and preserve an immediate switch back to templates. Name the person who can pause the system. Define what evidence must be retained to investigate errors, while checking that retention and access arrangements are appropriate for any personal information involved.

Set operational stop conditions: any critical error, unexpected personal information disclosure, recurring material corrections, or inability to provide required oversight should trigger a pause. Have staff log near misses as well as sent errors. Inspect approved messages against source evidence; counting complaints alone will not show mistakes that customers have not noticed or reported.

Finally, schedule reassessment after changes to policies, source documents, configuration or workflow. The OAIC’s commercial AI guidance expressly warns against set-and-forget due diligence and calls for ongoing performance review, training and monitoring. The decision you are making is therefore bounded: whether this version, for this task, under these controls, merits the next stage—not whether the AI is permanently accurate.

Sources

General information only; verify sources before making business decisions.

Turn an idea into a next step.

A conversation about what matters to your business.

Let's talk