AI knowledge assistant due diligence: a privacy-first audit
Audit an AI knowledge assistant before uploading business data, using a practical suitability scorecard, vendor evidence checks and clear stop conditions.
Stanic & Co.
Perspectives for business leaders
Before uploading business data, audit whether your chosen AI knowledge assistant is suitable for one defined task. Establish why AI is necessary, who can access inputs and outputs, how human oversight will work, and what happens when answers are wrong. Obtain evidence rather than relying on a product label. The OAIC recommends due diligence covering intended use, testing, privacy and security risks, access to personal information and human oversight. The audit below turns that guidance into an illustrative business approval process—not a prescribed legal checklist.
This is a product-selection decision, not a document-cleaning exercise. A carefully filtered collection can still be unsuitable for a product whose access, retention or output controls you cannot establish. The practical question is: should this particular assistant receive this particular information for this particular business purpose?
AI knowledge assistant due diligence starts with a bounded use
Write the proposed use in one sentence before reviewing features: “Our service team will use an internal assistant to find approved product-support instructions and draft answers for staff verification.” Name the users, information categories, intended outputs and decisions the assistant must not make. “Help everyone find everything” is too broad for an initial approval because it leaves the data boundary and consequences unclear.
Then document the problem with the current process. Are staff unable to locate instructions, or are the instructions themselves incomplete? Would better document ownership, search or navigation solve it? Replacing a manual process is not, by itself, evidence that AI is necessary. Record why generated answers add something useful beyond retrieving the original document.
The OAIC’s guidance on commercially available AI products says organisations should consider whether using personal information in an AI system is necessary and the best solution in the circumstances. It also recommends a privacy-by-design approach, including a Privacy Impact Assessment. Treat the audit as an input to that assessment, not a replacement for it.
Keep this review distinct from preparing the content. Once the product and purpose have been approved, use the separate personal-information filtering workflow to decide what should actually enter the collection. Product suitability and data minimisation are separate decisions; passing either one does not settle the other.
Do not proceed where nobody can define the intended use or explain why the proposed information is needed. For questions involving customer records, sensitive information or significant decisions about people, ask a qualified privacy adviser to assess applicable obligations and the proposed handling. This article provides an operational process, not a conclusion that your business or use case complies with Australian privacy law.
Ask for evidence about the complete information path
Create a short evidence file for the exact product, subscription and configuration under consideration. Include relevant contract terms, written supplier responses, configuration records and a diagram showing where information goes. Mark each answer as confirmed, conditional or unknown. A salesperson’s assurance that a product is “private” does not answer who can access uploaded documents, questions, generated answers or support records.
Ask separately about uploaded files, extracted content, prompts, outputs, logs and any retained conversation history. Who can access each category? Where is it processed? How long is it retained? Can it be deleted, and what exceptions apply? Does the supplier use any of it for model improvement, and under what terms? Do not assume either that every public product trains on customer inputs or that a paid subscription prevents every secondary use.
Request evidence for your actual use, not just a general security statement. Can ordinary staff see material outside their approved collection? What happens when access is withdrawn? Can a support operator inspect a conversation? What happens to a shared answer after the original document is removed? These are proposed audit questions; the supplied evidence does not establish any particular vendor’s capabilities.
The OAIC’s product-selection guidance specifically calls for considering who can access personal information entered or generated through the product. The NCSC’s secure AI development guidelines place security across design, development, deployment, operation and maintenance. Together, these support looking beyond the upload screen.
Use synthetic documents and fictional identities for initial demonstrations. Ask the supplier to show the controls rather than uploading real customer records to discover how they work. Where a critical answer remains unknown, record that uncertainty as a failed approval condition—not as an assumption to resolve after launch.
Copy this AI suitability and privacy scorecard
The following scorecard is a proposed Stanic & Co. decision aid. Its scores and thresholds are illustrative, not requirements prescribed by the sources. Adapt them to your business and the consequences of error. Copy this header into your approval record: “Purpose: __; users: __; permitted data: __; prohibited data: __; product/configuration: __; accountable owner: __; evidence date: __; next review: __.”
Score each criterion from 0 to 2: 0 means absent, unknown or unsuitable; 1 means partially evidenced with an unresolved limitation; 2 means documented and demonstrated for the intended use. Attach evidence and an owner to every entry. A feature name alone earns no score.
-
Necessity: Is there a defined problem that a simpler tool cannot adequately address? Example: approved support instructions take too long to locate; a search-only trial will provide the comparison.
-
Input/output control: Can the business establish access, retention, deletion and permitted supplier uses across inputs and outputs? Example: written terms and configuration evidence cover both uploaded files and chat history.
-
Human-in-the-loop: Is there a workable verification and escalation process before consequential use? Example: staff must compare a proposed customer answer with the approved source and escalate missing evidence.
-
Accuracy monitoring: Is there a repeatable test and error-reporting process? Example: a test pack includes answerable questions, missing information and contradictory documents; unsupported answers are recorded as failures.
-
Transparency: Are users told what the assistant does and how its limitations and information handling are communicated? Example: internal instructions explain approved uses; any customer-facing interface clearly identifies itself as AI.
Illustrative decision rule: a total below 8 out of 10 means stop and redesign. A total of 8 or more permits consideration of a limited pilot, not automatic ingestion approval. Regardless of total, stop if input/output control scores 0, a critical information-handling question is unresolved, or the proposed use lacks an appropriate basis for handling personal information.
For example, scores of 2, 1, 2, 1 and 2 total 8. That still does not justify uploading customer records if the control limitation is unknown conversation retention. Resolve that issue or change the data scope first. Record the final decision as “reject”, “redesign” or “approve bounded pilot”, with conditions, an expiry date and a named signatory. No score guarantees safety, accuracy or compliance.
Define human oversight that staff can actually perform
A “verify” button is not an oversight process. Specify what the reviewer must inspect, what evidence is sufficient and when they must refuse to use an answer. For a support assistant, the reviewer might check that the cited instruction applies to the correct product and situation, that important exceptions are preserved, and that the answer does not introduce unsupported promises.
Set boundaries before the pilot. For example, allow drafting from approved support instructions but prohibit deciding customer eligibility, interpreting an individual’s health information or granting exceptions to policy. These are illustrative boundaries, not a universal list. Select them according to the business task, information sensitivity and consequences of a wrong answer.
Human review has limits. A reviewer without relevant knowledge may not detect a plausible error. A source link does not establish that the answer accurately reflects it. A busy employee may approve without comparing the text. Do not count a review step as an effective control unless the person has access to the evidence, enough time and clear authority to stop the work.
The OAIC guidance warns that AI systems can produce inaccurate or false results. It also explains that inferred or incorrect generated information about an identifiable person can constitute personal information. A proposed answer is therefore not harmless merely because it was generated rather than copied from a record.
Assign a business owner to approve scope and accept residual risk, an information owner to approve the collection, and a technical owner or supplier to demonstrate controls. In a small business, one person may hold several roles, but record each responsibility. Review time and escalation effort belong in the operating budget; if the business cannot support meaningful checking, narrow the task or choose the non-AI alternative.
A hypothetical failure: retention uncertainty becomes a release problem
Consider a fictional Australian service business choosing an assistant for customer-account questions. The owner uploads a customer database during a demonstration. They assume deleting the uploaded file also removes extracted content, saved conversations and generated summaries, but have not checked the supplier’s terms or demonstrated deletion behaviour. This scenario is hypothetical, not a reported incident or a claim about any vendor.
During testing, a staff member asks for an account summary. The assistant produces customer details in an answer that is then shared with a wider internal group. Later, the owner deletes the source file and believes the information is gone. A saved conversation still contains the summary. The failure does not require the supplier to train a public model: unverified retention and sharing controls are enough to create the problem.
The proposed audit would have stopped the initial upload. Input/output control would score 0 because the information path was unknown. The owner could instead demonstrate the task with fictional accounts, obtain retention evidence and decide whether customer-level data was necessary at all. If the intended benefit was finding general service instructions, the database was outside the justified scope.
An illustrative detection test is to place a unique fictional marker in a synthetic record, generate an answer, remove the source and withdraw the test user’s access. Check the application paths the user can still reach, including saved conversations and shared outputs. A clean result only supports the tested conditions; it does not prove deletion from backups or supplier systems. Those questions require separate evidence.
If a comparable real failure occurs, stop further ingestion and affected use, restrict access, preserve the information needed for investigation and ask the supplier to confirm the relevant handling and deletion options. Identify which outputs were shared and who received them. Involve the business’s privacy or legal adviser to assess obligations and response steps. Reopening should require evidence that the failed control has been corrected, not simply deletion of the original file.
Compare the assistant with a living knowledge base
The simpler alternative is a maintained document library with standard search, named content owners and restricted access. Staff find the approved instruction and read it directly rather than receiving a generated synthesis. For this comparison, call it a living knowledge base: a collection whose owners review, update and retire content as part of normal operations.
Use the same representative tasks for both options. Ask staff to find an instruction, identify an exception and handle a question for which no approved answer exists. Record whether they found the correct source, whether the resulting response was supported and how much checking was needed. This is a proposed evaluation protocol, not a published performance benchmark.
The document-library option still needs security, permission management and content maintenance. It is not automatically private or accurate. Its practical advantage in this proposed workflow is avoiding a generated-answer layer: staff can use the original approved text without first checking whether an assistant has changed its meaning. Its disadvantage is that finding and interpreting several documents may take more effort.
The assistant may be worth a bounded pilot when staff genuinely need help combining approved information and can verify the result. Prefer the library where questions are straightforward, the collection is small, or review effort consumes the expected convenience. Also prefer it where the chosen AI product cannot demonstrate acceptable data controls. Do not solve a vendor-evidence gap by asking staff to “be careful”.
The OAIC’s announcement of its AI guidance stresses that AI products should not be used simply because they are available. For a broader technical assessment after this product-suitability decision, use the complementary business knowledge assistant readiness guide. Keep the decisions separate: first establish whether the product is suitable, then whether the proposed implementation is ready.
Make approval conditional and reversible
Finish the audit with a written decision, not a completed scorecard alone. State the approved task, information boundary, user group, configuration and unresolved limitations. For a pilot, identify the fallback process and the person authorised to pause it. The business owner should sign the decision after the information and technical owners provide their evidence.
An illustrative release condition is: all critical supplier questions are resolved, synthetic-data checks pass, reviewers can perform the required verification, and the non-AI fallback remains available. Start with the smallest useful collection. Do not approve broader customer records merely because a demonstration using general instructions worked.
Specify observable stop conditions. Examples include an unauthorised user receiving restricted information, a consequential answer lacking supporting evidence, unexplained changes to retention or access settings, and reviewers routinely bypassing verification. Treat a scope change—such as opening the assistant to customers—as a new decision requiring review, rather than an ordinary feature toggle.
On a stop, disable the affected capability or collection, return users to the approved library, investigate the failure and repeat the relevant checks before release. Keep enough decision and test records to explain what changed. The NCSC guidelines include incident management, logging, monitoring and update management across the system lifecycle; the specific stop conditions here are proposed business procedures.
The OAIC also says product due diligence should not be “set and forget”, recommending ongoing performance review, training and monitoring. The decision-ready outcome is therefore conditional: approve only the purpose and information handling you can justify and evidence today. If the controls remain unclear or the simpler library does the job adequately, do not ingest the data. Revisit the decision when the evidence or business need changes.
Sources
General information only; verify sources before making business decisions.