Back to the reading room
Business insights

Is your off-the-shelf AI pilot secure enough to start?

Use a practical security readiness scorecard to decide whether an off-the-shelf AI pilot can start, needs tighter controls or should stay on hold for now.

Stanic & Co.

Perspectives for business leaders

11 min read

An Australian SME can move quickly with an off-the-shelf AI pilot by narrowing what the tool can access and do before staff begin testing. Make security part of the pilot design: use synthetic data first, disable unnecessary connections, assign an accountable owner and test how to stop the workflow. Do not treat the vendor’s default settings as evidence that your particular use is safe.

The practical question is not whether a product is secure in every situation. It is whether your team can operate a specific, limited trial safely with the controls and people available. The Pilot Security Readiness Scorecard below turns that question into a start, restrict or stop decision.

Off-the-shelf AI pilot security starts with a smaller scope

Buying an existing tool can reduce implementation effort, but it does not settle how your business should configure or use it. Business.gov.au’s AI guidance describes off-the-shelf tools as generally cheaper, quicker and easier to implement than custom solutions. It also advises businesses to understand the data a tool uses, how it is stored, and whether the tool protects business and customer information.

For a first pilot, separate testing usefulness from authorising access to live operations. A tool can demonstrate whether it produces a useful draft without receiving your full customer history, connecting to a shared drive or sending anything externally. This narrower trial gives the owner evidence for the next decision without granting every permission at the start.

Write the scope in one sentence: “Test whether the tool can draft an internal response from approved, non-sensitive reference material; it cannot send messages, search customer records or change business systems.” Then list the permitted users, inputs, connections and outputs. Anything outside that boundary requires a fresh decision.

This is a proposed operating approach, not a prescribed security standard. It applies the principle in the NCSC’s secure AI development guidelines that security should be a core requirement throughout design, development, deployment and operation. The guidelines address providers, including those building on existing tools and services; an SME buyer can use the same lifecycle questions when assessing its supplier and its own configuration.

Do not use a small pilot label to excuse a large exposure. A short trial connected to a restricted document repository is still a consequential access decision. If the intended benefit can only be tested with sensitive records or consequential automated actions, pause for a more detailed assessment rather than calling it a sandbox.

Complete the Pilot Security Readiness Scorecard

Copy the following fields into a document for each proposed pilot. These categories and traffic-light rules are illustrative Stanic & Co. procedures, not ratings prescribed by the sources. Here, a sandbox means a separate test environment using synthetic or approved public information, without production connections. A green result authorises only that bounded test—not live deployment.

Pilot record:

  • Task and intended benefit: what decision will the trial inform?
  • Business owner and technical administrator: who approves scope, and who changes settings?
  • Data sensitivity: Low—synthetic or approved public material; Medium—non-public operational material; High—personal, confidential or commercially sensitive records. If uncertain, classify upwards pending advice.
  • Access level: Public—only approved public/test sources; Internal—selected business material; Restricted—customer, personnel, financial or other tightly controlled systems. Record read, write and send permissions separately.

Continue the record:

  • Human oversight requirement: Full—every output checked before use; Sampled—only selected outputs checked; Automated—outputs acted on without prior review. Record both the oversight needed and the staffing available to deliver it.
  • Vendor data usage policy: record whether input/output use for training is excluded, an opt-out is available, or the position is unknown. Attach the relevant terms and setting evidence; availability of an opt-out is not proof it is enabled.
  • Operating capacity: name the person who can inspect activity, receive reports and disable access.
  • Evidence: record settings, account/product scope, unresolved questions and the decision date.

Green—ready for bounded sandbox testing: all inputs are synthetic or approved public material; no production access or external actions are enabled; vendor handling is sufficiently understood and accepted for those inputs; full review is staffed; and a named administrator has demonstrated the stop procedure. Every condition must pass.

Amber—extra controls required: internal material, broader permissions or sampled review are proposed, but the owner has identified a practical restriction. Do not authorise that broader scope yet. Replace internal data with synthetic examples, remove connections or restore full review, then reassess. Red—stop the current scope: high-sensitivity data lacks approved handling, vendor use is unknown for non-public inputs, required review cannot be staffed, or nobody can reliably disable access. Do not average these blockers into a passing score.

Example entry: “Internal FAQ drafting; Low sensitivity; Public/test access; Full oversight; documented vendor handling; no connectors; operations manager owns trial; administrator confirms account disablement.” This is green only if the evidence supports each statement. Changing the input to customer correspondence makes it a new assessment, not a minor variation.

Check the settings that determine the actual boundary

The scorecard needs evidence, not ticks based on a sales demonstration. Ask the supplier or administrator to show what the specific account and product can access, which features are enabled, where outputs are shared, and what administrators can inspect. Record unavailable controls as limitations rather than assuming they exist in every subscription.

For this proposed pilot procedure, inspect three boundaries before inviting users:

  • Input boundary: what can staff paste, upload or connect?
  • Action boundary: can the tool only draft, or can it send, publish, update or delete?
  • Visibility boundary: who can view uploaded material, generated content, shared conversations and activity records?

Prefer a configuration that prevents out-of-scope actions over a written instruction asking staff not to perform them. For example, if sending messages is outside the trial, do not grant sending permission. If an upload feature cannot be disabled, assess whether training and supervision are enough for the proposed data; if not, choose another tool or a different test.

Treat training use, retention, sharing and deletion as separate questions. A statement about model training does not answer who can access a shared workspace or how long a file remains available. For the training question, use the more detailed vendor data-use checking process. If personal information is proposed later, use the separate pilot deletion verification guide before changing scope.

If terms or settings do not establish an acceptable boundary, keep non-public data out. Where personal information or contractual confidentiality is involved, ask an appropriately qualified adviser to assess the proposed handling and obligations. This article supplies an operational decision process, not a legal conclusion about a particular product or business.

Match human oversight to the people actually available

Name one business owner with authority to approve scope, fund oversight and stop the trial. Name a technical administrator who can remove permissions and disconnect integrations. In a small business, one person may hold both roles, but the record should still distinguish the decisions. Business.gov.au recommends senior accountability for AI safety, human oversight and a clear process to override or stop a tool when something goes seriously wrong.

The daily operator needs a short checklist: confirm the input is permitted, check the output against its source, record corrections and report unexpected behaviour. The owner needs a review point to examine those records and decide whether to continue. The administrator needs a tested route to disable the workflow without waiting for the next management meeting.

Estimate reviewer capacity before choosing the pilot. Illustratively, 40 drafts a day at three minutes each require two hours of checking, excluding interruptions, investigation and rework. If only 30 minutes is available, cap the trial at a manageable volume or choose another task. Do not silently substitute sampled review for the full review your risk assessment requires.

Human review is not a guarantee. A reviewer may lack the source information, subject expertise or time needed to spot a plausible error. Checking the final answer also cannot undo an upload that should never have occurred. Business.gov.au specifically warns that AI can present false or misleading information as fact and recommends checking generated content and decisions before use.

Use full review only where reviewers can meaningfully verify the output. If they cannot, narrow the task to something verifiable or stop. The NIST AI Risk Management Framework supports incorporating trustworthiness into AI design, use and evaluation, but it is voluntary guidance—not a certification that your review process or product is safe.

Hypothetical failure: a safe drafting trial gains unsafe access

Consider a hypothetical Australian service business testing a support-drafting tool. The approved scope uses invented customer questions and public service descriptions. During the trial, a staff member uploads a real customer export to get more realistic summaries. The owner has not established the supplier’s training-use position. Separately, the tool has a knowledge-base publishing feature that was left enabled.

The staff member adds the uploaded material to the knowledge base used by a customer-facing assistant. In this hypothetical configuration, a later test query returns a customer detail. The failure is not that uploading information necessarily makes a public model learn it. Training use, retrieval from an uploaded file and publishing material into a customer-facing knowledge base are different mechanisms. Here, the demonstrated exposure comes from the publishing connection.

The scorecard should have stopped this scope before the upload: the input changed from low to high sensitivity, access extended to restricted records, and an external publication path existed. Unknown vendor handling was an additional blocker, not evidence that training actually occurred. A full review of draft replies would not have prevented the initial upload or publication.

Detect this class of mistake without exposing real people. In the proposed pre-flight test, upload a synthetic record containing a unique invented marker and check whether it appears in unintended workspaces or customer-facing test responses. Inspect available activity records for unexpected uploads, publications and permission changes. These checks can reveal specific boundary failures; a clean result does not prove every possible route is safe.

If a real exposure is suspected, stop the pilot, disable the affected connection or publication path, restrict access and preserve relevant evidence. Establish what was uploaded, where it became available and who may have accessed it. Follow the business’s incident process, involve the supplier and obtain appropriate privacy or legal advice. Do not promise complete reversal merely because a file disappears from the interface; verify the supplier’s retention and deletion position before restarting.

Choose the simpler alternative when controls exceed the benefit

An AI pilot is not the only way to improve an inconsistent process. For repeatable support responses, a human-managed spreadsheet can map enquiry types to approved wording, escalation contacts and source links. A document template can prompt staff to include the right facts without generating new statements. Both can be maintained inside an already approved business environment.

These alternatives avoid sending task data to an additional AI service if staff keep the workflow within that environment. They do not automatically guarantee privacy, security or data residency. Access controls, sharing settings, storage arrangements and staff behaviour still matter. The advantage is a smaller process to understand—not an exemption from information management.

Compare the options against the same task. Record preparation time, handling time, checking effort, correction effort, administration and the consequences of mistakes. For the AI option, include time spent managing accounts, reviewing activity, confirming settings and investigating unexpected behaviour. A fast generated draft is only one part of the operating cost.

Choose the spreadsheet or template when the task mostly involves selecting standard wording, the volume is modest, or the team cannot sustain AI-specific oversight. Do not use the scorecard as a way to justify a tool when the underlying process has no agreed source of truth. Resolve conflicting instructions first; otherwise neither reviewers nor the tool have a reliable basis for producing an answer.

Choose a bounded AI trial when there is a clear question that templates cannot readily answer—for example, whether drafting varied internal explanations reduces total handling effort without increasing correction work. Business.gov.au advises starting with the business problem rather than adopting AI for its own sake. Keep the simpler workflow available as the fallback while collecting evidence.

Approve a test, not an open-ended deployment

Finish with a short authorisation record. The following is an illustrative template: “We approve [task] for [named users] using [permitted inputs], with [connections and actions] disabled. [Owner] reviews results on [date]. [Administrator] can stop access through [tested method]. The trial ends or pauses when [conditions] occur.” Attach the completed scorecard and supporting evidence.

Define stop conditions before the trial begins. Suitable proposed conditions include any non-permitted upload, unexpected external visibility, an unapproved permission change, repeated failure to complete required review, or uncertainty about a changed supplier setting. A stop means return to the approved manual process while the issue is investigated—not continue operating while someone finds time to assess it.

Collect only the evidence needed for the decision: whether the task was completed usefully, total handling and checking effort, significant errors, boundary exceptions and administrative workload. Avoid copying sensitive content into a separate trial log unnecessarily. Record enough context to investigate a problem without creating an uncontrolled second repository.

At the review point, choose one of three outcomes: stop and retain the simpler workflow; continue within the same boundary to resolve a specific unanswered question; or assess an explicitly expanded scope. Do not let a successful synthetic-data trial automatically authorise customer data, write access or autonomous sending. Each changes what can go wrong and what your team must manage.

The decision-ready rule is straightforward: start only when the permitted data, permissions, oversight and stop mechanism fit the team’s actual capacity. Off-the-shelf speed is useful when it shortens setup without hiding operational responsibility. If you cannot demonstrate that boundary, reduce the scope rather than accepting an unmeasured risk to keep the trial moving.

Sources

General information only; verify sources before making business decisions.

Turn an idea into a next step.

A conversation about what matters to your business.

Let's talk