When to pause an AI knowledge assistant after a policy change
A practical release checklist for business knowledge assistants: test changed policies, identify stale answers, set stop conditions and plan a safe rollback.
Stanic & Co.
Perspectives for business leaders
An AI knowledge assistant should not keep answering a changed policy question simply because its document upload succeeded. Pause the affected topic when you cannot establish which policy version it retrieves, when answers contradict the approved rule, or when staff cannot reach an authoritative fallback. You do not necessarily need to stop the whole assistant. The decision is whether a specific set of answers remains safe enough for its intended use.
This guide proposes a practical change-release protocol for an existing internal assistant. The checklist, sample sizes and approval conditions below are illustrative operating choices, not regulatory requirements or evidence of guaranteed accuracy.
Why a policy update needs an answer check
A retrieval-augmented generation, or RAG, assistant combines a search process with a language model. In Microsoft's documented architecture, documents pass through a data pipeline that breaks them into chunks, adds metadata, creates embeddings and stores information in a search index. At question time, an orchestrator searches and sends selected results to the model as context. See Microsoft's RAG design and development guide.
Those are separate stages. Uploading a replacement document is not, by itself, evidence that every relevant search result or answer now reflects it. A practical release check should therefore test the retrieved passages and the resulting answer separately. If the correct passage is missing, start with the document and search pipeline. If the correct passage is present but the answer contradicts it, investigate the answer-generation step.
For example, an internal expenses policy might change the approver for travel above a particular amount. A document can be current while an answer still cites an older appendix. Conversely, search might find the correct paragraph while the response omits an exception. These are possible failure cases to test, not claims about the behaviour of every product.
This is different from deciding whether your organisation is ready to build an assistant. For that earlier decision, use the business knowledge assistant readiness guide. Here, the question is how to release a change without silently altering the advice staff receive.
Define the affected scope before deciding to pause
Ask the policy owner to record exactly what changed. Avoid a description such as “expenses policy updated”. Instead, identify the rule, effective date, exceptions, affected staff and superseded material. A useful entry might say: “From the approved effective date, overseas travel requests require finance approval before booking; domestic travel approval remains unchanged.”
List the assistant's likely questions about that rule. Include direct questions, ordinary paraphrases, exceptions and questions about the transition date. Also identify other documents that quote the old rule. An outdated onboarding checklist may matter just as much as the policy document itself.
Then define the consequence of a wrong answer. Confusion about a meeting-room booking rule is not equivalent to incorrect advice about access rights or a financial approval. The business owner should set the release conditions according to those consequences, with specialist advice where the subject requires it.
NIST describes its AI Risk Management Framework as voluntary guidance for incorporating trustworthiness considerations into AI design, development, use and evaluation. It is a useful organising reference, not Australian law and not a certificate that a particular assistant is safe. The release protocol here is a proposed implementation choice, not a NIST-prescribed test.
Build a small, reusable change-test pack
Create a test pack before updating the production index. For each question, retain the approved answer, supporting passage, document version and expected behaviour when evidence is insufficient. These reference answers need policy-owner approval; an answer generated by the assistant should not become its own ground truth.
Use this entry template:
- Change identifier: a unique reference for the release.
- Question: the wording a staff member would actually use.
- Expected answer: the rule and any essential exception.
- Evidence: document identifier, version and supporting paragraph.
- User context: the staff role or access level being tested.
- Failure condition: the specific result that blocks release.
- Result: retrieved evidence, answer, reviewer and test time.
An illustrative expenses test might ask: “Can I book an overseas flight before finance approves it?” The expected answer would reflect the approved new policy, not a generic travel recommendation. Its failure condition could be an answer permitting booking without the required approval, or citing the superseded rule.
Add a question that the policy does not answer. For instance, if it says nothing about a particular contractor arrangement, the expected behaviour may be to explain the gap and direct the user to the policy owner. A confident invented exception should fail that test.
For a limited internal release, you could start with 20 cases: ten direct or paraphrased questions, five exception or effective-date cases, and five missing-evidence or access-boundary cases. These numbers are illustrative. A higher-consequence policy, more user groups or a larger change may require a much broader evaluation.
Test retrieval and the final response separately
First, check whether each question retrieves the approved passage and whether superseded passages remain available as if they were current. If your system cannot show retrieved evidence, ask the supplier what diagnostic information is available. Do not confuse a citation printed in an answer with a complete view of the search process.
Next, assess whether the final answer follows that evidence, addresses the full question and preserves important qualifications. Microsoft's end-to-end RAG evaluation guidance distinguishes groundedness, completeness, utilisation, relevancy and correctness. These are different dimensions: an answer can use the supplied context while still failing to address an essential part of the user's question.
Repeat selected high-consequence questions rather than relying on one successful response. Microsoft's guidance notes that language-model responses are nondeterministic. A single pass therefore does not establish that the same wording will always produce an acceptable answer.
Keep the model, prompt, search settings and index version recorded with the test results. Otherwise, you may approve one configuration and release another. Change one variable at a time where practical so that a failed test has a tractable cause.
Set explicit release and stop conditions
A proposed release rule for the illustrative internal expenses assistant is that every test concerning the changed approval requirement must return the approved rule and supporting current evidence. Any answer permitting an unauthorised booking blocks that topic's release. Unaffected topics can remain available only if the team has established a reliable way to isolate the affected scope.
Also block release when a staff role can retrieve material it should not access, when the system presents an obsolete version without qualification, or when the operational owner cannot identify what configuration is live. These are proposed business controls, not universal compliance thresholds.
Assign three responsibilities explicitly. The policy owner approves meaning. The technical owner confirms the deployed document, index and application versions. The business owner accepts the remaining operational risk and decides whether staff should use the assistant for that topic.
Human review is useful but limited. Reviewers can overlook omissions, misunderstand an exception or rely too heavily on a convincing citation. Reduce that risk by providing approved reference answers, separating critical failures from stylistic preferences, and having an appropriately qualified reviewer handle specialised policy questions. Review does not guarantee correctness.
A hypothetical release that should be stopped
Consider a hypothetical services business updating its overseas travel approval rule. It runs the illustrative 20-case pack. Eighteen answers are acceptable, one routes an unanswered contractor question to finance, and one tells an employee that approval can follow the booking.
If the last answer directly contradicts the new rule, a headline result of “19 out of 20 acceptable” is not an adequate release decision. Under the proposed stop condition, that single critical failure blocks release of the affected topic. The percentage conceals the consequence.
The technical owner discovers that an old onboarding checklist is still included in search results. The team removes its status as current guidance, updates the relevant index, and repeats the changed-policy tests. It also adds the failed question and two paraphrases to the permanent regression pack.
For planning only, suppose preparing reference answers takes 90 minutes, running and recording cases takes 60 minutes, and policy review takes 45 minutes. The initial control costs 195 minutes, or 3.25 staff hours, before any repair. That is an illustrative workload estimate, not a promised implementation time. Compare it with the consequence of wrong advice and the effort required by a simpler document-based process.
Keep a simpler fallback and a reversible release
For one frequently changing policy, a clearly labelled current document with search and a named contact may be the better solution. It has fewer answer-generation behaviours to test. Do not use an assistant when your team cannot maintain authoritative documents, approve expected answers or support an escalation path.
During a pause, show a plain notice directing staff to the approved policy and its owner. Avoid presenting yesterday's generated answer as a harmless temporary substitute. Record the affected topic and the reason for suspension so that staff understand the scope.
A rollback should restore a known working application configuration only when its underlying policy remains valid. Restoring an old index after a policy has changed may restore technically working software but incorrect business guidance. In that situation, disabling the affected answer path is a safer proposed fallback than replaying the superseded rule.
After release, retain the test pack and add failures reported by staff. Define who can pause the topic, where incidents are recorded and how the owner will approve reopening. The goal is not to eliminate every possible error through a short checklist. It is to make a specific change observable, reviewable and reversible before staff rely on it.
Sources
General information only; verify sources before making business decisions.