Back to the reading room
Business insights

Business knowledge assistants: a practical RAG readiness guide

Prepare a business knowledge assistant with a practical RAG test pack covering source freshness, permissions, citations and safe handling of missing answers.

Stanic & Co.

Perspectives for business leaders

8 min read

Building a business knowledge assistant is a significant step for any Australian organisation looking to improve productivity. Many teams assume that to get an AI to understand their internal processes, they must train or fine-tune a model. For questions that depend on changing internal documents, retrieval-augmented generation (RAG) is one useful pattern to evaluate. Search, curated FAQs or a document-navigation tool may be simpler and sufficient. Understanding the difference between training and retrieval is the first step toward building a system that your team can actually trust.

Understanding the RAG difference

A general-purpose language model does not automatically have access to your current internal documents. Without a permitted source, it may produce an unsupported answer or decline to answer. Fine-tuning changes model behaviour using examples; it is not a reliable substitute for retrieving frequently changing facts, and its cost and duration depend on the model and task. Retrieval and fine-tuning can be complementary rather than mutually exclusive.

Retrieval-Augmented Generation is different. Instead of forcing the model to memorise your data, RAG gives the model an open-book exam. When a user asks a question, the system first searches your internal documents to find the relevant information. It then hands that information to the AI along with the user's question, effectively saying, "Using these specific documents, answer the user's question." The model does not need to be retrained; it simply reads the context you provide in real-time.

The ingestion pipeline

Before your assistant can answer questions, it needs to be able to read your data. This is the ingestion phase. You cannot simply dump a folder of thousands of PDFs into a system and expect perfect results. The data must be prepared, which usually involves cleaning, formatting, and breaking documents into smaller, manageable pieces called chunks.

If you feed a 200-page policy manual into a system without chunking, the AI will struggle to find the specific paragraph relevant to a user's query. By breaking documents into logical sections—such as individual policy clauses or project milestones—the system can retrieve only the exact information needed. This can improve retrieval relevance, but chunk boundaries and the eventual answer still need testing. A source excerpt can omit an exception or lose the context of a larger policy.

Document ownership and source freshness

One of the most common reasons RAG systems fail is "data rot." If your knowledge base contains three different versions of an expense policy, the AI might retrieve the outdated one. You must establish clear ownership for every document in your repository. If a document does not have a clear owner responsible for updating or archiving it, it should not be in the assistant's knowledge base.

Freshness is equally critical. Your system should be designed to automatically sync with your primary document stores, such as SharePoint, Google Drive, or internal wikis. Set an explicit freshness requirement: for example, a tested update interval and a maximum tolerated lag. The right interval depends on the consequences of stale information. Monitor successful updates, deletions and permission changes; do not promise immediate synchronisation unless the connector actually provides it. If the system relies on a static snapshot of your data from six months ago, it will quickly become a liability rather than an asset.

Access controls and security

In an enterprise environment, not everyone should have access to every document. A junior staff member might be able to see general office policies, but they should not have access to sensitive payroll data or confidential client contracts. Your RAG system must respect your existing access controls.

When the system retrieves information, it should only search documents that the specific user is authorised to see. This is often handled by mapping the user's identity to the document permissions. If you cannot guarantee that the system will filter results based on user permissions, you risk exposing sensitive information to the wrong people. Never treat the AI as a separate security layer; it must inherit the security posture of your existing document management systems.

Citations and abstention

Trust is the currency of any internal tool. If an AI provides an answer, it must be able to prove where that information came from. A robust RAG system should always provide citations, linking back to the specific page or paragraph in your source documents. This allows the user to verify the information themselves.

Equally important is the ability for the AI to say, "I don't know." If a user asks a question and the system cannot find a relevant answer in the provided documents, it should be configured to abstain from answering rather than making something up. Encouraging the AI to admit its limitations is a feature, not a bug. Abstention reduces one source of unsupported answers, but a prompt alone cannot guarantee it. Test refusal behaviour and review it alongside retrieval and citation accuracy.

A hypothetical example

Consider a hypothetical Australian logistics firm, "FastTrack Freight." They decide to build a RAG-based assistant to help their customer service team answer shipping queries.

  1. The team identifies their source data: the current "Standard Operating Procedures" (SOP) PDF and the "Client Service Level Agreement" (SLA) database.
  2. They ingest these documents, ensuring that the SOP is tagged with the "Operations Manager" as the owner.
  3. A customer asks, "What is the maximum weight for a standard pallet?"
  4. The system retrieves the specific paragraph from the SOP regarding pallet dimensions and weight limits.
  5. The AI generates a response: "According to the current SOP (Section 4.2), the maximum weight for a standard pallet is 1,200kg."
  6. The response includes a clickable link to the original PDF, allowing the agent to verify the source.

If the customer asks about a competitor's pricing, the system searches the documents, finds no mention of it, and correctly responds: "I do not have information regarding competitor pricing in my current knowledge base."

Practical preparation sequence

Before you write a single line of code or purchase a platform, follow this sequence to ensure your organisation is ready:

  1. Audit your data: Identify which documents are accurate, current, and necessary. Delete or archive everything else.
  2. Define ownership: Assign a human owner to every document category. If a document is not worth maintaining, it is not worth including.
  3. Map permissions: Document who needs access to what. Ensure your IT team can integrate these permissions into the AI retrieval layer.
  4. Establish a feedback loop: Create a simple way for users to flag incorrect or outdated answers. This is the most effective way to improve the system over time.
  5. Start small: Build a pilot for a single department with a well-defined set of documents before rolling it out to the entire company.

Evaluation and continuous improvement

How do you know if your assistant is actually working? You cannot rely on "vibes" or casual testing. You need a formal evaluation process. Create a set of "golden questions"—a list of 50 common queries with known, correct answers. Every time you update your system, run these questions through it to ensure the accuracy of the responses has not degraded.

Monitor the system for "hallucination rates" and "retrieval failures." If the system is consistently failing to find the right documents, you may need to improve your chunking strategy or refine the metadata attached to your files. Treat the assistant as a living product that requires ongoing maintenance, not a "set and forget" software installation.

A practical evaluation pack, not just a list of questions

Microsoft's RAG documentation describes query understanding, information access, token limits and security as distinct implementation challenges. Its Azure-specific features should not be assumed to exist in another product. Your test pack should therefore separate finding the right evidence from writing a plausible answer.

Build five categories: answerable questions; questions with no answer in the source set; conflicting or outdated documents; permission-restricted questions; and adversarial instructions embedded in documents. For every test, record the authorised user, expected source, expected behaviour and unacceptable outcome. A request about payroll by someone without payroll access is a permission test, not simply an accuracy test.

Consider a hypothetical leave-policy question. The correct answer may depend on role, location and effective date. Record which policy version should be retrieved and which exceptions must appear. If the assistant cites the right document but omits an important exception, mark the answer wrong. A clickable citation proves a reference exists, not that the sentence follows from it.

For a stale-document test, withdraw an old policy and check that both its content and its cached extracts disappear within the agreed interval. For a permissions test, change a user's access and test retrieval, previews, citations and cached answers. Document security must be enforced by the application and retrieval system, not merely by instructions asking the model to keep secrets.

Put a release boundary around the assistant

Start with one bounded source collection and a named document owner. Specify the support route for unanswered questions, the update procedure and the circumstances that disable the assistant. A material permission leak is a stop condition even if the assistant answers most routine questions correctly. A source outage should produce a clear unavailable response, not a confident answer from stale memory.

Track retrieval failures, unsupported statements, stale answers and user-reported defects separately. A single satisfaction score can hide an unacceptable security failure. Record model, retrieval settings and source versions so that a later change can be investigated. Keep logs proportionate and protect them: queries and retrieved extracts may themselves contain personal or confidential information. The OAIC's commercial-AI guidance is a useful starting point for privacy assessment, not a certificate of compliance.

Before expanding the document set, repeat the evaluation pack and check that the extra information improves the intended task. More indexed documents can increase ambiguity and maintenance work. If users mainly need to find a policy rather than have it rewritten, compare the assistant with a conventional search experience. Plan staff verification habits alongside the technology using our AI team training guide.

Sources

General information only; verify sources before making business decisions.

Turn an idea into a next step.

A conversation about what matters to your business.

Let's talk