Skip to content
AI & Automation13 min read

AI Knowledge Base: How to Build an AI Assistant That Uses Company Documents

How to build an AI knowledge base assistant: sources, permissions, retrieval, cited answers, refusals, feedback, ownership, rollout and privacy considerations.

01

Quick answer

An AI knowledge base is a retrieval-augmented assistant over your company's content. Choose authoritative sources, sync them with their permissions, parse and index them for hybrid retrieval, and have the assistant answer only from retrieved sources with citations, saying clearly when it cannot find an answer and handing off to people. The part most teams underestimate is the feedback loop: owners must see unanswered and poorly rated questions and fix the underlying content, or quality decays.

Used by employees, this is usually called an internal AI assistant. The sections below on architecture, permission filtering, retrieval quality, monitoring and rollout cover what it takes to connect one safely to company data.

02

Where This Fits

Technical foundations are in the RAG guide and organizational architecture in enterprise RAG architecture. Customer-facing use overlaps with AI customer support automation.

03

Internal vs Customer-Facing Knowledge Assistants

Internal assistantCustomer-facing assistant
SourcesPolicies, processes, tickets, wikisPublic docs and help content only
PermissionsPer user and groupPublic, or per customer account
Tone and riskPractical, can link internal toolsBrand voice, careful on commitments
EscalationTo a team channel or ownerTo support agents with context
MeasureTime saved, adoptionResolution, deflection, satisfaction
04

Choosing and Preparing Sources

Quality in, quality out. Start with sources that are authoritative and maintained. Remove or label outdated versions, resolve conflicting documents and assign an owner to each content area. Resolved support tickets can be valuable but need cleaning, because they contain one-off answers and personal data. See chunking strategies for preparing documents.

05

Answers, Citations and Refusals

Every answer should cite the passages it used, linking to the source document and section, so users can verify. When retrieval finds nothing relevant, the assistant should say so and suggest where to go next, not guess. For policy questions, quote the policy text rather than paraphrasing loosely. Show the document's date when freshness matters.

Citations are the visible part of a fuller record; data provenance for AI covers what to store behind each answer.

06

The Feedback Loop

  • Thumbs up and down with an optional comment on every answer
  • Reports of questions with no good sources, grouped by topic
  • Owner dashboards per content area
  • A simple process to fix or add documents and re-index
  • Regression tests on questions that were fixed

Want an assistant your team can actually rely on?

ZSpace Labs builds knowledge assistants with permission-aware retrieval, citations and the feedback tools content owners need.

Start a Project
07

Interface and Experience

Put the assistant where people already work: the intranet, the help desk tool, chat platforms or your product. Show sources clearly, allow follow-up questions, let users open the source document in one click and make escalation obvious. For internal use, sign-in should be single sign-on so permissions apply automatically.

08

Security and Privacy

Enforce source permissions at retrieval, exclude highly sensitive repositories unless there is a clear need, log queries and sources for audit, and check where model providers process data. Retrieved documents may contain text that tries to manipulate the model, so a knowledge assistant should not hold powerful tools; see prompt injection prevention.

Redaction, retention and provider terms for personal data are covered in AI data privacy.

09

Measuring Success

MetricWhat it shows
Answer rateShare of questions answered with sources
Rating and correction ratePerceived and actual quality
Citation accuracy (sampled)Whether sources support answers
EscalationsWhere people still need help
Gaps fixed per monthHealth of the feedback loop
Time saved or tickets deflectedBusiness value
10

Advantages and Limitations

A knowledge assistant makes scattered information usable, shortens onboarding and reduces repeated questions to experts. Its limits are the content itself: missing, conflicting or outdated documents produce weak answers no matter how good the model is. It also needs ongoing ownership, which is an organizational commitment rather than a one-off project.

11

How to Build It Step by Step

  • 1. Pick a domain and audience (for example HR policies for employees)
  • 2. Collect real questions and expected answers
  • 3. Select and clean sources, assigning owners
  • 4. Build the RAG pipeline with permissions and hybrid retrieval
  • 5. Design answers with citations and refusals
  • 6. Evaluate on the question set
  • 7. Launch to a pilot group with feedback
  • 8. Expand domains once the loop is working
12

Tools and Platform Options

OptionFitsTrade-offs
AI features in your workplace suite or wikiContent already in one platformLimited cross-source coverage and tuning
Help desk AI for customer answersSupport content in a help deskTied to that platform's content
Enterprise search and RAG platformsMany sources, internal useLicence cost, connector coverage
Custom RAG applicationSpecific sources, UX or integration needsBuild and maintenance effort
13

A Content Governance Model

Treat the knowledge base as a product with editors. Assign an owner per content area, set review dates on documents, mark one source as authoritative when documents overlap, archive superseded versions so they leave the index, and use the assistant's gap reports in regular content reviews. Track content freshness as a metric alongside answer quality. Without this, even a well-engineered assistant degrades as documents drift out of date.

14

Internal AI Assistant vs General Chatbot

An internal AI assistant is an assistant for employees that answers from your organisation's own systems, applies each employee's existing access rights and shows where every answer came from. A general chatbot answers from what its model learned in training, plus whatever a user pastes in. The difference matters most when an answer has to be correct for your company, today.

A general chatbot is still useful for drafting, summarising and brainstorming. The problem starts when staff use it for questions it cannot see the answer to, such as 'what is our travel policy for contractors?', or paste confidential documents into it to get one.

DimensionInternal AI assistantGeneral chatbot
GroundingAnswers from retrieved company documents and recordsAnswers from training data and pasted text
PermissionsFilters sources by the signed-in user's accessNo knowledge of your access rules
CitationsLinks each answer to the source passageUsually none, or public web links
Data freshnessAs fresh as the last sync of each sourceFixed at the model's training cut-off unless it browses
Main risksLeaking restricted content if permissions are wrong; stale answers if sync lagsConfident answers that contradict your policies; staff pasting confidential data
Who maintains itContent owners plus a platform teamThe vendor; you control only usage policy
15

Architecture for an Internal AI Assistant Connected to Company Data

The architecture has one rule that shapes everything else: identity flows through every step, and retrieval happens before the model sees any text. The model never receives a passage the user could not open in the source system.

How the parts fit. The identity provider (single sign-on) tells the assistant who is asking and which groups they belong to. The retrieval layer runs hybrid search over an index in which every chunk carries the access list copied from its source, filters results to that user, then reranks the survivors. The model writes an answer only from those passages and cites them. Every interaction is logged for feedback and audit. For the organisation-wide version of this design, see enterprise RAG architecture; for connecting AI to ERP, CRM and other line-of-business systems, see enterprise AI integration.

Reference architecture: internal AI assistant
Employee (intranet, Teams, Slack, browser)
        |
Identity provider (SSO: user ID + groups)
        |
Assistant service (session, guardrails, audit log)
        |
Retrieval layer
  - hybrid search (keyword + vector)
  - permission filter: user ID + groups
  - rerank top candidates
        |
Connected sources, each synced with its ACLs
  SharePoint | Google Drive | Confluence
  Ticketing system | CRM (live lookup)
        |
Language model (answers only from passages)
        |
Answer + citations (links to source sections)
        |
Feedback and logging
  ratings, no-answer events, sources used,
  permission tests -> content owner dashboards
16

Identity and Document Permissions

Answer first: an internal AI assistant must only retrieve what the person asking could open in the source system. This is usually called security trimming or permission filtering, and it has to happen at retrieval time, before any text reaches the model. Instructions in a prompt ('do not reveal HR files') are not a permission system.

Verified fact. Microsoft's Azure AI Search documentation describes four approaches to document-level access: security filters (generally available), POSIX-like ACL and RBAC scopes, Microsoft Purview sensitivity labels and SharePoint ACLs, the last three in preview when we checked. The documentation calls document-level access essential for RAG applications and agentic systems that ground answers in company data, and describes the security-filter pattern as one that 'trims search results based on a string containing a group or user identity'. Other platforms use different mechanisms, but the principle is the same.

Our recommendation. Do not build one shared index with a super-user service account that can read everything and then rely on the application to hide results. A single bug or a cleverly phrased question then exposes everything that account could read. Instead, store access metadata with every chunk, pass the signed-in user's identity and group memberships on every query and filter inside the search engine. If the assistant can also act (create a ticket, update a record), its tools should run with the user's own permissions, not a privileged account; see AI agent access control.

  • Sign-in through your identity provider; no anonymous or shared accounts
  • Access lists stored with every chunk, copied from the source system
  • Group memberships resolved at query time or synced on a short schedule
  • Permission changes and deletions propagated at least as fast as content edits
  • Highly sensitive repositories (HR cases, legal matters, board papers) excluded until there is a clear need and owner sign-off
  • Automated tests in which users with different roles ask questions they must not get answers to
17

Retrieval Quality and Data Freshness

Most wrong answers trace back to retrieval rather than the model: the right passage was never found, or an older version ranked higher.

Chunking starting points (verified). OpenAI's file search tool defaults to chunks of 800 tokens with 400 tokens of overlap. Microsoft recommends starting Azure AI Search at 512 tokens with 25% overlap, which is 128 tokens. Treat these as starting points to test, not answers: policies with numbered clauses, tables and long manuals usually need structure-aware chunking. See RAG chunking strategies.

Hybrid search and reranking. Keyword search finds exact terms such as policy numbers, product codes and names; vector search finds paraphrases. Microsoft's hybrid search documentation notes that keyword search is better for 'product codes, highly specialized jargon, dates, and people's names', and merges both result lists with Reciprocal Rank Fusion. A reranker then reorders the top candidates before they reach the model.

Contextual retrieval. In Anthropic's published tests, adding a short context prefix to each chunk before indexing (contextual embeddings plus contextual BM25) reduced the top-20 retrieval failure rate by 49%, and by 67% when reranking was added. Those results come from Anthropic's own datasets; measure on your own question set before assuming similar gains.

Data freshness. Agree a freshness target for each source with its owner, and treat deletions and permission changes as first-class sync events, not only new content. Show the document date in answers where it matters. The table below gives illustrative starting targets, not standards. More detail is in data freshness for AI.

SourceIllustrative sync targetOwnerWatch for
Policies and handbooksOn publish, or dailyPolicy or HR ownerSuperseded versions left in the index
Wiki and Confluence pagesHourly to dailySpace ownersPages nobody has reviewed in a year
SharePoint and Google DriveChange feed, or hourlyDepartment leadsPermissions changing without re-sync
Ticketing systemHourly, resolved tickets onlySupport leadPersonal data in ticket text
CRM recordsLive lookup through the API rather than indexingSales operationsRecord-level access rules
18

Monitoring an Internal AI Assistant

Ratings alone do not tell you whether answers are right; people rarely rate answers they cannot check. Combine user feedback with sampled review and automated tests. For each interaction, log the question, user, sources retrieved, sources cited, model and prompt version and response time. Treat those logs as sensitive, because they contain questions and document excerpts, and set a retention period. Tooling for traces and dashboards is covered in LLM observability.

SignalHow to measure itWhat it tells you
Answer qualityWeekly sample of answers graded against their sources by a reviewerDrift that ratings miss
Citation accuracyShare of sampled answers whose cited passage supports the claimWhether citations can be trusted
No-answer rateShare of questions where the assistant declinesRising: content gaps. Near zero: it may be guessing
Permission-violation testsScheduled questions from restricted test users; the expected result is zero leaksWhether security trimming still works after changes
UsageActive and repeat users, questions per team, top topicsAdoption, and where content matters most
Latency and costResponse time and tokens per answerWhether retrieval or prompts need tuning
19

Rollout Plan for an Internal AI Assistant

Our recommendation. Start narrow, with one domain, one audience and content owners who have agreed to fix what the assistant exposes. Build an evaluation set from real questions with expected answers and sources before the pilot, and expand only when quality and permission tests pass. Phases and exit criteria below are a planning template; set the thresholds with your own stakeholders.

For a UAE organisation, add bilingual Arabic and English evaluation questions, data residency checks and PDPL questions to phase 0; see AI knowledge base for UAE businesses.

PhaseScopeExit criteria
0. PrepareOne domain (for example HR or IT help); owners named; sources cleaned; an evaluation set of real questions with expected answers and sourcesOwners sign off sources; evaluation set agreed
1. PilotA small group from the team that asks these questions mostEvaluation results meet your threshold; zero permission-test failures; feedback loop running
2. DepartmentThe whole department, same domainStable quality over several weeks; no-answer topics being fixed
3. ExpandAdd domains one at a time, each with owners and its own evaluation questionsEach new domain passes its tests before launch
4. OperateRegular content reviews, sampled grading, permission tests and model or prompt change reviewsOngoing: quality and freshness tracked monthly
20

Worked Example

An illustrative scenario, not a client case: a healthcare software vendor's support team spends hours finding configuration answers spread across release notes and old tickets. An internal assistant indexes current documentation and curated resolved tickets, cites sources and flags unanswered questions to product owners weekly. New support staff reach confidence faster, and documentation gaps become visible for the first time.

21

Common Mistakes

  • Indexing everything, including outdated drafts
  • No citations, so users cannot verify
  • Guessing instead of refusing
  • No owners for content areas
  • Ignoring permissions

Planning a knowledge assistant for employees or customers?

Talk to ZSpace Labs about AI knowledge base development and assistant interface design.

Start a Project
22

Australian Organizations: Privacy, Automated Decisions and Consumer Law

The architecture is the same everywhere, but Australian organizations should check a few local points before launch.

  • Privacy Act: if your organization is covered, the OAIC says privacy obligations apply to personal information entered into AI systems and to outputs that contain it, and recommends due diligence on AI products. Keep personal information out of the knowledge base unless it is needed and permissioned.
  • Overseas processing: where the model, vector database or logs are hosted outside Australia, APP 8 may apply. Check vendor hosting and sub-processors, including for embeddings and evaluation tools.
  • Automated decisions: from 10 December 2026, privacy policies of covered organizations must describe the kinds of decisions made by computer programs that could significantly affect individuals' rights or interests. An internal assistant that only answers questions is unlikely to be such a decision, but one that decides eligibility or outcomes may be. Get advice.
  • Customer-facing answers: answers given to customers are the business's representations under the Australian Consumer Law, so test answers about refunds, warranties and pricing carefully and hand off to people when unsure. See AI customer service in Australia.
  • Governance: see AI governance for Australian businesses for policies, vendor checks and monitoring.
23

Conclusion

An AI knowledge base is a content program with an AI interface. Get sources, permissions, citations and ownership right, and the assistant improves over time. Related: enterprise RAG, RAG guide and AI customer support automation. For bilingual Arabic and English knowledge bases and UAE data rules, see how to build an AI knowledge base for a UAE business.

FAQ

Common questions.

An assistant that answers questions from a company's own documents and records, using retrieval to find relevant content and a language model to compose answers with citations, while respecting who may see what.

No. A help centre is a set of articles; an AI knowledge base answers questions by retrieving from those articles and other sources, and can serve employees, customers or both.

Authoritative, maintained sources that answer frequent questions: policies, product documentation, process guides and resolved support tickets. Exclude drafts, duplicates and outdated material.

Ground answers in retrieved sources, require citations, instruct it to say when sources do not contain the answer, evaluate faithfulness and give users a way to report wrong answers.

Sync access controls from source systems and filter retrieval by the user's identity, so the assistant never answers from content the user cannot open.

Content owners for each area, supported by reports of unanswered and poorly rated questions. The assistant's quality depends on the documents behind it.

Yes, with public content only, strong refusal behaviour, clear hand-off to people and testing for sensitive topics.

Answer rate, citation accuracy, user ratings, escalations, time saved for staff, ticket deflection for customer use and the number of content gaps fixed.

An internal AI assistant is a tool for employees that answers questions from the organisation's own documents and systems, such as policies, wikis, shared drives and tickets. It retrieves relevant passages under the employee's own access rights, writes an answer only from them and cites the sources, so staff can check the answer and open the original document.

A general chatbot answers from its training data and whatever a user pastes in, with no knowledge of your policies or access rules. An internal AI assistant retrieves from your current company content, filters results by the user's permissions, shows citations and says when it cannot find an answer. That makes it suitable for company-specific questions where a general chatbot would guess.

Whether the Privacy Act covers them and what personal information the assistant can reach, where the model, vector store and logs are hosted (APP 8), whether any answers feed decisions that significantly affect people (automated decision transparency obligations start on 10 December 2026), and, for customer-facing assistants, that answers about consumer rights are accurate. OAIC guidance on commercially available AI products is a useful starting point.

Get in touch

Have a project in mind?

Whether you're building a new digital product, improving an existing website, or looking to automate part of your business — let's talk.