The most common mistake: one giant document that mixes everything together
Many companies start their knowledge base by copying the entire internal manual, or a twenty-page PDF with every policy jammed together. For a human reading top to bottom, that works. For an information-retrieval system, it's almost the worst thing you can do: if the answer to "what's the return deadline?" is buried on page 14 of a document that also talks about shipping, warranties and payments, the system has to guess where to cut, and it frequently pulls in too much or too little context.
One article, one topic
The rule that works best is simple: each article should answer a single question or cover a single topic, in a self-contained way. "Return policy" is one article. "How to change your payment method" is another. This isn't just tidiness — it's what lets the search system (a hybrid search combining semantic similarity via embeddings with keyword search) pull exactly the fragment that answers the question, without noise from adjacent topics.
Tags and categories aren't decorative metadata
When every article has properly assigned category and tags, two things improve: search becomes more precise, and your team can audit knowledge base coverage at a glance — seeing that "Billing" has eight articles but "International shipping" has none is exactly the kind of gap that later translates directly into questions the assistant can't answer well.
Public vs. internal content
A well-built knowledge base distinguishes between public articles, which can also be shown on a self-service help center in addition to feeding the assistant, and internal articles, which only the assistant and your team see — escalation procedures, limits on what can be promised without authorization, or sensitive information you don't want a customer to read directly even though the assistant needs it to answer well. Treating everything as public (or everything as internal) forces you to choose between exposing information that shouldn't be exposed or depriving the assistant of context.
Maintenance: a knowledge base is never "finished," it's tended to
An assistant trained on a price list from six months ago is going to answer poorly even if the rest of the system is perfect. The discipline that separates a knowledge base that works from one that degrades is periodically reviewing which questions the assistant couldn't answer well, and updating or creating articles accordingly, instead of letting the gap accumulate.
Write for a machine and a human at the same time
The best article is one a customer would understand if they read it directly on a help center: clear language, no internal jargon, with the answer in the first few lines instead of at the end. If you write thinking only "the AI will interpret it the same either way," you end up with ambiguous articles that neither the system nor a human customer can fully make use of.