The question that determines everything
Before any technical decision, you need to answer one question clearly: what specific problem is this chatbot solving?
The most common chatbot project failure is trying to answer that question with "everything" — a bot that answers FAQ questions, helps users navigate the website, qualifies sales leads, books meetings, and resolves support tickets. Chatbots with too broad a scope are built at too high a cost and do too many things poorly instead of one thing well.
Start with the single highest-value problem. Build a bot that solves that problem reliably. Then expand scope once you have proven the core value.
The content problem: GIGO applies
AI chatbots are only as good as the content they are trained on. If your FAQ is incomplete, out-of-date, or written in internal jargon, your chatbot will give incomplete, out-of-date, or confusing answers. The most technically sophisticated chatbot in the world cannot compensate for poor source material.
Before you commission a chatbot build, audit your content. Which questions do users most commonly ask? Are those questions answered clearly and accurately in your documentation? Where are the gaps? This content audit is genuinely the most valuable work in any chatbot project — and it is something your team is better placed to do than any developer.
What to evaluate before choosing a vendor
How do they handle wrong answers?
Hallucination — the tendency of LLMs to generate confident-sounding but incorrect answers — is the biggest trust risk for production chatbots. Ask any prospective vendor how they measure answer accuracy and what happens when the bot does not know the answer. The right answer is: low-confidence responses should be flagged and escalated to a human, not guessed at.
Can you see what the bot used to form its answer?
Source citations are non-negotiable for any customer-facing chatbot. If a user asks about your warranty terms and the bot answers, they should be able to see which document that answer came from. This builds trust and makes it easy to identify and fix incorrect answers.
Who owns the conversation data?
Conversation logs are valuable — they tell you what your customers are asking, where your content has gaps, and which questions the bot is struggling with. Make sure you own this data and have full access to it, not just aggregate metrics.
How does it handle updates?
Your product changes, your policies change, your pricing changes. A chatbot that requires a developer to update when documentation changes is a liability. Ask how content updates flow through to the bot — the answer should be "immediately" or "within minutes", not "we will schedule a maintenance window".
Setting realistic expectations
A well-built knowledge-base chatbot can deflect 40–60% of tier-1 support queries. It can answer questions at 3am without staffing costs. It can maintain consistent, policy-accurate responses without the variability of human agents.
It will not replace your support team. It will not handle complex, emotionally charged interactions. It will occasionally make mistakes — the goal is to make those mistakes rare, visible, and recoverable.
The teams that get the most value from chatbots are the ones that treat them as tools that free up their humans to handle the interesting, high-value work — not as replacements for the people who understand the nuance.