Building an AI Chatbot That Actually Helps Customers
Executive TL;DR
- Customers do not hate bots for being robotic. They hate bots that cannot do anything, answer confidently when they do not know, and hide the way out. Give the bot real actions behind a policy layer, a visible abstain path, and a one-click handoff that carries the transcript.
- Ground every factual claim in retrieved sources or a live account lookup, and make 'I'm not certain, let me get someone' a first-class, tested outcome. An unanswered question routed in ten seconds beats a confident wrong answer every time.
- Measure resolution, not deflection. A deflection target rewards the bot for trapping people; track resolved-without-reopen, handoff latency, and CSAT on bot-handled conversations — and keep a labelled eval set so every prompt change is regression-tested.
Remember the last time you got pissed off at a support bot? I guess it was not the quality of writing. It was you asking about your order, and it rattled off the shipping policy. Or it provided a tidy, plausible answer that was wrong. Or you typed “agent” four times, and it kept giving help articles.
These are three different engineering failures. None of those are fixed by a better model. The first is a capability problem. A bot that has no access to your systems can only talk about policies. The customer wants their actual order, their actual invoice, their actual refund. The second is a grounding problem: a model prompted with a question without context will generate fluent text regardless, because that’s what it does. A third is a deliberate product decision that somebody made, generally because there was a deflection target associated with the project.
That one at the end needs a name. Deflection—the percentage of conversations that never reach a human—is the metric by which most support-AI projects are judged and it incentivizes the very behavior that customers describe as infuriating. A bot that is brilliant at deflection and terrible at everything that matters.
So before you choose a tool, decide what you are building. Something that answers questions, or something that resolves themjudged,.
Scripted flows, retrieval-grounded answers, an agent with tools, or a vendor platform — which support bot should you build?
| Scripted decision tree | Retrieval-grounded answers | Agentic bot with tools | Vendor platform | |
|---|---|---|---|---|
| Cost to launch | Low — days of configuration | Moderate — content pipeline, embeddings, retrieval tuning | Highest — tool integrations, a policy layer and testing per action | Low upfront, per-resolution or per-seat pricing that scales with volume |
| What it can actually resolve | Only the paths someone predicted in advance | Anything your documentation already explains | Account-specific work: order status, address changes, refunds within policy, plan upgrades | Whatever its integrations cover, which is often a lot out of the box |
| Wrong-answer risk | Near zero — it only says what you wrote | Low if answers cite sources and abstain when retrieval is weak; high if the model answers ungrounded | Highest consequence — a wrong action costs money, not just goodwill; mitigate with policy limits and confirmation steps | Varies, and you often cannot inspect why it answered as it did |
| Ops burden | Grows ugly — every edge case is another branch to maintain | Moderate — the real work is keeping the help content accurate | High — integrations, permissions, audit logs, action monitoring | Low day to day; you inherit their incidents and their release schedule |
| Lock-in | Low | Low — your content and embeddings stay yours | Medium — the policy layer and tool schemas are yours, the orchestration may not be | High — conversations, tuning and workflows live in their system |
| Who maintains it | Support ops, until the tree sprawls | Support content owners, which is a healthy incentive | Engineering, permanently | The vendor, with you tuning at the edges |
| Fails when | Customers phrase things their own way, which is always | The question is about this customer's data rather than general policy | You ship actions before you have an audit trail and spend limits | Your support is your differentiator, or your systems are unusual |
When to pick each option
If you have a mainstream stack, moderate volume, and support is a cost center (not your product), buy a vendor platform. You’ll be live in weeks with something that addresses the common cases.” Negotiate on two things people forget: an export path for your conversation history and transparency into why an answer was given. Price per resolution looks attractive till volume grows; model at 3x today's traffic before signing.
Build retrieval-grounded answers when your docs are good and most tickets are explanatory—how does billing work, what is the refund window, and how do I configure X? This is the best cost-to-value ratio on the list, and it has a useful side effect: it surfaces which help articles are missing or wrong, because the bot fails loudly where your content is thin. From day one, there are 2 non-negotiables. Every factual claim cites the source it came from. When retrieval returns nothing above the confidence threshold, the bot says so and offers to connect with a human.
Build an agentic bot with tools for when your top ticket drivers are account-specific—where is my order, change my address, cancel this subscription, resend the invoice. This is where real resolution lives. And real risk lives. So the actions sit behind a policy layer that your support leadership owns. Which actions are allowed, up to what value, for which customer tiers, and which require explicit confirmation. All actions are logged, rate-limited, and reversible where possible. Start with two or three read-write actions to handle the largest ticket categories instead of wiring up everything.
A scripted tree is only appropriate for narrow, high-stakes flows where variance is unacceptable—identity verification, regulated disclosures, and payment collectionslice and. It is a legitimate tool for a small slice, and a trap as a whole strategy.
Rendering diagram…
// The whole difference between a helpful bot and an infuriating one is what
// happens when it does not know. Make that path explicit and testable.
const MIN_SCORE = 0.72; // tune against your eval set, not by feel
export async function handleMessage(msg: Message, ctx: Ctx) {
// 1. Frustration is a routing signal, not a prompting problem.
if (ctx.turnsOnSameIssue >= 2 || detectFrustration(msg.text) || asksForHuman(msg.text)) {
return escalate(ctx, { reason: 'customer_signal' });
}
// 2. Account questions go to tools, not to the model's imagination.
// The policy layer decides what this customer may do, before any call runs.
const intent = await classify(msg.text);
if (intent.kind === 'account_task') {
const decision = policy.evaluate(intent.action, ctx.customer);
if (!decision.allowed) return escalate(ctx, { reason: 'policy_block', action: intent.action });
if (decision.needsConfirmation) return askToConfirm(intent, ctx);
const result = await tools.run(intent.action, ctx.customer.id); // logged, rate-limited
return reply(renderResult(result), { toolUsed: intent.action });
}
// 3. Factual answers come from retrieved sources — or they do not come at all.
const docs = await retrieve(msg.text, { topK: 5 });
const grounded = docs.filter((d) => d.score >= MIN_SCORE);
if (grounded.length === 0) {
// The most valuable three lines in the system. Saying "I don't know, here is
// a person" in ten seconds beats a fluent wrong answer every single time.
return escalate(ctx, { reason: 'no_grounding', query: msg.text });
}
const answer = await model.answer({
question: msg.text,
context: grounded,
instruction: 'Answer only from the provided context. If it does not contain the answer, say so.',
});
if (answer.refused) return escalate(ctx, { reason: 'model_abstained' });
return reply(answer.text, {
citations: grounded.map((d) => d.url), // the customer can check you
escapeHatch: true, // on every turn, not just failures
});
}
// Log reason codes on every escalation. 'no_grounding' spikes tell you which help
// articles to write; 'policy_block' spikes tell you which action to automate next.
// The escalation log is your product roadmap.MVP development planner
Scope, budget, and launch plan
Choose the product, platforms, and modules. The range, phases, and team update as you go. It is a planning figure, not a quote.
Newsletter
One architecture teardown or evaluation, every other week
Practical write-ups on framework choices, migrations, and scaling decisions - written for people who have to defend the call in a meeting.
Newsletter
Get the 2026 Tech Stack Guide
Join the KarmaKoders newsletter for architecture notes, stack evaluations, and build playbooks.