Self-Hosted vs Hosted AI Chatbot: What the Build Actually Costs

"We'll just build it ourselves, the API is only a few cents per call."

The API is only a few cents per call. That is true, and it is the least significant number in the decision. This post lays out what the build actually contains, so the comparison is between real alternatives rather than between a subscription price and an optimistic weekend.

The part that is genuinely easy

Getting a model to answer a question about your documents is a weekend. Chunk the docs, embed them, store the vectors, embed the query, retrieve the nearest chunks, put them in the prompt, stream the answer. The libraries are good and the tutorials are accurate.

If someone tells you this part is hard, they are selling something. It is not.

The problem is that this is roughly 15% of a product.

The other 85%

Retrieval quality. The demo works because you tested the questions you already knew the answer to. Real users ask things phrased differently, ask about things not in the docs, and ask two questions at once. Tuning chunk size, overlap, and the relevance threshold is empirical work with no shortcut, and it is where most of the actual quality lives. Expect to iterate on this for weeks after launch, not before.

Knowing when to refuse. A bot that confidently invents a refund policy is worse than no bot. Getting reliable refusal means a similarity threshold you tuned against real misses, and it will be wrong in both directions until you have real traffic to tune against.

Ingestion. PDFs with two columns. Scanned PDFs with no text layer. DOCX with tracked changes. Tables that lose all meaning when flattened to text. Every one of these is a small project, and the last 10% of formats takes as long as the first 90%.

Re-indexing. Your docs change. Something has to notice and re-embed only what changed, without taking the bot down.

Cost control. This is the one people skip and it is the one that bites. Without a hard spend ceiling per tenant, one enthusiastic user or one retry loop turns a $30 month into a $3,000 month. You need a per-org cap enforced before the call, not a billing alert after it.

Abuse. Your endpoint calls a paid API. It will be found. Rate limiting per IP is the floor, not the ceiling.

Prompt injection. Covered in its own post, because it does not have a clean fix and it needs architectural decisions rather than a prompt tweak.

The widget. An embeddable script that works on any customer site without breaking their CSS, without being broken by their CSS, and without tripping their Content Security Policy. Then mobile. Then accessibility. Then keyboard navigation.

The boring rest. Conversation history, human handoff, an admin view for what people asked and what failed, analytics, GDPR deletion, a DPA if you are storing anyone else's documents.

Honest numbers

Inference genuinely is cheap. A grounded support answer from a modern small model costs a fraction of a cent, and a 20-message conversation lands well under a cent. At 10,000 conversations a month you might spend a few tens of dollars on model calls.

So the API cost argument is correct. It is just answering the wrong question.

The real comparison, at 10,000 conversations a month:

Infrastructure is not the expensive part. Your time is. Even at a conservative rate, four weeks of engineering is worth several years of a $99/month subscription, and the subscription does not accrue maintenance.

When self-hosting is genuinely right

This is not a one-sided argument. Build it yourself when:

• Data cannot leave your infrastructure. Regulated industries, or a customer contract that forbids it. No amount of vendor SOC 2 fixes this. It is the strongest reason and it is common.
• The retrieval is your product. If search quality over your corpus is the thing customers pay you for, you cannot outsource it.
• You need something no vendor does. Genuinely unusual ranking, a proprietary ontology, a domain-specific reranker.
• The volume is enormous. At millions of conversations, per-unit vendor pricing eventually loses to owning the stack. That crossover is far higher than most teams estimate.
• You are learning deliberately. A completely valid reason. Just budget it as learning, not as delivery.

If none of those apply, you are choosing to maintain infrastructure that is not your business.

A middle option people forget

Self-hosting a product someone else maintains is a real third choice. You get data residency and control, without owning retrieval quality or the widget. Several vendors ship a self-hosted deployment mode, and this is often the actual answer for teams whose constraint is data location rather than customisation.

Ask about it before assuming the choice is binary.

How to decide in an afternoon

1. Write down your ten most common support questions.
2. Try a hosted product on your real docs. An hour, most of it spent waiting for indexing.
3. Ask those ten questions.
4. If the answers are good, the build is not the interesting problem and you should buy.
5. If the answers are bad, find out why first. It is almost always the documents, not the model, and no amount of building fixes stale docs.

Point 5 is the one worth internalising. Teams that build because the hosted trial gave bad answers usually rebuild the same bad answers on their own infrastructure, having spent six weeks to get there.

Full disclosure: we build Askably, a hosted document-grounded widget with a self-hosted deployment mode, so we sell both sides of this. Its comparison against the hosted alternatives is worth reading next to this page rather than instead of it. The list of what the 85% contains is drawn from having had to build all of it.

If you built rather than bought: what took longest? My guess is document ingestion, but I would like to be wrong.