One prompt, every customer

One customer was costing me twenty dollars a day. The fix wasn't a cheaper model, or a shorter prompt, or fewer features. It was noticing that about eighty-five percent of what I send is the same for everybody — and that I'd been paying to send it separately, over and over, to each of them.

Where the money actually was

I pulled apart a single eighty-cent conversation turn, line by line, expecting to find the work.

The work was pennies. The turn was dominated by one line: writing the context — the roughly fifty thousand tokens Dustav reads before it says anything — into cache. Writing to cache costs double the normal rate. Reading from it costs a tenth. Same bytes, twenty-fold spread, depending entirely on whether you're the one paying to put them there.

I'd written about this before, at the single-user scale: the read is never the villain, the re-write is. What I hadn't done was ask the obvious follow-up question. Not "how do I write less often?" but "why am I writing this per customer at all?"

The thing I'd been staring past

Of that fifty thousand tokens: who Dustav is, how it works, its safety floor, every playbook for every capability, and the definition of every tool it can call.

None of that is about your business. It is byte-for-byte identical for every single customer.

The part that's actually yours — your facts, your folders, the people you deal with — is the tail. Call it fifteen percent. I had been treating the whole fifty thousand as one indivisible blob and paying to write it fresh for each business, when eighty-five percent of it was the same blob every time.

Why one key changes everything

Here's the mechanism, and it's the part that made this go from "nice optimization" to "different order of magnitude."

The cache is matched on a prefix, from the first byte forward, and it's scoped to the account — not to the user, not to the conversation. Two requests that begin with the same bytes hit the same cache entry, even if they're for different customers.

Which was useless to me for the entire life of the project, because every customer brought their own API key. Their own key meant their own account meant their own private cache, and each one paid, separately, to fill it with the same eighty-five percent.

Then I changed how the product is billed — everyone moved onto one metered key, and I bill usage instead. I made that call for commercial reasons. The engineering consequence I hadn't thought through: all of my customers now share one cache. If their requests start with the same forty-eight thousand tokens, they are all reading one entry.

So: put that shared block first, prime it centrally once, ping it occasionally to keep it warm, and every customer reads at a tenth of the price instead of writing at double. The keeping-warm costs about a dollar a day, platform-wide, forever, regardless of how many customers there are.

An eighty-cent conversation turn became about ten cents. Reading an email went from eighteen cents to one. The first real turn after it went live went from 77.8 cents to 11.2.

Order is the whole trick

You don't get any of this for free, and the reason is that "prefix" is literal. The match runs from byte zero and stops at the first difference.

So a context assembled in the order that made sense to a human — your business up top, because it's the important part — shares nothing, ever, with anyone. One customer's name in the first paragraph and the other eighty-five percent behind it might as well not exist.

Most of the actual work was reordering: universal first, per-business tail last, and a clean seam between them. The savings are entirely a property of the byte order.

The failure mode has no error

This is the part I'd tell anyone attempting the same thing, because it's genuinely nasty.

If one byte of per-customer data leaks into the universal span — a name, a date, a count of open items, anything — nothing breaks. There's no exception. No warning. No wrong answer. The model behaves identically and the output is correct.

You just silently stop sharing, and quietly pay full price forever, and the only way you'd ever know is to go read a token breakdown and notice a number that should be near zero isn't.

So the guard is a test, and it's the single most valuable thing in this whole change: assemble the universal span for two different customers and assert the two are byte-identical. Not "similar." Identical. It's the only way this design can be defended against a well-meaning future edit that drops something useful and personal into a shared paragraph.

The phantom

The bug worth telling, because it cost me an afternoon and it's a general trap.

I built the warmer, deployed it, opened the app, sent a message — and the first turn paid full price. The warmer ran fine. The cache entry existed. The turn just didn't read it.

The warmer was priming a prefix that nothing sends. A real conversation turn attaches a few small internal tools that the warmer wasn't attaching — about four hundred tokens' worth — and tool definitions sit before everything else in the cache key. Four hundred tokens in the wrong place isn't a partial miss, because prefix matching has no such thing. It's a total miss. I had been carefully keeping warm an entry no real request would ever match.

The rule I took from it: a warmer has to reproduce the exact bytes on the wire, not approximately the same content. If it constructs its request through a different code path than the real thing, it will drift, and the drift is invisible — you'll see a healthy cache and a full-price bill and no connection between them.

The part I want to sit with

Every optimization I attempted inside the old billing model was arithmetic on one customer at a time. Shorter prompts, fewer tools, cheaper models for the small jobs. All real, all incremental, all capped by the fact that each customer was an island.

The twenty-fold win wasn't an engineering idea at all. It came from a commercial decision — taking the API key away from the customer — and it was sitting there the whole time on the other side of a choice I thought was purely about pricing.

Worth remembering the next time an efficiency ceiling feels like physics. Sometimes the constraint isn't in the code. It's in the shape of the business, and moving it changes what the code is even allowed to do.

The one-sentence version

Eighty-five percent of what I say to the model is the same for every customer — and the day they all started reading it out of the same cache instead of each paying to write their own copy, the cost of a conversation fell by a factor of eight and stopped scaling with the number of people using it.