Rebuilding Boardy's context layer
Making an agent cacheable
Over the summer I rebuilt the context layer of Boardy with one goal: get the provider's prompt cache to actually hit. Cost per run dropped 30% at constant model prices. Uncached tokens per run fell by over 90%. The agent now serves about 92% of its input tokens from cache, under 1.5% of model calls start cold, and two runs on the same conversation inside the cache window reuse a median 99% of the previous request.
Prompt caching only works when the start of your request is byte-identical from one run to the next, and most agent architectures never give it that. Ours certainly didn't. Here's what it took.
How the cache works
A prompt cache is a prefix match. The provider reads your request from the front, and a later request is served from cache for as long as its bytes match an earlier one. The first byte that differs ends the match. Everything after that point is processed and billed at full price.
The part most agent codebases aren't built around is the ordering. A request serializes as the tool definitions, then the system prompt, then the messages. So if anything varies in the tool definitions, you lose the whole request. If anything varies in the system prompt, you lose the whole conversation behind it. And if anything varies at the end of the messages, the next run can't pick up where this one stopped.
If your agent renders its prompt from a template, picks its tool list based on who it's talking to, and rebuilds the conversation history every turn, you have variation in all three places. That was us. Caching had been "enabled" for a long time before anyone measured it, and when we did, almost nothing was being reused from one run to the next.
Making it smaller doesn't help
When a prompt is too big, the obvious move is to shrink it, and we spent weeks on exactly that. We moved content out into skills. We found rules that were stated in three places and kept one. It was good for maintainability. It did almost nothing for the cache, because a smaller prompt that still varies per user is still a prompt that varies per user.
I considered deferring tools too. The idea is that you only send the handful the model uses all the time, and it asks for anything else by name when a conversation calls for it. I didn’t end up doing it. Our tool list isn’t big enough for the savings to matter yet, and it wouldn’t have fixed the actual problem anyway, which was that the list changed from one user to the next.
Why the tool list was varying in the first place
So why wasn't it constant? Because we were removing tools the current user couldn't use yet. If someone hadn't finished a setup step, the tools that depended on that step were left out of their request. That sounds like a sensible way to keep the prompt small, but look at what it's really doing. It saves a few tokens, and it quietly tells the model something about the user through what's missing. And since the tool doesn't exist from the model's point of view, the model can't tell the person why it can't help them. It just can't.
So for each of those tools we asked: if it stayed in the list and simply refused when called, could the refusal say anything useful?
Most of the time, yes. Take scheduling. If someone hasn't connected a calendar, the scheduling tool used to be missing from their request entirely. Now it's there for everyone, and when it runs for that person it comes back with "this user hasn't connected a calendar", and the model tells them that and offers the link. That's a better outcome than the tool silently not existing. The reason someone can't do something is usually something they can fix, so the tool stays visible and refuses with the reason. The only things worth hiding are the ones where the reason isn't actionable and the person should never hear it anyway.
Almost everything landed on the visible side. The tool list stopped depending on the individual user, the checks that used to decide whether a tool was present moved to the moment the tool is called, and as a side effect the agent got more honest. It can say why it can't do something now.
We then did the same thing to everything else that had been conditionally present in the prompt. What the model can see is the same for everyone. Whether a thing applies to this person gets decided when the model tries to use it, with a message that says what to do instead.
Where the state goes instead
If neither the system prompt nor the tool list can carry user state, it has to live in the messages. And for caching to work, the messages have to be append-only. Here's the shape we ended up with.
When a conversation session opens, we compute everything the model needs to know about the person once, render it into one user context block, store it, and send those exact bytes at the top of every run in that session. It never gets updated. If something about the person changes mid-session, the model hears about it as a new event in the conversation, not as an edit to the block at the top. It's stale by design, and the checks at execution time cover anything the model gets wrong because of that.
After that block comes the conversation as it actually happened: inbound messages, the agent's replies, its tool calls and their results, in order, then whatever woke this run. Nothing gets rewritten, so a later run's request starts with an earlier run's request.
Nothing goes after the history either. If the model needs to see something this turn, we store it as an event in the conversation so the next run replays it the same way.
Then the prompt rewrite
The prompt rewrite was the thing I wanted to do first. It only became possible last, because every conditional in it was there to carry a fact that now had somewhere better to live.
The template went from a program with branches and variables to a document with almost none, byte-identical for everyone. A few rules we wrote it under: tool schemas own mechanics and the prompt owns judgment, so anything about how to fill in a parameter lives on that parameter's description. Every rule has exactly one home. And no worked tool-call transcripts in the prompt, because the model copies the nearest literal example rather than obeying the most emphatic sentence.
One warning. When you move content out of the prompt, you find out which of it was load-bearing. Something we'd been treating as optional turned out to be driving a core decision, and the quality of that decision dropped when it stopped being in front of the model every turn. Run a real A/B against the old prompt, and when something regresses, fix it with a general principle instead of a special case.
The two things I'd keep
Caching is an architecture property. You don't turn it on; you design the request so this run's bytes are a prefix of the next run's, and you write tests that fail when that stops being true.
And presence shouldn't carry information. If the model could act on the reason, keep the capability visible and refuse at execution time. If it couldn't, hide it.
The cost drop showed up in the bill within a day. The bigger win was a context layer we can actually hold in our heads, and a prompt that reads like a document instead of a program.