Designing a Coding Agent Without a KV Cache

A design note from September 2026 gathers six coding-agent complaints under one sentence. Suppose a language model had no KV cache. What would these agents look like then. The answer contains no stronger model. It contains one thing, someone deciding what goes in front of the model on every turn.

Two minutes and change, English voice, captions burned into the picture and switchable in the player.

1. One counterfactual, six symptoms#

A coding agent today is a while loop, a model, and a handful of tools. The thing that decides how useful it feels sits outside that loop, in the text the loop hands to the model each time it turns.

The KV cache is why that text is an append-only transcript. Reusing a cached prefix is nearly free. Change any byte near the front and everything after it is reprocessed. The economics never made it into a design document. It is written in the bill, and it shapes the skeleton of every mainstream agent.

Imagine the cache away and two things happen at once. State can be explicit, so context gets assembled instead of accumulated. Six long-standing complaints turn out to share one floor.

The note leaves a seat for the alternative and calls it Jev. It does not write code. It sits beside the model as a decision layer. The harness hands it the current application state, meaning the goal, the context, the rules, the available actions and the previous actions, along with a predefined question. Jev returns a typed answer with a probability attached. A choice and a score are the common shapes, and the note adds a third, the noul decision. Typed output is what makes the answer checkable. The harness can validate it, apply thresholds and branch on it without parsing prose.

The note compresses the design into one table, six high-frequency decision points with one typed answer each.

All six are asked every turn, thousands of times in a session. The note puts the gain here, outside the model.

2. The route that saves money costs more#

The intuitive plan gives planning to the frontier model, hands execution to a cheaper one, and brings the frontier model back to review. The note prices it with list rates, Opus at 5 dollars per million input tokens and 25 for output, Sonnet at 3 and 15. Let X be context tokens, Y generated output tokens, and Z tokens read during the work, such as command output and file contents.

Plug in a plausible session shape, X at 0.65, Y at 0.12, Z at 0.23, and pure Opus comes to 4.15 while the routed path comes to 6.19. The route meant to save money costs 49 percent more.

The gap comes from two places. Going down, the cheaper model has to load the context first. Coming back, the frontier model has to reread what the helper produced. Whenever the session is long, X is large, whenever the work reads far more than it writes, Z towers over Y, or both, the arithmetic never recovers.

The arithmetic condemns the unit of pricing. The right unit is a context rebuild. Routing becomes viable on two conditions. The harness can hand the cheaper model a small purpose-built context, and the return trip does not force the frontier model to reread everything.

3. The money goes to reading, not writing#

Before redesigning anything it helps to see where a session spends its budget. The note gives an estimate by subtask, weighted to the input side, so a file reread many times is counted each time.

The striking row sits near the bottom. Writing code is the reason a coding agent exists, and it is the smallest line item in the ledger.

Outside evidence points the same way. Microsoft’s fastcontext project reports that on GPT-5.4 trajectories, reading and searching account for 56.2 percent of all tool-use turns and 46.5 percent of the main agent’s tokens, as reported by the project and not independently reproduced here. The two sets of numbers count different things and describe one finding.

If either set generalizes, the largest efficiency gain in a coding agent comes from smarter retrieval. A better model does not produce it, and neither does a better diff format.

4. The visibility ladder#

The central proposal turns context from a static noun into a per-turn action. For every user query the harness asks Jev two things. Whether the previous context is still worth keeping, so reuse the cache or rebuild from scratch on cost and quality. And how to build the next context so it holds everything relevant and nothing else.

The simplest form gives every chunk a noul. Every tool input, every tool output, every piece of internal reasoning, and every exchange with the user passes through it. A later version replaces the score with a level. The ladder runs from hide to short, then long, then full.

The same 2,400-line grep output is twelve relevant hits for one question and invisible for the next, and it is never deleted from state.

Compression therefore changes position. The old way compresses once, before the question is known. The ladder compresses per query, after it is known. A summary written before the question arrives will reliably throw away something the question needed.

The note adds a visual idea. If the harness can heatmap which lines of a grep output matter for the current question, it can filter that output down to any amount the budget allows. The author adds that it would also look cool.

The same mechanism unblocks sub-agents. Sub-agents are expensive today because deciding what context to pass is expensive. Once that layer assembles itself, sub-agents can run far more often, and they become a control a user can turn, spend more for faster or better results, or run conservatively.

5. Three tiers of tools, condition-bound instructions#

Tools are in an odd spot. Their schemas must be declared in full inside the system message, whether or not this turn needs them. That eats a large share of context, and selection quality does not improve to match. The note’s working hypothesis stacks two causes, a high cardinality of tools and tool usage that sits off the policy the model was trained on.

The fix is tiered disclosure.

A model cannot propose an action it does not know exists, so the first tier has to list every capability cheaply. Its descriptions can be as short as a skill description. The binding requirement is that the third tier disappears from context once it is done.

If that layer holds, the batteries debate from section 1 dissolves. A built-in costs almost nothing until it is used, so an agent can ship with hundreds of tools and thousands of documentation pages. The note adds a commercial note, that near-zero-cost integrations make a strong co-marketing channel.

The same idea extends to instructions. AGENTS.md files load in full today. The note argues for loading sections conditionally. Front-end work pulls in the style guide, work inside one subdirectory pulls in that directory’s list of footguns, and the note suggests every subdirectory carry one. Writing prose pulls in voice samples and a list of things to avoid.

The distinction from skills matters here. A skill tends to mean do this now. A conditional instruction means keep this in mind. Condition-bound instructions also need one property skills lack, immunity to compaction. A skill loaded early in a long session eventually gets summarized out, while an instruction bound to a condition reloads whenever the condition holds.

6. Routing by sensitivity, one shared retrieval pass#

Routing today has two axes, difficulty and cost. The note adds a third, trust. Some open-weight models served through low-cost providers cost far less than frontier APIs, and data passing through those endpoints may not stay private.

The proposal estimates which files a subtask will touch, attaches a policy to each file type, and routes the subtask accordingly.

That last row carries the broader point. A team may avoid one vendor’s models because it researches models itself, or avoid certain providers for safety-sensitive work. Once routing is policy-driven, those preferences become configuration and stop depending on discipline.

The other half is background work. The note sees a common shape across several popular agent workflows. An HTML progress page that updates as work proceeds, an argument that understanding is the new bottleneck, evals generated in the background, an ELI5 skill that explains a system with a few large pictures, and an agent that maintains a small deployed page with screenshots for a phone during a long run.

These tasks share three properties. They run in the background, they only read, and they never contend for a lock. Doing their retrieval once and sharing it is the cheapest saving available, because section 3 put retrieval at the top of the token budget. The explicit split between reads and writes earns its keep here, since read-only tasks never collide.

7. What the design costs#

Gathering six symptoms in one place reads well. Each piece has a price when you build it.

Jev adds a model call per turn, with latency and spend on the hot path. The decision layer can be wrong, and typed output makes an error checkable without making it correct.

Routing by sensitivity rests on an estimate of which files a task will touch. That estimate is itself model output, it can be wrong, and the failure mode is data leaving the boundary. Of all the errors in this design, that one is the hardest to undo.

Tiered disclosure turns “a tool the model does not know about” into a permanent blind spot. A badly written first-tier line means that tool is never selected, and nothing in the system reports the omission.

The numbers need their labels. The 4.15 and 6.19 use list prices and one illustrative session shape, and the author ran no real billing. The token share table is marked illustrative by its own author. The fastcontext percentages are reported by the project.

The counter-example survives too. Routing really does save money on short sessions that read little, which the formula concedes on its own. An agent delivered as a plugin never gets control of context assembly, and this whole design is unavailable to it. That is where the note’s opening assumption comes from, that some capabilities can only be built natively.

8. How to check the arithmetic yourself#

Five things reproduce in a minute or two.

The hands-on part takes ten minutes. Take your last long session, tag its tokens by subtask, and run the section 2 formula on the same tokens for “hand it out and take it back”. If Z is clearly smaller than Y, your session may genuinely suit routing. If Z runs at twice Y, the bill will look like 4.15 against 6.19.

9. Two open questions#

The first one. Deciding relevance costs tokens too. Scoring every chunk, heatmapping a grep output, those actions carry a cost that never enters the 4.15 against 6.19 ledger. At what point the scoring costs more than it saves is a calculation the note does not make, and I will not guess on its behalf.

The second one. If context is rebuilt every turn, how much is the prefix cache still worth. After Jev, the cache turns from a constraint into an option, and the price is a lower hit rate. How much lower depends on the overlap between two rebuilds, and nobody has measured that overlap. It may decide whether this design saves money or loses it on long sessions.

References#

• “Jev Engineering for Coding Agents”, an independent study synthesis, September 2026, based on design notes by Diogo Almeida (TypeSafe). https://github.com/yibie/jev-engineering-zh
• TypeSafe, the Jev decision model. https://typesafe.ai
• Microsoft, the fastcontext project. https://github.com/microsoft/fastcontext
• Zhang, A. “Recursive Language Models”, 2025.
• Tools named in the note, headroom, rtk, ast-grep, ast-outline, fastcontext, fff. https://github.com