The extraction pipeline chewed through 2,400 documents overnight. Cost: $380. The next morning I diffed the inputs against the previous batch and 87% came back as near-duplicates, differing by trivial whitespace. I had paid $330 to re-derive answers I already had.
That wasn't a cache miss.
My cache had no right to hit in the first place.
A TTL can tell you when something is old. It cannot tell you when something is wrong. And for an AI extraction pipeline, wrong is the only thing that matters.
So I rebuilt the caching layer on a different premise: caching is a statistical validity problem, not an expiry problem. Then I paired it with a second idea, the kind that sounds obvious right up until you have to implement it: reasoning depth is a budget allocation problem, not a model selection problem.
What runs in production now is a two-stage system:
- Confidence-gated cache: per-selector reuse vs partial rebuild using a multi-signal score and conformal thresholds.
- Reasoning budget allocator: per-span compute decisions under a fixed budget using a value-of-insight objective.
Together they cut API costs by 90% and took batch processing from hours to minutes.
The key insight
The obvious way to cache an AI extraction pipeline:
- hash the input
- store the output
- add a TTL
That works for pure functions. Extraction isn't a pure function.
Even with identical text, the "right" output can change because:
- your feature set changes (new fields, different normalization)
- your template changes (versioned prompt / schema)
- your downstream expectations change (what counts as acceptable)
- your similarity assumptions were wrong (two texts look close but differ on a critical constraint)
So instead of asking "is this cached value fresh?" I ask:
"is this cached value still valid for the specific selectors I'm about to use?"
Selectors are the trick. I don't treat the extraction artifact as one blob, I treat it as a set of spans grouped by selectors (field groups). The cache gate returns one of two things:
("reuse", entry.artifact)("rebuild", dirty_spans)
That second path is the whole point: partial rebuilds.
The budget allocator then takes the spans and spends compute only where quality sits below the target.
One gate answers "is reuse statistically justified?" The other answers "if not, what's the cheapest way to fix it?"
How it works
Stage 1. Confidence-gated cache: score similarity like you mean it
I compute a single similarity score s between the new request and the cached metadata. Four separate measurements go into it.
Here's the exact scoring logic I run:
α, β, γ, η = 0.6, 0.3, 0.08, 0.02
s = α * _cosine(np.array(req.get("embed", [])), np.array(meta.get("embed", [])))
s += β * _feature_drift(req.get("fields", {}), meta.get("fields", {}))
s += γ * min(72, (time.time() - meta.get("created_at", time.time())) / 3600.0) / 72
s += η * (0 if req.get("fields", {}).get("template_version")==meta.get("fields", {}).get("template_version") else 1)
return float(s)
Tuning that for the first time was a surprise. The score isn't semantic similarity with a little seasoning on top. It's a weighted argument about why cached output might be invalid.
The four measurements:
- Embedding cosine (weight
α = 0.6) - Feature drift across key fields (weight
β = 0.3) - Age decay capped at 72 hours (weight
γ = 0.08) - Template version mismatch (weight
η = 0.02)
That 72-hour cap matters. I don't want "very old" dominating the score forever. Age is a weak prior, not a verdict.
My one analogy for this whole post: this score is a four-sensor smoke detector. One sensor (embeddings) can be fooled by "similar enough." Another (feature drift) catches the quiet but deadly changes. Age is the battery slowly draining your trust. Template mismatch is the someone-swapped-the-wiring alarm.
Stage 1.5. Conformal prediction: thresholds that come from reality
A fixed threshold is where these systems go to die.
Pick a global constant, ship it, and you'll do one of two things:
- reuse too aggressively and serve stale extractions, or
- rebuild too often and defeat the point of caching
So I compute a conformal threshold tau from calibration history. The gate reflects empirical error rates rather than a hand-tuned constant. The threshold comes from historical scores where realized span error exceeded eps, and I sort those bad scores and pick a quantile controlled by delta.
Here's the exact logic:
over = sorted([s for s,e in calib_scores if e > eps])
if not over: return 1e9
idx = int(max(0,(1-delta)*(len(over)-1)))
return float(over[idx])
Two details I like about this:
- If there are no over-epsilon examples yet, I return
1e9. Deliberately permissive: the system starts out reusing and learns its way into being stricter. deltacontrols which quantile I take, which turns the whole thing into a choice about risk tolerance rather than a guess at a number.
Stage 2. Reuse vs partial rebuild: decide per selector, return dirty spans
Now the part that makes this operationally useful. I don't decide "cache hit" globally. I decide per selector, and I hand back the spans that need work.
s = score(req, entry.meta)
dirty = []
for sel in req.get("touched_selectors", []):
selector_tau = entry.dc.selector_tau.get(sel, entry.tau_delta)
if _worst_probe_delta(entry.probes.get(sel,[])) > eps or s > selector_tau:
dirty.extend(entry.dc.spans.get(sel, []))
if not dirty:
return ("reuse", entry.artifact)
else:
return ("rebuild", dirty)
The non-obvious engineering win is that dirty is a list of spans, not a boolean. Caching stops being a blunt instrument and becomes a scalpel:
- If only one selector looks risky, I rebuild only its spans.
- If everything looks safe, I reuse the full artifact.
Two independent failure modes mark a selector dirty:
_worst_probe_delta(...) > eps(probe-based evidence of staleness)s > selector_tau(similarity score exceeds the selector's conformal threshold)
I want both. Similarity is predictive, probes are forensic.
A side mechanism I still use: adaptive TTL sampling (BDAT)
The conformal gate handles the validity axis, whether reuse is statistically defensible right now. There's a second axis it deliberately ignores: time. BDAT covers that. I maintain TTL parameters per selector and update them based on staleness observations, so the system learns how quickly each selector's reality moves out from under it.
The update logic looks like this:
params = entry.selector_ttl[selector]
if was_stale:
params['beta'] = max(1, params['beta'] - 0.5)
params['alpha'] = min(10, params['alpha'] + 0.5)
else:
if actual_ttl > params['last_sampled_ttl'] * 1.5:
params['alpha'] = max(1, params['alpha'] - 0.2)
params['beta'] = min(10, params['beta'] + 0.2)
This is one of those pieces that looks "small" but changes behavior over time. TTL policy isn't frozen. Selectors keep moving toward whatever production traffic teaches them.
What surprised me here is how asymmetric the update is. When something is stale I move the parameters more aggressively (±0.5) than when it isn't (±0.2, and only under a condition). That matches the real pain: stale reuse is more expensive than an unnecessary rebuild.
The relationship to conformal tau is direct. BDAT adjusts when to re-evaluate, and tau decides whether to rebuild once you do. A selector whose TTL keeps shrinking is one whose conformal threshold will tighten too, because more frequent checks mean more calibration data, and more calibration data means tau converges faster. Two feedback paths on the same evidence, one temporal, one statistical.
Stage 2. Reasoning budget allocator: spend compute like it's cash
Once the cache gate hands back either a reused artifact or a pile of dirty spans, there's still a second problem:
Even inside a rebuild, not all spans deserve the same attention.
The naive move is picking a model tier for the whole extraction. Different blunt instrument, same bluntness.
Instead, I treat each span like a line item in a budget.
Step 1: sort spans by uncertainty
Spans arrive with context. I sort them by a combined uncertainty score:
spans = artifact_ctx.get("spans", [])
spans.sort(key=lambda s: (
s.get("ctx", {}).get("retrieval_dispersion", 0) +
s.get("ctx", {}).get("rule_conflicts", 0) +
s.get("ctx", {}).get("cache_margin", 0)
), reverse=True)
This ordering is where the allocator gets its teeth.
retrieval_dispersion: scattered retrieval means an uncertain span.rule_conflicts: rules disagreeing means an uncertain span.cache_margin: a span that barely cleared the cache gate is uncertain.
The weirdest spans go to the front so they get first claim on the budget.
Step 2: choose an action by value-of-insight
For each span I evaluate the action candidates and take the one with the highest value-of-insight (VOI):
qgainis how much quality I expect to gaincostis the compute costlatencyis the latency costlamandmutrade off cost vs latency
Then I pick the max.
Here's the exact code I run:
for s in spans:
if total_budget <= 0 or s.get("quality", 0) >= target_quality:
continue
candidates = [
("reuse", cached_text, 0.01, 0.0, 0.0),
("small", llm_mini_result, 0.15, 1.0, 1.0),
("tool", tool_result, 0.22, 1.8, 1.2),
("deep", llm_full_result, 0.30, 3.5, 2.0),
]
name, text, qgain, cost, lat = max(candidates, key=lambda c: _voi(c[2], c[3], c[4], lam, mu))
if cost <= total_budget:
total_budget -= cost
s["text"] = text
s["quality"] = min(1.0, s.get("quality", 0) + qgain)
s["action_taken"] = name
return assemble(spans)
Two things make this hold up in production:
- The early exit: a span that already meets
target_qualitydoesn't get touched. - The
reusecandidate:("reuse", ..., 0.01, 0.0, 0.0).
That "reuse gives 0.01 quality gain" is a very opinionated line in the sand. It encodes something I learned the expensive way: reuse doesn't hand you perfect certainty, just a small nudge in confidence, earned because the span existed and passed the cache gate.
And because reuse costs 0.0, most spans clear the bar without spending anything.
How the two systems snap together
The confidence-gated cache is the first gate. It answers:
- "Is this selector safe to reuse?"
- "If not, which spans are dirty?"
The reasoning budget allocator is the second gate. It answers:
- "Given a fixed budget, which spans deserve compute?"
- "What action maximizes quality per unit cost and latency?"
Here's the architecture as it exists conceptually in my pipeline:
Forget the boxes. What matters is the contract between them:
- the cache gate outputs spans with enough metadata (
quality,ctx) for the allocator to make sane decisions - the allocator respects
target_qualityandtotal_budgetso it can't run away
What went wrong (and what I changed)
The failure mode that pushed me toward this design was simple. I was caching like a web server.
A TTL-based cache for AI extraction looks comforting because it's familiar. It also gives you the wrong safety guarantee.
- A long TTL saves money but increases the chance of serving stale extractions.
- A short TTL reduces staleness but rebuilds too often.
Tuning won't rescue that. The axis itself is wrong.
The axis that matters is: how similar is this request to the one that produced the cached artifact, in the ways that affect correctness?
So "time since write" stopped being the primary decision variable, replaced by:
- embedding similarity
- feature drift
- capped age decay
- template version mismatch
Then I stopped pretending the whole artifact is one unit of work and made the gate return dirty spans.
The second failure mode was compute allocation. Even after partial rebuilds I kept overspending, because a rebuild still meant "run the expensive path." The allocator fixed that by making every span compete for budget.
Nuances and tradeoffs
1) The score is a blend, not a model
I like that the score is explicit weights (α, β, γ, η). It's debuggable.
The tradeoff is that you're committing to a worldview. Overweight embeddings and you'll miss structural change. Overweight feature drift and you'll rebuild too often on harmless edits.
I chose weights that keep embeddings dominant (0.6) while letting drift be loud (0.3). Age and template mismatch are present, intentionally small.
2) Conformal thresholds require calibration data
The conformal tau computation depends on calib_scores with observed errors. Early on, you may have none, hence the 1e9 default.
That's the trade: you start permissive and tighten as reality arrives.
3) Partial rebuilds are only as good as your span mapping
Returning dirty spans helps only if entry.dc.spans[sel] is accurate.
Mis-assign spans to selectors and you'll either:
- rebuild too much (safe but expensive), or
- rebuild too little (cheap but wrong)
4) The allocator is greedy
The budget controller walks the spans in sorted order and spends budget if it can. Pragmatic, fast.
The tradeoff is that it isn't globally optimal. It's a greedy knapsack with a VOI heuristic. In practice, the uncertainty sorting makes it behave like I want, fixing the sketchiest spans first.
5) VOI weights (lam, mu) encode product priorities
The allocator's behavior changes dramatically depending on how you set the cost and latency penalties.
That's the design working, not a defect. The same pipeline can run in a cheap batch mode or a fast interactive mode by changing what you punish.
The takeaway I wish I'd internalized earlier
Caching AI extractions isn't about time. It's about whether reuse is defensible.
Same for reasoning depth. Picking a model tier answers the wrong question, and the real one is where a fixed budget buys the most certainty.
Once I treated both as gating problems, first statistical validity and then cost-optimal depth, the pipeline stopped paying full price for answers it already had.