The Day After Shelving Autonomous Agents, a New Idea Landed — The 'Investigation Recipe' Pivot

In the previous post, I wrote about putting "playbook + autonomous agent" on hold after analyzing all 94 incidents.

ROI was borderline. Nice-to-have territory. Operating costs exceeded direct savings. That was the conclusion.

But "on hold" isn't "done." Something had been simmering in my head, and it suddenly took shape the next day.

The pivot point

I re-examined the fundamental reason for shelving the autonomous agent.

What the 94-incident analysis revealed was that the root cause of impact oversights was an "awareness problem."

"We could have investigated if we'd thought to, but we didn't realize we should."

Breaking this down more carefully, two problems emerge:

The autonomous agent tried to solve both. That's why it got heavy.

Then I thought: wait.

What if solving just the "awareness problem" is enough?

Looking back at the incidents where impact was overlooked, developers could have actually investigated "if they'd known where to look." They had the investigation skills. What was missing was the clue: "for this PR, what should I go check?"

If so, the agent doesn't need to investigate impact itself. It just needs to dynamically generate and present investigation steps (recipes): "for this PR, please investigate impact using these steps."

Developers execute the actual investigation as before. The agent provides the path, not the destination.

This is what I'm calling the "investigation recipe" approach.

Working out architecture with Claude Code

I brought this idea to Claude Code and we worked through it together.

The first topic was "how to design dynamically changing perspectives." The reason static checklists go through the motions is that every PR gets the same perspectives. Dynamic generation presents pinpointed perspectives based on PR content.

But Claude Code came back with a question:

What if the output is "investigation recipes" rather than "perspectives"?

This reframing made things much more concrete.

"Watch out for column X type change" as a perspective still leaves developers stuck — they don't know what to check or how. As an "investigation recipe," it becomes:

[Suspicious location]
  Migration file L12
  Column X type change (width reduction)

[Expected risk]
  DB column overflow. Write-side actual values may exceed defined range

[Impact investigation steps]
  1. Check actual value range in production DB
     SELECT MIN(col_x), MAX(col_x) FROM table_a
     WHERE created_at > NOW() - INTERVAL 6 MONTH;
     → Does it fit within the new type's range?
  2. Check write-side code
     Search for other INSERT/UPDATE to this column
  3. If inputs exceeding the limit are possible, consider CLIP/truncation

The investigation "path" is visible. Developers can act on this.

Architecture design direction

I organized how to implement this approach.

Three candidates:

Architecture A: Single LLM + search

Summarize diff and changed symbols → keyword search playbooks for Top-K → single LLM call for recipe generation. Cheap and fast, but specificity depends on LLM guessing.

Architecture B: Autonomous agent

Scaled-down version of the shelved approach. Tool calls to walk related files. More specific, but cost problem returns.

Architecture C: Pattern-first hybrid (recommended)

Deterministic pattern detection passes only "hits" to the LLM.

Pattern detection uses recurring patterns extracted from past incident analysis (column type changes in migrations, old/new parallel table sync gaps, external auth updates, etc.) detected via regex or AST. Only matched patterns trigger LLM calls — no match means zero LLM invocations. Cost is controllable.

Why I recommend this — several reasons:

1. False positive handling is fundamentally different

When an autonomous agent produces a false positive, it's a "wrong conclusion" that damages trust. A recipe approach's false positive is an "unnecessary recipe" — the developer judges "this doesn't apply" and moves on. Much higher noise tolerance.

2. Validation runs on diff + prompt alone

Autonomous agent validation required tool connections and was expensive. Recipe approach allows retroactive testing against past incident diffs: "if this recipe had appeared back then, would the developer have noticed?" Manual or low-cost.

3. More patterns = more organizational knowledge

When a new incident occurs and a pattern file is added, coverage automatically expands next time. Post-mortem driven "organizational learning."

The grounding granularity question

One design judgment point: "how much specific code information should the recipe include?"

At L2, it approaches the autonomous agent. Without limiting tool calls, it becomes the same shelved approach.

Phase 0 (retroactive testing) starts at L0, progressively moving to L1 and L2 while watching the specificity-noise balance. Even at L2, tool call count limits serve as guardrails.

Document restructuring

Along with the pivot, I fully restructured the documentation.

Originally everything was packed into one proposal document, but with Approach A (autonomous investigation) on hold and Approach B (investigation recipes) emerging, the structure was getting confused.

docs/impact-analysis/
├── 01-proposal.md               ← Overall direction (restructured around Approach B)
├── 02-incident-analysis.md      94-incident analysis details
├── 03-approach-comparison.md    Approach comparison
├── 04-rule-management.md        Playbook design (legacy asset)
├── 05-agent-design.md           Agent design (legacy asset)
├── 06-review-recipe.md          Investigation recipe approach draft (current mainline)
├── 07-autonomous-agent-approach.md  Autonomous investigation (on-hold alternative)
└── 08-derivative-initiatives.md    Derivative measures (4 items)

The especially important decision was "don't discard shelved Approach A — preserve it."

If Approach B's Phase 0 validation shows recipe capture rate falling below threshold, there's no need to rebuild the argument from scratch. 07-autonomous-agent-approach.md holds the full cost estimates, design principles, and re-evaluation triggers. It's a "memo to future me."

Honest current feel

The "investigation recipe" idea came the day after the hold decision. Working through the architecture with Claude Code had a "this is clicking" feeling. The instinct that "this approach has legs" is real.

But Phase 0 retroactive testing hasn't happened yet.

Until I actually generate recipes against past incident diffs and verify "would developers have noticed if this had appeared back then," I can't confirm whether this instinct holds.

Compared to autonomous agents, the cost structure is fundamentally different (operating cost estimate drops to ~1/3), false positive tolerance is higher, and validation cost is lower. These three are clear design advantages.

On the other hand, "will developers actually read recipes and act" is a separate question from whether it's technically solvable. The hypothesis is that checklists went through the motions "because they were static," but even dynamic generation gets ignored if noisy. That's something only Phase 0 can answer.

I enjoy the work of refining design, but the stage where intuition can guide decisions is long past. It's time to go get numbers.

Next, I plan to write about the Phase 0 retroactive test design.

Further reading

If you want to go deeper on incident analysis and detection strategy design, this book is a good starting point.