Analyzing All 94 Incidents Flipped the Design Twice

In the previous post, I wrote the agent design skeleton for impact analysis automation, centered on a Soundy approach. At that point, the design was based on "9 incidents analyzed as a representative sample."

After that, I got access to a much larger dataset. So I ran a full analysis. And several design assumptions flipped. This is that story.

From 9 to 94

The previous sample was 9 incidents. This time, I obtained 94 incidents spanning over two years and reclassified them using the same framework.

The analysis covered two influencer marketing products (Product A and Product B). Product A had about 90 incidents, Product B had 4.

First check: A=0 held up

The previous finding — "zero A-pattern incidents preventable by static analysis" — remained true across all 94.

Static analysis tool investment remains low priority. The basis for this judgment was upgraded from a 9-incident sample to the full 94.

The surprise: D-pattern carries more weight than expected

The previous design was built around "B (implicit dependency oversight) is dominant." But looking at all 94, the picture shifted.

Incidents where impact analysis could have potentially discovered the issue: 29 (31%). Breakdown:

D-pattern at 11. The 9-incident analysis only had 2, so the impression is quite different.

Concretely, D-pattern includes "column width exceeded actual data range," "environment variables weren't updated before release," and "library failed due to Intel/ARM runtime differences." All of these are invisible from code diffs alone. You need to check DB and AWS state to see the problem.

This upgraded the justification for "designing investigation to cover DB and AWS, not just code" from 9 incidents to 94.

The trap hiding in "spec/implementation gap" classification

54 of 94 incidents (60%) fell into "spec/implementation gap." Since this was the largest category, my first instinct was "let's design measures to reduce spec/implementation gaps."

But looking inside, it decomposed into three distinct natures:

The "spec/implementation gap" label doesn't distinguish between these natures. Treating them as one group makes countermeasure discussions drift. What this system can prevent is only the "~15" portion — that separation matters.

Recurring patterns emerged

Organizing 94 incidents, several patterns of structurally identical problems appeared across multiple incidents.

For example: multiple "DB column definition exceeded actual data range" cases. Multiple "user info sync drifted between external auth service and DB." Multiple "old and new tables running in parallel during renewal, with one side's updates missed."

These can be verbalized as rules (or narratives). Not one-off discoveries, but knowledge: "when this pattern appears, check here next."

The design flipped: dropping YAML structured rules

Up until now, "rules" were defined in YAML format: trigger / question / scope / depth. The agent reads rules and decides investigation scope.

But discomfort grew with this approach. Looking at all 94, recurring patterns (R1–R5) accounted for about 10 incidents. The remaining 19 were different patterns every time.

Pre-defining rules for patterns that differ every time is hard. Meanwhile, modern LLM agents (coding agent CLIs) autonomously call tools and explore until they reach a goal. This autonomy shines exactly when exploring "patterns that differ every time."

So I pivoted.

YAML structured rules → Markdown narratives (playbooks)

Specifically, create a .care/ directory structured like this:

.care/
├── README.md              # Instructions for the agent
├── incidents/             # Past incident narratives
│   ├── 2024-xx-incident.md
│   └── ...
├── patterns/              # Abstracted recurring patterns
│   ├── R1_DB_column_overflow.md
│   └── ...
└── focus-areas.md         # Resources requiring special attention

Each file in incidents/ describes: what happened, direct cause, structural cause, and "discovery hints." The "discovery hints" section is key — it becomes the agent's clue for "where to look next."

Writing post-mortem digests is lower friction than filling in YAML formats. And it naturally overlaps with existing incident response workflows.

For autonomous agents, use "downstream guardrails"

One role of YAML structured rules was "strictly controlling the agent's investigation scope." Pre-defining how far to look via scope/depth.

But the strength of autonomous agents is deciding their own next action. Over-constraining scope upfront kills that strength.

So I switched from "upfront constraints" to "downstream guardrails."

- Max tool calls: 30/PR
- Max token consumption: 100k/PR
- Timeout: 5 min/PR

When a guardrail is hit, the report explicitly states "investigation ended mid-way." Report even if incomplete. This balances "preventing runaway while preserving autonomy."

Stage splitting

Having one agent do "playbook matching, hypothesis generation, evidence collection, evaluation, and report formatting" all at once blurs the instructions.

So I split by role:

Only the Investigator gets the powerful model + autonomous loop, concentrating investment on the highest-value part.

Inter-stage information passing was honestly where I got stuck. Passing structured JSON looks clean but loses the LLM's nuances and reasoning traces. Four principles as countermeasures:

MCP wasn't needed

Initially I planned to "design DB/AWS access via MCP (Model Context Protocol)." But in a Jenkins environment, just granting the agent IAM roles + DB credentials + Git auth enables direct tool calls. The MCP abstraction layer was unnecessary.

"MCP gives loose coupling" is valuable when multiple LLMs share the same tools. It doesn't apply to Jenkins' fixed toolset.

Honest CTO-perspective evaluation

With the design solidifying, I stepped back for an ROI calculation.

Calculated at median estimates:

Direct labor savings alone don't cover operating costs.

"It's worth it when you include business impact" is an argument, but looking at all 94 incidents, most were recovered in hours, and cases with clear brand/revenue impact were limited.

Honestly: at this point, it's "nice to have" territory.

Current conclusion: on hold, not abandoned

Phase 1 kickoff is on hold for now. But "on hold" isn't "case closed" — it's "conclusion for now." I'll revisit when these trigger conditions are met:

Value gained without building anything

Even without building the system, this investigation produced concrete outcomes.

Immediately actionable derivative measures:

α: Add "discovery hints" and "similar cases" fields to incident report template

Format change only (hours of work). Future re-evaluation data accumulates naturally.

β: Add recurring patterns to PR review checklist

When changing DB columns, verify against actual data ranges. When touching one side of parallel old/new tables, check both. Human review checklist items implementable in 1–2 days.

γ: Share static analysis investment priority judgment across the org

"A-pattern was 0 out of 94" is a judgment basis for vendor discussions.

The 94-incident analysis data itself remains as evidence usable for test strategy and monitoring strategy reviews.

Looking back

An analysis that started with 9 incidents expanded to 94, and the design flipped twice.

First flip: "YAML structured rules → Markdown narratives." For leveraging agent autonomy, downstream guardrails fit better than upfront constraints.

Second flip: "build → hold." An honest ROI calculation showed it doesn't justify the investment right now.

This "hold" decision was also derived from about 15 hours of analysis and deliberation. If you can decide not to start before starting, that has value too.

Next steps: run derivative measures α and β while accumulating data. Revisit when trigger conditions are met.

Further reading

If you want to go deeper on incident analysis and monitoring design, this book is a good starting point.