Embedding Impact Analysis Into CI — The Thinking Behind CARE

In the previous post, I analyzed 9 incidents from one quarter and concluded that "implicit dependency oversight" was the root cause.

The three approaches I proposed were:

But I'd only written about the direction — not how to actually implement it. This is the continuation: a thinking log of how the design progressed.

Starting with a checklist hit its limits immediately

The first idea was adding data-impact check items to the PR template. Simple, can start today.

But recalling the root cause from last time: "we could have investigated if we'd thought to, but we didn't realize we should."

Which boils down to: you can't check what you don't know exists.

Even if "verify old/new table sync" is on the checklist, if you don't know your change relates to old/new tables in the first place, you'll skip that checkbox. Checklists assume you already know what to look for — they don't fit this problem's structure.

And during crunch periods, checklists go through the motions anyway. I think everyone's experienced this.

Comparing "prevention cost" vs. "recovery cost"

If checklists are hard, there's always architectural refactoring. For example, unifying the shared DB tables that multiple services access directly behind APIs. That would be a fundamental fix.

But the development investment is substantial.

So I laid out the recovery costs for the 9 incidents side by side.

Most incidents were recoverable in hours to a day. Even with all-user impact, downtime was short.

But one incident was different in nature. A business process had proceeded in an incorrect state, and "undoing that fact" was inherently difficult.

This clarified the investment axis:

"Prevention cost" vs "Recovery cost + Business damage"

→ Most incidents: recovery is cheaper → invest in detection/recovery
→ Irreversible business impact: prevention is cheaper → defend only those
→ Need a method that's minimally costly yet effective for both

I reconsidered — there might be a more realistic solution than a major architectural overhaul.

The idea: build on existing CI infrastructure

That's when someone mentioned, "we're already running code complexity checks in CI."

When a PR is created, CI runs automatically and posts code quality metrics as PR comments. The mechanism already exists.

Why not extend this?

Hold a mapping as rules: "when this file or table is changed, somewhere else needs checking too." Then match PR changes against those rules.

This became the prototype for CARE (Context-Aware Review Engine). The name and concept are something I came up with.

Two key points.

Point 1: separate "what to investigate" from "actually investigating"

Going back to the root cause, the problem had two layers:

Checklists partially address the awareness problem but don't solve execution. Someone still has to do the investigating.

CARE's division of labor:

Humans focus on accumulating knowledge. The investigation work itself goes to the LLM.

Point 2: it's a self-reinforcing loop

When a PR has risk signals but doesn't match existing rules, doing nothing wastes an opportunity. Instead, the LLM analyzes the changes and suggests: "should this pattern be registered as a review point?"

Developers just choose Accept or Skip.

This creates a loop where rules grow richer with use. Rules accumulate from everyday PRs, not just from incidents.

How to design rule management

The hardest part was deciding where to manage rules.

Option A: distributed across repos (each repo has .care/review-points.yaml)

Upside: developers naturally update it alongside code. Rules can ship in the same PR.

Downside: most of these incidents came from "cross-repo impact." Distributed placement makes cross-repo perspectives hard to write.

Option B: centralized repository

repo-care-rules/
  ├── product-a/
  │   ├── api.yaml
  │   └── batch.yaml
  ├── product-b/
  │   └── main.yaml
  └── cross-service/
      └── shared-tables.yaml

The primary problem to prevent is "cross-repo dependencies," so centralized management is more natural. Cross-repo perspectives are straightforward to write. The full rule landscape is visible.

Downside: code changes and rule changes become separate PRs. But rules often update on a different lifecycle than code anyway, so that might be fine.

Given this problem's structure, centralized management (Option B) was the right call.

Designing against staleness

Centralized rules mean nothing if they're never updated. "Let's review periodically" doesn't work — experience confirms this.

What's needed: a mechanism where updates happen naturally.

Three pathways:

Pathway 1: PR-driven (daily development flow)

When a PR has risk signals but doesn't match rules, the LLM automatically proposes new rules. Developers just Accept/Skip. Rules grow naturally within daily development.

Pathway 2: incident-driven

A lightweight addition of 2–3 questions to the incident report template:

If yes, the LLM auto-drafts a rule from the incident report. No heavyweight post-mortem meetings needed. Right after recovery is when the cause is best understood — capture it mechanically at that point.

Pathway 3: manual addition

Whatever the first two don't catch, developers add directly. This is the last resort.

Record the origin of each rule

Recording "why this rule exists" for each rule is critical.

- trigger:
    tables: ["shared_table_name"]
  prompt: |
    This table is referenced by multiple services.
    On schema changes, verify impact on other services.
  origin:
    incident: "Incident name from YYYY-MM-DD"
    date: "2026-04-XX"

Without this:

Rules are Git-managed like code, with full change history. Not "someone wrote it in a Wiki" but "who changed what and why is in the commit log."

Validated against the 9 past incidents

Once the design was reasonably solid, I simulated whether this system would have prevented the 9 incidents.

Effective for 6–7 out of 9. Non-PR work (#6, data migration) is structurally out of scope.

Rollout: don't go fully automated from day one

When implementation phases came up, the decision was "no need for webhook auto-triggering from the start."

By setting up the execution environment in Phase 1, moving to automation is just "connect the webhook." No "verification and production behave differently" problems.

For LLM-as-agent costs, manual triggering means you can measure per-PR cost as you go. Whether to run on every PR or only risk-signal PRs is decided after seeing real data.

Honest assessment at this point

CARE is trying to solve two problems:

Last time I identified the root cause as both "awareness" and "execution." CARE can potentially solve both, as an extension of existing CI infrastructure, at realistic cost.

Whether it actually sticks is still unknown. Especially "will rules keep growing?" Whether CARE's self-reinforcing loop actually works needs Phase 1 validation.

This is still at the design stage — about to start testing. I'll continue writing about the process.