Design Got Clearer Once I Gave Up on 'Perfect' — CARE Agent Design Log

In the previous post, I designed the concept of CARE (Context-Aware Review Engine). The key points:

The skeleton was in place, but I hadn't properly answered the fundamental question: "Will this actually work?"

This time: how a hint came from an unexpected place, and how that led to concretizing the agent design.

Can't static analysis prevent this? Or can't it?

While organizing documentation, another question surfaced: "Can't we fully trace what a code change affects through static analysis?"

While researching that question, I found a Qiita article by @rairaii: "Can code change impact be fully tracked automatically? Technical limits and realistic choices for Forward Slice-based impact analysis automation."

The conclusion was clear.

Full automation is theoretically impossible.

Precise static determination of program behavior reduces to an undecidable problem (Rice's theorem). When you try to exhaustively trace all impacts from a change point — especially with dynamic languages or pointer arithmetic — the moment you aim for "zero misses (Sound)," nearly the entire program gets reported as affected. Even Google doesn't have a perfect interprocedural analysis infrastructure; the industry standard is "Soundy" (approximately sound) approximations.

Honestly, my first reaction was mixed. Part resignation — "if it's that hard, maybe it really is impossible" — and part questioning — "but wait, does this theory apply directly to our problem?"

Drawing a line between the ceiling and the floor

Stepping back, what that article showed was a "theoretical ceiling." Perfection is impossible — that's the ceiling.

Meanwhile, from the previous analysis I had "9 actual incidents" — that's the floor.

The ceiling is high. But the floor is right here. The "good enough" line can be drawn somewhere between them.

Reviewing the 9-incident data, the boundary between what's catchable and what isn't became quite concrete.

"Unknown perspectives" improve over time since rules grow through incident-driven and PR-driven feedback. Meanwhile, "non-PR work" and "runtime input dependency" are structurally out of scope — full stop.

When this framing clicked, the entire design suddenly felt coherent.

The structural difference between Forward Slice explosion and CARE

Thinking one step further, CARE has a fundamentally different design philosophy from Forward Slice.

Forward Slice explosion happens with the "exhaustively trace all impacts from the change point" approach. CARE doesn't do that. When a rule matches the diff, it goes to investigate specific, predetermined questions.

It's closer to targeted testing based on symptoms than exhaustive scanning. Like a doctor who doesn't run a full-body scan every time, but decides which tests to run based on symptoms.

From this perspective, CARE had unintentionally become a "Soundy" approach. Giving up on complete coverage is what makes it practical.

Three boundaries in the design

With the theoretical grounding sorted, I added three core boundaries to the agent design documentation.

Boundary 1: rules control investigation scope

Rules control "what" and "how far" to investigate. Where there's no rule, the LLM doesn't look either. This is the wall that prevents Forward Slice explosion.

The rule structure looks like this:

trigger: What diff patterns fire this rule (table schema changes, specific directory changes, etc.)
question: What to investigate (are there other services referencing this table?)
scope: How far to look (target repo list, DB schemas, etc.)
depth: How many hops to follow
origin: Why this rule exists (linked to incident)

The depth parameter is key — the cost difference between following 1 hop vs. 2 hops is significant. Starting with depth: 1 as the default, expanding as needed.

Boundary 2: the LLM's role is "evidence collection," not "judgment"

This might be the most important design point.

If you let the LLM judge "safe / dangerous," misjudgments become accidents directly. The scariest scenario: a human saying "the LLM said it was fine, so I didn't check."

So the LLM only presents facts.

Good output: "table_x is JOINed in Service B's UserService.java:142"
Bad output: "This change is safe"

Judgment is made by developers who know the context. Reviewers evaluate the "facts" the LLM found. Misjudgment risk is absorbed by the reviewer's eyes, and even if the LLM misses something, "listing the verification perspectives" alone is already valuable.

Boundary 3: explicitly state what's not covered

If you're drawing a "good enough" line, declaring what you're not catching is healthier.

Without this, "CARE didn't say anything, so it must be fine" overconfidence develops. Trust in the system and understanding of its limits need to come as a package.

Agent processing flow

With the three boundaries in mind, I designed the agent's processing flow.

The key is the "even without a match, catch risk signals" branch. If nothing happens when no rule matches, blind spots in the rules stay blind spots forever.

Tools the agent has

Three categories of investigation tools. All read-only.

Code investigation tools

DB investigation tools

Beyond read-only being obvious, DB tool access won't hit production directly. The plan is either exporting production metadata (schemas, statistics) separately, or using a read-only replica. To be finalized in Phase 1 implementation.

AWS investigation tools

Report design

Quality standards for agent-generated reports:

Report format is markdown, designed for posting as PR comments.

## CARE Impact Analysis Report

### Applied rule: Shared table schema change

**Investigation**: Checked for other code/services referencing table_x.

**Discovered facts**:
- Service B (UserService.java:142): JOINed in a SELECT statement
- Batch job C (batch/daily.py:89): Full table scan for CSV export

**Out of scope**:
- Index presence on production DB (planned for Phase 2+)

---
This report was auto-generated by CARE. Judgment is the developer's responsibility.

That last line matters. Explicitly stating "judgment is the developer's responsibility" guards against overreliance on the system.

Scope control and cost management

Phase 1 starts with manual triggers, so per-PR cost can be measured as we go. But several control parameters are set at design time for the automation phase.

The biggest cost concern: as rules grow, "investigation tasks per PR" increases. If 10 rules exist and all fire, 10 investigations run. Phase 1 validation will give a feel for these numbers, feeding into the automation phase investment decision.

What Phase 1 needs to validate

The design is done. But it's still a paper design.

What Phase 1 (manual validation phase) needs to confirm:

Investigation accuracy: are discovered facts actually relevant? (noise level)

Miss rate: are dependencies that should have been found actually missed?

Cost: token consumption and investigation time per rule

Usefulness: do reviewers feel "I could make a decision from this report"?

Tool access feasibility: can read-only DB/AWS access actually be configured?

Noise level is the make-or-break factor for adoption. The moment reports become "a list of things I don't need to look at," reviewers stop reading them. Then the self-reinforcing loop breaks. Measure the noise rate baseline in the first few weeks and use it to calibrate rule granularity.

Honest current feeling

Having the theoretical grounding sorted means I can now explain why CARE's design takes this shape.

"Soundy approach" is a label I found after the fact. The word wasn't there from the beginning. But finding the theoretical backing let me say with confidence: "this is a design chosen with conscious tradeoffs."

I gave up on giving up perfection. Only then could I draw a realistic line.

I'll continue writing about Phase 1 implementation and its results.

Further reading

If you want to go deeper on impact analysis design decisions and systematic approaches to monitoring and observability, these books are good starting points.