Analyzing 9 Incidents Revealed What Impact Analysis Is Really About

Let me be honest upfront: this is not a success story.

Nine incidents occurred in Q1. That's where this starts. Even though we knew "inadequate impact analysis" was the root cause, we kept writing post-mortems for individual incidents and calling it done. So I stopped and tried to decompose the structure using the actual data.

This article is a record of that analysis.

What does "impact analysis" actually mean?

The term "impact analysis" gets used all the time on the ground, but its definition is surprisingly vague.

When you change code, understanding in advance where that change will ripple — that's impact analysis. But in practice, looking at code alone doesn't cover it.

Breaking it into four layers makes it easier to think about.

The source code layer is about call relationships within code. When you change a file, will another file that imports it break? When you change a function's arguments or return type, has the caller been updated? Database table and column references fall here too.

The infrastructure/config layer is about server and network configuration files. With tools like Terraform, "managing infrastructure configuration as code" (IaC) is increasingly common. Changing these settings can unintentionally affect other servers or services.

The service-to-service layer is where multiple services call each other's APIs or communicate through message queues (asynchronous data exchange mechanisms). When you change one service's API spec, you need to verify that other services calling it still work.

The operations/observability layer is about log and alert configuration. Change a log output format, and the systems reading and analyzing those logs can break.

Classifying incidents by "oversight pattern"

To drive the analysis, I first organized "what kinds of oversights exist" into four patterns.

A. Explicit dependency oversight — failing to trace direct call relationships in code. As call chains get deeper, the ripple effect cascades through multiple levels. However, this is the kind of problem tools can supplement.

B. Implicit dependency oversight — missing parts that are actually dependent but don't appear in code call graphs. For example, "multiple services were referencing the same database table" or "systems were coupled through an async message queue." Hard to catch just by reading code.

C. Contract change impact misread — the change itself was identified, but its impact on dependents was misjudged. For example, upgrading an external library also changed the internal React version, breaking a different library.

D. Timing/ordering issues — problems caused by runtime timing or environment differences that aren't visible from static code reading. "The developer's PC had a different CPU architecture than production." "The deploy order was wrong."

The data was surprising

I mapped the 9 Q1 incidents to these patterns.

A was zero.

Not a single incident out of nine was caused by "failing to trace code call relationships." Meanwhile, "dependencies that were invisible in the first place" (B) accounted for five — more than half.

This was an important discovery. It means the countermeasure of "let's add static analysis tools (tools that analyze dependencies without running the code)" would not have caught a single one of these incidents.

Drilling into the 5 B-category cases

Looking deeper into the five cases, two sub-patterns emerged.

Pattern 1: Data sync gaps during legacy/new system coexistence (3 cases)

These were incidents from a period when Product A was undergoing a renewal. Old-spec tables and new-spec tables coexisted.

For instance, "we wrote logic that only updated the new table, but code referencing the old table still existed." Or "we ran a migration script from an external auth service (Auth0) to the database, but there was a time lag between the migration and the source data extraction, so stale data ended up in the DB."

The common thread: "the same conceptual data existed in two places, and when one was updated, the impact on the other wasn't considered."

Pattern 2: Shared data store dependency gaps (2 cases)

These happened in Product B. Multiple services and languages (e.g., both a Rust API and a Java API) were referencing the same database table, and a schema change was made considering only one side.

The other case: "the table definition doc said 'unnecessary table,' so the table was dropped — but a legacy API was still referencing it."

Project-level view reveals different characteristics

Product A has a chance of the problem self-resolving: once migration finishes, the old/new table coexistence disappears. Product B has a structural problem: as long as multiple services share the same DB, similar incidents will keep happening.

But don't be misled by the surface differences. The underlying problem is the same for both.

"When data is changed, there's no way to know who else is reading or writing it."

I looked into whether SaaS could solve this

I wanted to find a service that would fix this, so I researched. Organized by category:

Cross-repo code search (Sourcegraph, GitHub Code Search, etc.)

Tools that let you search code across multiple repositories at once. Instead of manually grepping each repo to "find all code referencing this table name," you can search everything at once. Low adoption cost. But it assumes "the person searching knows what to search for" — if they forget to check, the same thing happens.

Data catalogs / data lineage (DataHub, OpenMetadata, etc.)

Tools for visualizing and managing "where data lives, where it comes from, and where it flows." You can attach metadata to tables and columns — "what is this data for?" "who owns it?" — and check downstream impact when something changes.

But there was a major catch. These tools are primarily designed for data platform pipelines (e.g., data transformation flows feeding BI tools). Application code accessing databases — like Laravel PHP code referencing a table — can't be automatically tracked. You'd need manual registration or additional tooling.

Since most of our incidents were exactly "app code → DB access" oversights, just deploying a data catalog wouldn't solve it.

APM tools (Datadog, etc.)

APM (Application Performance Monitoring) automatically records how applications behave in production. You can see "which service is actually accessing which table" after the fact. Even dependencies you forgot to document get recorded if they actually execute. But code paths that haven't run in production yet (error branches, batch jobs, etc.) remain invisible.

Schema change detection tools (Atlas, etc.)

Tools that automatically check database schema changes inside CI pipelines. They can detect potentially breaking changes at the PR stage. But they support a limited set of languages and ORMs, making full coverage difficult in polyglot environments.

There was no silver bullet

Honestly: no single SaaS matched this incident pattern.

The reason is straightforward. The problem is "cross-product, cross-language visibility into application code accessing databases." It sits at the boundary between tool categories, and no single tool covers it.

Why couldn't we investigate what was investigable?

All nine incidents share one fact:

"We could have found the answer if we'd looked, but we didn't realize we should look."

But be careful here. Jumping to "so let's add a checklist" is premature. Because the real reason "we didn't look even though we could have" was that the cost of looking was too high.

There were three specific barriers.

First: "repositories span multiple codebases, and there's no way to search across them." Products A and B live in separate repositories. "Finding all code that references this table" required manually switching between repos and searching each one.

Second: "searching by table name doesn't find matches." Laravel uses Eloquent, an ORM that maps class names to tables — ProjectUsage maps to project_usage, so searching for project_usage in code yields nothing. Rust uses SQLX, where you need to trace query strings. The search method varies by language and framework.

Third: "references exist outside the main codebase." Checking only application code misses nightly batch jobs or admin panels that touch the same tables.

This isn't purely a tooling problem — it's about "knowing what to check." But even if you have that awareness, it's meaningless without the means to check. Both halves need to be in place to prevent incidents.

How to think about the first move

Three broad approaches are possible.

Option A: "investigate impact each time you make a change." Low cost, can start immediately. But risks remain — "forgot to check at the right time," "no record of what was checked."

Option B: "maintain a dependency map (a map of which code touches which data) in an always-current state." Just consult the map on each change — less person-dependent. But maintaining the map costs effort, and the map can go stale.

Option C: "embed it in the CI pipeline and auto-notify when changes are detected." The biggest strength: no human awareness dependency. But the highest build cost, and you need to define scope.

For a small start with room to scale, progressing from A to C incrementally is realistic. But it's important to lock in the frame of "start from the data store and reverse-lookup dependents" even at the A stage.

With a common frame, the same approach works across products. If you start with product-specific checklists, you'll rebuild from scratch when scaling.

The concrete first step

Embed data-impact checks into PR reviews.

Make the check mandatory for PRs that include any of the following:

What to check is simple. Which other services or code reference or update the table/data source being changed. Whether the same data is stored in another location. Whether other services are affected.

Add the following to the PR template. That's it.

## Data Impact Check (required when applicable)
- [ ] Searched for code referencing the changed table/data source
  - Results: (describe here)
- [ ] Confirmed no dual management of the same data
- [ ] Confirmed whether other services are affected
  - If affected, response plan: (describe here)

This systematizes the "we would have found it if we'd grepped" situation. If it's not filled in, the reviewer asks for it. Records remain, so you can trace "what was actually checked" after the fact.

"But wasn't the whole problem that searching across all repos was too hard?" — fair point. It's not fully solved yet at this stage. But by starting this process on one product, when "searching every time is painful" and "search misses happen" become felt problems, that's when you can make an informed decision about investing in something like Sourcegraph. Choosing tools after the problem is concrete leads to better investment decisions.

What this analysis taught me

When thinking about technical incident response, there's a strong pull toward "we can solve this by deploying a tool." I felt it myself.

But through this analysis, I was reminded that you can't choose tools until you understand the structure of the problem. It was only by looking at the data that I could say "investing in static analysis tools wouldn't have helped with any of these incidents." From facts, not intuition.

As a next step, I plan to map out the full picture of repositories, languages, and ORM configurations, then concretize the tool design for "reverse-lookup from data stores." That's a story for another post.

Further reading

If you want to go deeper on incident analysis and understanding service dependency structures, these books are good starting points.