Rewriting AI's 'Frequency-First' Automation Strategy to 'User-Impact-First'

In the previous article, I wrote about analyzing 200+ release procedure documents and building an "observation" doc. This is the sequel — a record of the next phase, where I started examining the automation strategy.

I had been largely letting AI draft the strategy document too. But the priority matrix it came back with was sorted by "frequency, high to low" — and I felt "no, this isn't right" and rewrote it.

It's a tech-decision log, but as a sequel to the previous article, please read it as a record of testing the boundary between "what to delegate to AI" and "what humans must judge" — this time in the context of strategy formulation.

Background: I let AI draft the strategy doc too

By the end of the previous article, I had finished reading 200+ release procedure documents and classifying them into 22 deployment patterns. About 40 procedure docs pile up every half-year, and I had them sorted by type.

As I wrote at the end of that article, the next phase is to decide "where to start automating" and turn that into a strategy document. The motivation was to put together a realistic sequence, accounting for existing assets like AWS ECS, Lambda, Pulumi, and Jenkins.

For the observation doc, the split had been: let AI handle bulk analysis, let humans verify whether the classification is valid. I assumed the strategy doc could proceed under the same vibe. Ask AI to "draft an automation strategy," then iterate on what it returned.

Premises

The priority AI came back with

As a first pass on strategy, I asked AI to draft a priority matrix.

Summarized, the priorities looked like this:

frequency × Pulumi fit → priority
- Common post-processing: 246 cases (all releases), fit ◎ → top candidate
- API deploys: dozens, fit ◎ → runner-up
- Batch systems: dozens, fit ◎ → runner-up
- Frontend systems: dozens, fit ◎ → runner-up
- DB / external API / emergency response: few → deprioritized

AI's reasoning: "the highest-frequency common post-processing pattern has the broadest labor-saving effect," "ROI is high," "this should be the first thing tackled in Phase 2."

On the surface, it looked coherent. AI's proposals — pulled from a large dataset — carry a certain weight of authority. The logic of "automate the most frequent thing first, it's most efficient" was persuasive enough that a corner of my mind almost went "well, sure, that tracks."

Friction: "from the user's standpoint, does that even touch quality?"

But as I kept reading the strategy doc, and after I asked AI to "review this as if you were the CTO" and let it self-review, strong friction emerged.

In the CTO review, AI raised — against itself — points like "the doc doesn't explain why common post-processing isn't the top priority" and "if ROI is the highest, why is a different pattern being placed at top priority?" In other words, AI defended its own logic, against itself, by saying "the most-frequent pattern should be top priority."

That's when the friction surfaced:

If we automate common post-processing, does the user's experience get any better?

The common post-processing pattern consists of clerical tasks that always run after deployment — tagging, updating the deploy register, appending to the release notes, announcing to stakeholders, and so on.

No matter how much you automate these, the product's behavior itself doesn't change. Engineer-hours go down, but the screens and APIs that users actually touch are completely unaffected.

API deploys, on the other hand, are the pattern where a failure produces errors that users see directly. The frequency is some fraction of the most-frequent pattern, but the "user impact on failure" per case is in a different league.

The reason AI was saying "common post-processing has high ROI" was an implicit assumption that "ROI = frequency × labor savings." The "user impact" axis had dropped out of the formula entirely. That was the true source of the friction I felt with AI's proposal.

Why did AI build it "frequency-first"?

I replied to AI: "common post-processing has high frequency, but from the user's standpoint, isn't it work that doesn't affect product quality?"

AI immediately acknowledged the bias in the evaluation axes — clearly admitting as an error that it had evaluated using only "frequency × labor savings" and hadn't accounted for the user-impact axis.

Something got put into words for me at that point.

AI tends to build evaluation axes around what is "measurable" in the data it's given. Frequency can be counted; automation fit can be judged from technical specs. But "how a user experiences this" isn't directly contained in the data. So unless you explicitly instruct, it falls out of the evaluation axes.

When you tell AI "evaluate by ROI," what gets included in ROI changes the result entirely — but AI doesn't explicitly state what it did and didn't include. This isn't AI hiding something maliciously; it's that the construction of the evaluation formula has a data-centric bias baked in.

Rewriting the evaluation axis

So I rewrote the priority evaluation axis itself.

Before (AI's draft):

priority = frequency × automation fit

After (rewritten):

priority = frequency × user impact on failure × automation fit

Once I rewrote and re-drew the matrix, the priorities flipped cleanly.

Two big changes happened here.

First, the most-frequent pattern (common post-processing) dropped out of top priority. Its only effect is labor savings, and its value as a verification case for the overall strategy is thin. We keep it, but place it at low priority.

Second, several patterns that AI had implicitly treated as "few cases, deprioritize" got promoted. DB version upgrades, table deletions, and similar "happens a few times a year but loses data on failure" patterns look like things to deprioritize when looking at frequency alone — but from the user-impact view, they're top-priority class.

That said, many of these are low on "Pulumi fit." I realized the right framing is to pull these in not as "automation" but as "procedure quality assurance" — as a separate axis of the strategy.

Result: putting API deploys back at top priority

In the end, I picked API deploys as the first automation target. Three reasons:

The most-frequent pattern (common post-processing) has the highest frequency, but satisfies none of the three above. It wasn't suitable either as a verification case for the overall strategy or as an early realization of user value.

What to delegate to AI, what humans should judge

In the previous article, I wrote about delegating bulk procedure analysis to AI. The boundary there was:

Through this round of work, the boundary in the strategy-formulation context also came into focus:

When you tell AI to "set priorities," AI combines what's "measurable" in the data and builds an evaluation formula. This is fast and accurate, but the validity of the evaluation axis itself is something AI can't re-question on its own. It presents the declaration "I am evaluating with this formula" and the result as a set — but asking "is this really the right formula?" falls to the human side.

This time, AI's draft almost half-passed because AI persuasively asserted "the most-frequent pattern should be top priority." AI was saying "I'll evaluate by ROI," so the logic looked coherent.

But once I re-examined what was inside ROI, the user-impact side had fallen out completely. An evaluation formula built only from "measurable things" is a typical distortion that comes out of AI, I felt.

I think the reason I caught it here is that I had been writing the previous article. Because I held the awareness that "there's a boundary between what to delegate to AI and what humans must judge," the habit of doubting AI's draft kicked in, just slightly. Without that, there's a reasonable chance I would have let the frequency-based priority through.

A side observation: cost-benefit estimates also showed the same distortion when delegated to AI

After rewriting the evaluation axes, I added a cost-benefit estimate to the strategy doc. I had AI draft that too.

In the first calculation, the payback period came out to about 6 years, and I caught on: "this doesn't match my gut." For automation at this scale, 6 years to break even is clearly too conservative in feel.

When I told AI "this is misaligned with my intuition," AI acknowledged accounting omissions and revised the estimate.

After putting these in correctly, payback came down to about half a year. The gap with my intuition closed.

The same structure as priority evaluation was at work here too. AI only counts what can be "directly tallied." Indirect effects, qualitative value, ripple effects that originate outside the immediate task — these don't enter the calculation unless explicitly instructed.

Ask "how many years is the payback?" and AI returns a number. But what's in the formula that produced that number isn't visible unless humans actively check. This too got embedded as a felt experience through this round of work.

Where this lands

A few things got put into words for me through this work — let me record them.

When you have AI draft a strategy document, the evaluation formula gets built around things measurable from the data. This is useful, but the validity of the evaluation axis itself cannot, in principle, be detected by AI's self-review. When AI flatly says "frequency × labor savings = ROI," what is not in that formula isn't visible unless humans actively ask.

I think delegating strategy documents to AI will probably continue. The speed of organization, aggregation, and documentation is incomparable to manual human work. But "what to include as the evaluation axis" and "where to place the subject" are probably parts humans shouldn't hand off — all the way through.

The fact that I had been writing the previous article — being conscious of "the boundary between what to delegate to AI and what humans must judge" — ended up paying off in this scene. The habit of being aware of the boundary will likely get re-questioned again and again in new situations.

What to try next

For the strategy document, I've organized as far as "what to prioritize for automation first" and "patterns where failure is unacceptable get pulled into a separate strategy."

Next, I move into the phase of actually starting automation for API deploys. How to build it under the constraints of shared AWS resources (shared ALB, ECS cluster) looks like the largest point of contention for the verification case. I'd like to capture what it was actually like, as a separate article.

As someone who builds strategy documents with AI, my biggest takeaway this time is: keep the habit of always asking, once, "is this really the right formula?" of any evaluation formula AI drafts.

Something I noticed while writing: both this round and the previous one, I was redoing the theme of "what to delegate to AI vs. what humans should judge" in different scenes. Last time it was "the validity of the classification"; this time it was "the validity of the evaluation axis." In both cases, the structure is the same: humans aren't checking the result AI returns, but the premise sitting upstream of it.

When you work with AI, the habit that probably gets re-questioned over and over is exactly this — looking at "the premise, not the result."

Related Books

Here are recommended books for those who want to learn more about AI-assisted strategy formulation, deploy automation, and operations in AWS environments, like in this case.