Three Pivots During Phase 0: The Hallucination Fix That Vanished Before I Wrote It

In the previous post, I wrote about pivoting to the "investigation recipe" approach for the impact analysis generator. I closed that post by saying, "next up is the Phase 0 retrospective test design."

This post was supposed to be that continuation. But when I actually tried it, what I meant to write about as "retrospective test design" turned into a record of redesigning the thing itself three times while testing.

And at the end, the "hallucination fix" issue I'd planned for Phase 1 got closed without writing a single line of code. This is how that happened.

Background: what this tool does

A quick recap. The impact analysis recipe generator is a CLI that detects "blind spots easy to miss in this change" from a PR diff and generates investigation leads.

The prototype flow looks like this.

The goal of Phase 0 was to run retrospective tests and confirm this flow holds up in real use. I had three evaluation axes in mind.

I started with a simple plan: run a handful of past PRs where impact was overlooked, and measure pattern detection and recipe quality quantitatively.

Pivot 1: product-specific assumptions had leaked in

The first test went surprisingly smoothly. I ran a PR that matched the "old/new table sync" pattern; the tool detected the pattern precisely and correctly linked it to related past incidents.

There were prompt-side bugs (braces in JSON examples being misread as str.format placeholders) and a model-name mix-up (the deployment name was slightly off), but the overall flow worked. I thought "not bad" and was about to move to the next PR — and that's when it happened.

While re-reading the generated recipe, my hand stopped.

Hmm, I can't actually tell if the investigation steps shown here are correct...

Specifically, the problems looked like this:

In short: it looked concrete, but the LLM was just filling in steps in a "familiar-looking format" with no guarantee anything actually existed. Classic hallucination.

At this point, I filed two mitigations as issues to handle in Phase 1: a "verification mode" (execute generated commands and flag misses) and "L1 grounding" (include the repo's file tree in the prompt). This issue becomes relevant later.

But at the same time, there was a more fundamental uneasiness.

Looking at the playbook, I saw specific column names, specific flag names, specific internal table naming conventions — product-specific knowledge had seeped in. With that mixed in, I couldn't tell whether a recipe was "a generally correct procedure" or "the tool guessing at our internal context." I couldn't even measure the Phase 0 factuality metric because the yardstick itself was wobbling.

I asked Claude Code: "It's rough that our prototype already has product-specific knowledge baked in. I want to rebuild from zero."

What came back was eight generic patterns (schema changes, flag pinning, dual data paths, and so on). They looked plausible at first glance, but something nagged at me.

These are fine, but aren't you being pulled by the existing file's information? Did you really think from zero?

Claude Code's reply hit harder than I expected.

To be honest, yes — I was pulled by it.
About half are rephrasings of existing categories, and only half are new. So "from zero" was actually half re-labeling.

From there came a new proposal: reorganize not by technical category but by blind-spot axis (cognitive axis).

Instead of classifying by tech (DB / API / env), make the patterns "cognitive axes reviewers tend to miss." Laid out like this, I finally felt the vocabulary had caught up with the "awareness problem" this tool was meant to address. It's a direct re-examination of "we could have investigated if we'd thought to, but we didn't realize we should" — the line from the previous post.

I also replaced incident examples. Instead of case-specific IDs, I used anonymized generic failure archetypes like "E-B1-01: silent behavior change from inverted default arguments." Because these aren't actual incidents, I banned phrasing like "this happened in the past" and positioned them as "typical blind-spot examples" instead.

That was pivot one: rebuild the pattern taxonomy from zero.

Pivot 2: detailed procedures are too heavy — switch to a checklist

With B1–B7 rebuilt, I generated a recipe on the same PR again. Pattern detection accuracy improved, and the steps now used PR-specific identifiers concretely. Technically, it was better.

Better — but a different unease crept in.

The generated recipe had 12 steps. Each step had 2–3 lines of explanation. Reading the whole thing took 5–10 minutes. As a thorough investigation, done seriously, it would be comprehensive. But would everyone actually go through this for every PR?

The typical failure mode is "the steps are so rigorous they're heavy." Overly-correct checklists mostly get skipped. That's the same pitfall I wrote about in the previous post — "static checklists become dead letters" — and I was about to fall into it again with dynamic generation. Dynamic or not, if the content is too heavy, you end up in the same place.

So I changed the recipe format itself.

Three principles:

After this change, the recipe for the same PR got dramatically lighter. "I'd actually read this" — that feeling finally landed.

Pivot two: detailed steps → checklist.

Pivot 3: listed per pattern, you read the same thing three times

I thought this was it. Then I ran it through CI against a real case, and the next problem surfaced.

For one PR, three blind spots triggered simultaneously: B6 (external contract), B4 (timeline consistency), and B3 (cross-environment delta). A recipe was generated per pattern, and the three stacked vertically in the PR comment.

Result: roughly half the check items across the whole thing were duplicates.

The same change summary appeared three times. The same check items appeared multiple times with slight rewording. For a reader, it was a steady drumbeat of "I just read this."

What I hesitated on here was where to merge.

My first thought was "only dedupe the checklist." But looking closer, the change summaries and blind-spot mentions were duplicating too. Partial patching wasn't enough. I switched to rebuilding the report structure as a whole.

Here's the final shape.

# Impact Analysis Recipe

## Summary of this PR
(diff highlights, once)

## Detected blind spots
- B6 External contract: ...
- B4 Timeline: ...
- B3 Env delta: ...

## Checklist
(merged and deduplicated)

Implementation-wise, I kept per-pattern recipe generation and added one more LLM call at the end to merge and dedupe all recipes. Cost: +1 API call, roughly 3,000 extra tokens. Individual recipes stay as artifacts so you can drill down if needed.

Pivot three: per-pattern display → aggregated report.

The hallucination-fix issue closed without being implemented

After those three pivots, I looked at the issue list again.

The "hallucination fix" issue I'd planned for Phase 1 was the one I filed back at pivot 1 — add a verification mode and promote to L1 grounding to suppress hallucination.

When I re-examined what the problem had actually been, though, here's what I saw.

Most of what looked like hallucination at the time had evaporated as a side effect of the design changes.

At the time, I was planning to solve this with "verification mode" (actually run generated commands and surface misses) and "L1 grounding" (put the file tree in the prompt so only real files can be referenced). Neither was cheap to implement — verification mode especially, since it drags in a sandboxed execution environment.

But with just these three changes, most of it was already gone:

I closed that issue. Shelved, to be revisited only if hallucination becomes a real operational problem.

The implementation became unnecessary before I implemented it. That was my favorite moment this time.

What Phase 0 felt like, and where I go next

Quick recap of what happened in this Phase 0:

The clean flow I originally imagined — "run retrospective tests, measure capture rate, pass Phase 0" — ended up pretty different. Honestly, there were moments mid-way when I wasn't sure whether I was making progress or going backward.

Looking back, what worked for me this time was the stance of "keep questioning the design while testing." Pivot 1 in particular — pausing to ask "did you actually think from zero?" — seemed to unlock the next two pivots together. If I'd skipped that question, I would have stacked detailed-step format on top of product-specific knowledge and barreled straight into implementing the hallucination fix. The work would have multiplied.

Phase 0 has only half-accomplished its original goal of "getting numbers." I haven't done enough quantitative measurement of capture rate or noise rate yet. But the coarse design issues have been surfaced, and the recipe shape has settled. "I've arrived at a design from which numbers can be taken" — that's what it feels like right now.

Next is how this checklist format actually gets digested in real use. Will developers actually check the boxes, or will it get skipped? That "human-side acceptance" is really what Phase 0 was supposed to measure.

Only after a detour of rebuilding the design can the actual testing finally begin. Prototypes, maybe, mostly feel like this.

References: collaborating with Claude Code

The pivots in this post started when I paused to ask Claude Code, "did you actually think from zero?" For a broader look at working with AI while shaping designs, the following books are good references (Japanese).