When \"Absolute\" Becomes a Trap — Adding an Untestable-Exit to the Testing Phase

What was happening

While operating DevLoop Runner, a particular kind of friction kept piling up.

In the testing phase, "tries 3 times, fails, stops" cases seemed to be occurring structurally. And the way they got stuck was always the same: it happened — almost without exception — when the acceptance criteria included items that couldn't be verified physically.

For example:

When these kinds of items make it into the acceptance criteria, the test-execution agent honestly writes "no real device available, can't verify." The reviewer then fails the quality gate, revise runs, fails again — and this loops three times before stopping with an exception.

It's strange that being honest is what gets you stuck. I started suspecting this was a spec-level bug.

First, read the code

The hunch "this might be a spec bug" can easily turn out to be wishful thinking, so I read the code first. I'd been through execute a few times before, but this time I traced the testing-phase decision logic carefully.

Here's the structure I found.

Where the 3-retry rule lives

In src/phases/core/review-cycle-manager.ts, MAX_RETRIES = 3 is hardcoded. Each time the testing-phase review fails, it loops through revise → review → … up to three times, then throws an exception and stops.

The "FAIL logic" of the testing phase

src/prompts/testing/ja/review.txt had this rule: among the three quality-gate items (tests have been executed / the main test cases pass / failures have been analyzed), if even one is FAIL, the final verdict is automatically FAIL. There was a "80 points is good enough" softening clause too — but it only applied after the quality gate had been cleared. In other words, the quality gate was an absolute condition.

The only escape hatch was for "tests not needed"

src/prompts/testing/ja/execute.txt only covered cases like "test implementation was skipped" or "documentation-only changes" — the skip procedures for when tests themselves aren't needed. Guidance for "tests are needed but can't be physically executed" was missing entirely.

Revise only offered two choices

revise.txt only presented "go back to Phase 4 (implementation) / fix the environment." Physical constraints can't be solved by either implementation fixes or environment fixes, so no number of retries makes any difference.

That's when the structure finally became visible.

For physical-constraint cases, there's no exit from the lower-left loop in the diagram. The more honestly the agent reports, the deeper into this loop it falls.

Lining up three options

Now that I knew where the spec hole was, I needed to think about how to fix it. "Just patch it" tends to push distortion elsewhere, so I lined up three options.

A is structurally clean. If acceptance criteria carried an upstream flag like "auto-verifiable / physical-constraint," the testing side would never have to guess. But that means redesigning the output structures of requirements / test_scenario, the metadata of existing issues, and consistent treatment in every downstream phase. Too painful.

C adds a code-level guard on top of B, but mechanically deciding "is this really a physical constraint?" is fundamentally hard. Misclassifying it could ironically encourage the very abuse the guard is supposed to prevent.

B is the minimum-invasion option — just prompt edits. The abuse risk does remain, but I judged that the realistic move was to ship B, observe it, and then escalate to A if needed.

My natural tendency, when in doubt, is to "pick the structurally correct one," which would push me toward A. This time I prioritized "stop the pain." Move within the range that resolves the immediate deadlock without sacrificing overall coherence. This connects to a lesson I learned earlier when thinking about verification-cycle design — the value of not over-engineering.

Fixing a spec bug with prompts alone

The implementation was just six prompts (ja/en × execute/review/revise), two report prompts, and one regression test. I didn't touch the TypeScript in testing.ts at all.

Concretely, here are the changes I made.

Addition to testing/ja/execute.txt

I defined a format for an "## Untestable items (physical constraints)" section, and made the following mandatory for each item:

The point is: don't let "it's a physical constraint, so untestable" be the whole story. Force the agent to think through an alternative verification method. Without this, the section just becomes a door for skipping abuse.

Softening the quality-gate verdict in testing/ja/review.txt

I added the rule: "When untestable items are recorded with a justified reason, evaluate 'the main test cases pass' on the verifiable scope only." With this, even when physical constraints block some items, the verdict can pass through as PASSWITHSUGGESTIONS.

A third option in testing/ja/revise.txt

This is the heart of the change. The choices became "back to Phase 4 / fix the environment / record as untestable items." Physical-constraint cases now have a way out of the infinite loop.

An "Unverified items" section in the report prompt

Untestable items get aggregated into the PR body, and ultimately a human review covers the gap. In an ambiguous area like this, leaving a path that hands the final call to a person — instead of letting AI close the loop — really matters.

Why prompts alone were enough

In hindsight, this is a slightly interesting point.

A lot of the behavior in workflow systems is governed by rules written in prompts. What testing.ts actually checks is barely more than "is python3 installed?" — the substance of pass/fail logic lives in the natural-language rules inside prompts. So rewriting "what's an absolute condition at the quality gate" or "what choices revise presents" on the prompt side is enough to fix even structural bugs.

This is something I've felt several times in LLM-driven development — the check "before changing code, can I do this with prompt edits?" is worth running consciously every single time. It cut the implementation cost dramatically here too.

What I learned

Once the implementation was done and I sat back to organize, several lessons surfaced.

Design "absolute conditions" together with their escape hatches

If you place "all PASS is an absolute condition" at a quality gate, you also need to design what can pass through as PASSWITHSUGGESTIONS at the same time. What was getting stuck this time was the asymmetry: the absolute condition was clearly written, but the conditions for the escape hatch weren't defined.

When you design absolute conditions, deciding "these are non-negotiable" is comparatively easy. The harder part is deciding "how to handle the case where it's non-negotiable but structurally cannot be satisfied" at the same time. This time, that part was forgotten.

Separate the meanings of "skip"

The existing prompts only covered "skip when tests themselves aren't needed" and missed "tests are needed but can't be physically executed."

Going forward, when adding new skip judgments, I want to consciously distinguish:

Without this three-way split, I'll keep digging the same hole.

Enumerate "where to fix" exhaustively in revise

If you write revise as a binary "fix it or send it back," you get stuck the moment a problem arrives that neither side can solve. What I learned is that revise's options should be designed across three axes: "send upstream / fix in this phase / record properly."

In particular, I think putting "record properly" in the option set from the start is important. When letting AI make calls, don't just offer "fix" and "send back" — always reserve an exit that hands the call to human review.

Process lesson: even for exploratory questions, return proposals with grounding

I started this from a vague "I'd like to do something about this," but the moment proposals came back grounded in specific code locations (filenames and line numbers), decision-making accelerated dramatically.

Abstract option lists alone make it hard to tell which is realistic. Proposals tied to evidence in the code are easier to choose, and they reduce hesitation when implementation actually starts. I think this isn't unique to AI conversations — it's the same in human discussions.

A template for layering proposals

The 3-option structure this time (A: root fix / B: minimum invasion / C: middle ground) feels reusable for problems of this shape. The landing of "start at B, observe, then escalate to A" balances risk and effect, and agreement came quickly.

"When in doubt, pick the harder one" is my usual axis, but lately I've been finding that "escalate to harder ones in stages" works out better in practice.

What's next

Option B has stopped the immediate pain, but option A — the root-cause fix — is still ahead. After 2–3 weeks of operation, I plan to observe:

Based on that, I'll decide whether to escalate to option A. If abuse becomes visible, option C — adding code-level guards — also remains on the table.

The balance between absolute conditions and escape hatches is, I think, something you tune incrementally as you operate. A perfect line probably can't be drawn from the start. The right place gets found only by repeating "stop the deadlock" and "tighten when abuse is spotted."

Related Books

Here are recommended books for those who want to learn more about AI workflow design, quality-gate design, and testing strategy like in this case.