Calling It an Initial Version: Closing the Prototype Loop on an LLM PR-Comment Tool

In the last post, I wrote up the work of tightening the PR-comment tool (the "recipe generator") from both the prompt side and the code side, under the banner of "raise the quality of the silence."

「また同じ警告か」をどう避けるか：PRコメント生成ツールの品質を、沈黙の質から問い直す

PRレビュー補助ツールが Phase 0 で動いたあと、出てくるコメントの品質に違和感が残った。grounding・動詞の縛り・沈黙の質──LLM を組み込んだツールを「使いながら育てる」試行錯誤の記録。

Tech

開発ラボ

After adding grounding, a checklist cap with forbidden categories, verb discipline, and so on, the output got reasonably sane — but the last thing left on the table was token consumption.

This post is the record of starting from that token reduction, moving into re-tightening the report content, improving the formatting, and finally deciding that "this isn't really a prototype anymore — I can call it an initial version."

In dev-labo terms, the work of moving a tool from the "build phase" to the "let others use it" phase. The loop hasn't closed. But there was a shape to the stopping point: whose loop should this be?

Three token-reduction tactics

At the end of the last post, I listed three tactics.

I bundled these into one issue and landed them in a single PR.

Tactic B: get on top of prompt caching

GPT-5-class models cache a long prefix when it matches exactly and discount it (the threshold depends on the model). This tool's detection call sends a long prompt that includes the playbook for all seven patterns — exactly the kind of payload that should benefit from caching.

The snag was that the initial implementation had the playbook description (several hundred lines) in the user prompt. The PR diff sits at the end of the user prompt, so every call changes the user prompt as a whole. Even if the system prompt was fixed, the "long enough contiguous prefix match" condition for a cache hit wasn't always being met.

The natural fix is to put the static bits in the system prompt and keep only the volatile PR diff in the user prompt. That's what I did.

There's a small tradeoff: the in-memory system-prompt string on PatternDetector grows. But per-call outbound tokens shrink. Optimizing for cost wins here. I also added a test that verifies the system prompt is byte-identical across multiple calls — so if a future change breaks the cache condition, CI catches it.

Tactic C: drop the diff from consolidation

consolidation takes the per-pattern recipes and merges them into a final report. I'd been passing the PR diff into it, but thinking about it properly, the per-pattern recipes already contain the concrete identifiers. What consolidation reads is the recipe body, not the diff itself.

So I dropped diff_text from consolidate()'s signature. Breaking change — I updated every caller and every test at the same time.

No hesitation on this one. When I asked "does consolidation actually need the diff?" the honest answer was "I passed it just in case; the merge logic wasn't using it." Arguments passed 'just in case' are usually future deletion candidates — a rule of thumb that held again here.

Tactic A: narrow generation's diff

This was the big one. generation is called per pattern, so the diff is sent once per pattern. A PR that matches five patterns sends the diff five times. The bigger the diff, the more linearly the waste grows.

But the files each pattern actually cares about are a subset of the diff. detection already returns matched_files, so we can pass only the hunks from those files to the generation step.

I made one small design call here: include neighboring files in the same directory too. If "enumerate callers" (B1) matched services/user_service.py, the actual callers might live in a different directory — api/user_controller.py or wherever. If I strictly narrow to matched_files, caller files get dropped.

"So why not just send everything to be safe?" A reasonable question. What I remembered here was the verb discipline from the last post. The generator only produces a checklist; the developer is the one who actually enumerates the callers. If narrowing means a file doesn't appear in the diff sent to generation, that caller is still going to get written into the checklist as "enumerated" — and the developer checks it at their end. So we can reduce the diff without degrading checklist quality.

As a safety net, if the narrowed result comes out empty, we fall back to the full diff (with a warning log). That one I kept conservative.

Combined result

Running it, the generation diff size came out 73–78% smaller per pattern.

Total token consumption dropped to about 60–70% of the prior number. Consolidation's prompt_tokens fell from the tens of thousands into the 3000s.

Prompt caching hits were trending well in the logs too, so the heaviest part of the pipeline got cheaper.

After the reduction, quality issues resurfaced

Thinking "OK, I can run this now," I sat down and actually read the fresh checklists carefully. And new uneasiness showed up.

Tokens had dropped, but the shape of the items bothered me. Last round I'd squashed "hallucination" and "code-review-ness." This round was a different species of problem.

Three things stood out.

1. Compound items

One of the five items had crammed four concerns into a single long sentence: "description of the change + downstream impact survey + prompt-log route + alerting impact." You couldn't make a clean "done / not done" binary call on a single item.

It looked like a side effect of trying to stay under the six-item cap by compressing multiple concerns into one item. The rule was being obeyed; obeying it was causing harm.

2. Over-generalization

There was an item saying "enumerate across dev/staging/production." But this tool has no environment separation. The diff mentioned nothing about environments. The LLM had reached for "risks that are generally worth worrying about" — with no PR-specific grounding.

Last round I'd added grounding against hallucination, so non-existent identifiers had mostly stopped appearing. But this new failure was a "confirmation item that presumes a mechanism that doesn't exist." Because there were no identifiers in it, it was generic prose, and it slipped past the grounding filter.

3. Unsubstituted placeholders

Items like <function_name> were making it to output — literal angle-bracket placeholders. They should have been replaced with concrete names (e.g., extract_diff_file_list). Just a formatting-instruction miss — a clear, fixable issue.

Four tactics for the fixes

Split into its own issue and worked through in order.

Tactic B had a counter-intuitive worry: "if you lower the cap, doesn't it push more compression, not less?" Hard to predict the net effect. So I started with the low-risk A+D, and saved B for last. Leaving the tactic whose effect I can't predict until the end is a habit I've built — I can see the prior tactics' effects before adding the unpredictable one.

One more thing that clicked in my head while writing these out: lowering the cap and forbidding compression point in different directions. I wanted to push the model toward deletion (drop the weakest item) rather than compression (cram more into one). Tactic A (no compound items) says "don't compress." Tactic B (cap = 5) says "but you're allowed to drop one in exchange." Together, they make "drop the weakest item" the preferred output, not "compress everything to fit."

Results, and an unsolved problem left behind

Post-fix run:

Stuff like "dev/staging/production" that isn't in the diff stopped showing up.

But the meta-PR-specific hallucination I mentioned last post was still there. When the tool runs on a PR that edits the tool's own prompt files, the LLM invents references like "downstream detection rules" or "checks generated from CI job environment differences" — things that don't exist outside this tool.

I'd parked this as a "turtles all the way down" problem last time. The root cause is writing the prompts and being the subject the prompts process happen in the same repo. A structural issue.

That forced a call: "squash this meta-PR hallucination first, then draw the line?" or "treat this level as an 'initial version' and start using it?"

I went with the second. Three reasons.

(3) was the decisive one. If I keep tightening solo, the optimizations start drifting toward the desk. "Something felt off on my PR, so I fixed it" — repeat enough times and you start catching uneasiness that wouldn't even surface on another developer's PR.

Polish: the report's format

The last thing left was formatting. The information in the checklist was correct. But it was hard to read.

- [ ] Enumerate callers of recipe_consolidator.py's CHECKLIST_MAX_ITEMS(5) and
  cap_checklist_items within the repo
  (e.g., git grep -n "CHECKLIST_MAX_ITEMS\|cap_checklist_items")
  — downstream: PR comment generation pipeline / recipe_cli / CI jobs, etc.

One line crams "target of the change + downstream impact + investigation command" together and makes it hard to scan. A checklist ought to let your eyes skim across item headings and tick "yeah, confirmed this." At this length, each item takes real time to read.

I moved to a two-tier format: a heading line plus sub-lines.

- [ ] Enumerated callers of `CHECKLIST_MAX_ITEMS` / `cap_checklist_items`
  - Downstream: PR comment generation pipeline / recipe_cli / CI jobs
  - How to check: `git grep -n "CHECKLIST_MAX_ITEMS\|cap_checklist_items"`

Read only the heading line and you get the gist. Downstream impact and investigation command go under it as sub-lines. Identifiers go in backticks so they read distinctly from natural prose.

I wrote this format — and the "wrap identifiers in backticks" rule — into the prompt.

A small bug on the code side

I hit a small bug at this stage. cap_checklist_items(), which enforces the item count limit, was implemented to only count and drop the parent - [ ] lines. Sub-lines were catch-alled through.

So when a parent was dropped, the orphaned sub-lines stayed behind, floating with no owner. A genuinely strange state.

The fix added a dropping_subitems flag: after a parent is dropped, subsequent indented lines get dropped together.

if _CHECKLIST_ITEM_RE.match(line):
    if item_count < max_items:
        result.append(line)
        item_count += 1
        dropping_subitems = False
    else:
        dropped += 1
        dropping_subitems = True
    continue
if dropping_subitems and line.startswith((" ", "\t")):
    continue
result.append(line)

Writing the fix, I thought: "this is a bug that couldn't have existed before the two-tier format." Single-tier format made it impossible.

Lesson: if you change a format, revisit the code that handles that format at the same time. Obvious in the abstract. I caught myself assuming "a prompt-only change will be enough" and had to correct course.

A small detour along the way

Around this time, Docker Desktop stopped responding for a while, and I couldn't run pytest locally. docker version hung. docker ps hung. The Docker Desktop process was running; the CLI just wasn't getting through. Stuck in something like an "Engine Starting" state, cause unknown.

I confirmed correctness by tracing the logic and committed first. "Run the tests once Docker Engine is back" — that's the tradeoff. In a situation where chasing perfection stalls you, pick the route that doesn't stall. Docker eventually came back, tests passed, and the post-merge Jenkins run produced the expected output — so no harm done.

The habit of separating procedural perfection from actual progress is probably something I picked up from my infra days.

The judgment of "draw the line here"

Looking at the two-tier-formatted report coming out as intended, I decided "this isn't a prototype anymore — I can ship it as an initial version."

There's no absolute criterion for drawing the line. But I want to put into words what I actually used as the basis for saying "this is V1."

(5) weighed the most.

The difference between prototype and initial version, for me, was "am I tuning alone, or am I pulling others into the loop?" Up to this point, I'd been alone with the prompts — testing on my own PRs, picking up my own uneasiness. From here, other people use it on their PRs, pick up uneasiness from their angle. Same "tightening," but the tools of the tightening (whose eyes, on what PRs, articulating what kind of uneasiness) are completely different.

"Every lap around the loop reveals the next uneasiness," as I wrote last time, still holds. The loop doesn't close. But there was a shape to the stopping point: whose loop should this be?

Looking back at the whole thing

Starting from the "cognitive-axes" idea (B1–B7), through grounding, checklist constraints, verb discipline, the three token-reduction tactics, item-quality fixes, placeholder bans, and the two-tier format. Counted in PRs, the major ones come out to about eight.

This wasn't a one-shot build. You use it, pick up the uneasiness that surfaces, and squash it on both the prompt and code sides. Somewhere along the way, I stopped thinking of it as "done." There's always one more uneasiness queued up. That's just the nature of the thing — you learn to live with it.

But the call to "stop here for now" is a separate thing from "done." A decision to move from 'me tightening alone' to 'others using it'. "Done" and "drawing a line" can be thought about separately.

Another thing I kept feeling throughout this project: building an LLM-embedded tool firms up concepts in your head as you go. Last post I wrote about "the difference between investigation and review" and "the flavors of empty" crystallizing. This round:

These distinctions sharpened one by one as I wrote the code. Concepts I'd kept vague got forced into articulation by implementation constraints. Building things drags introspection in with it — that feeling, again, was there the whole time.

Next phase, and the series's stopping point

From here, the center of gravity shifts to operational feedback. What uneasiness shows up when the tool runs on other people's PRs? Do the checklists actually help, or do they get skipped? Does meta-PR hallucination show up on normal PRs, or not?

These are questions only real use can answer. So I stop here and wait.

Some likely next concerns:

At that depth, it's no longer an LLM problem — it's a does the tool take root in the workflow problem. A different kind of difficulty. But without the observability to see that, there's nowhere for an initial version to go.

Phase 0 (get it running) → Phase 1 (quality of silence) → Phase 2 (token reduction and polish). This series stops here. The recipe generator moved from prototype to initial version. The phase of growing it through use starts now.

The call to draw a line and the call to pronounce something done are different things. That's probably the biggest lesson from this round.

Further reading: on shipping and drawing the line

A classic that puts the theme not as "done" but as "delivering something you can trust," and lays out pipelines and shipping judgment systematically. I wrote this post at the scale of a personal tool, but the same kind of judgment recurs at larger scales too.