Silence as a Signal: Tuning an LLM PR-Review Tool by the Quality of Its Quiet

In the last post, I wrote up the point where Phase 0 of my impact-analysis "recipe generator" started working.

「ハルシネーション対策」を実装する前に、設計変更で消えていた──Phase 0 検証で起きた3つのピボット

影響範囲調査レシピジェネレーターの Phase 0 遡及テスト中に、設計そのものを3回作り直した記録。最後には Phase 1 で予定していた「ハルシネーション対策」Issue が、一行もコードを書かずに close になった。

Tech

開発ラボ

I said it "worked," but honestly, what actually worked was at the level of "the code didn't crash and a PR comment got posted." Once I started using it on my own PRs, the quality of the comments gave me pause in several different moments.

This post is the record of the trial-and-error that followed, under the banner of "raise the quality of the silence" — pushing on both the prompt side and the code side. I'm still mid-journey, but the traps you fall into when using an LLM as a PR-review aid came into fairly clean focus along the way.

Background: what the tool does

To recap quickly: this tool reads a PR's diff, detects the "blind spots" reviewers tend to miss (seven patterns, B1 through B7), and posts a checklist as a PR comment so those points get confirmed.

The pipeline looks like this.

Detection, generation, and consolidation each call Azure OpenAI. Essentially, it's a three-stage LLM pipeline that narrows down the "where should you look?" checklist.

Technically it ran. But whether the comments are worth reading is a different question.

First uneasy moment: the signs of going through the motions

Reading the comment posted on one particular PR, I froze for a second. Summarized, it went roughly like this.

- External contract: the path that sends PR metadata to an external LLM has been enabled, creating risks of confidential/personal-information leakage and external-transmission policy violations
- External spec dependency: a change in prompt format risks breaking downstream parsers and tests

- [ ] Obtained legal/security approval
- [ ] Verified references from external clients and other repos
- [ ] Confirmed no overreliance on test-covered cases

A seven-item checklist. The moment I read it, I thought: this is bad.

Four things bothered me.

That was when it really hit me: noise and genuine insight are not symmetric. Every time you emit noise, reviewers accumulate a learned habit of "the comments from this tool can be skimmed." Once that happens, when you finally post something genuinely important, nobody reads it anymore.

The real sweet spot for PR-comment tools, I think, is this: the moments it stays silent are when it is most trusted.

Fix 1: force grounding

The first thing I tackled was hallucination.

I wanted to stop vague detections like "kind of feels like B3" and the invention of blind-spot names that don't exist. So I put a grounding requirement on every detection result.

The rule is simple. A detection is dropped unless it satisfies both:

I wrote the same rule into the prompt and added a code-side filter too — two layers.

def _is_grounded(self, match: PatternMatch, diff_files: set[str], diff_text_lower: str) -> tuple[bool, str]:
    # Validate matched_files
    invalid = [f for f in match.matched_files if f not in diff_files]
    if invalid:
        return False, f"matched_files not in diff: {invalid}"

    reasoning = (match.reasoning or "").strip()
    if not reasoning:
        return False, "empty reasoning"

    # Does reasoning reference a file basename?
    basenames = {f.rsplit("/", 1)[-1].lower() for f in diff_files}
    reason_lower = reasoning.lower()
    if any(b and b in reason_lower for b in basenames):
        return True, "file basename referenced"

    # Does an identifier in reasoning appear in the diff body?
    tokens = {t.lower() for t in _IDENTIFIER_RE.findall(reasoning) if len(t) >= 3}
    for token in tokens:
        if token in diff_text_lower:
            return True, f"identifier {token} referenced"

    return False, "no concrete diff anchor"

Implemented, ran it again. Result: "no matching patterns."

All three detections had been dropped.

Checking the logs, the LLM had detected B6 (high), B1 (medium), B7 (low). But none of them made it through the grounding filter.

Debugging by cross-referencing the saved prompt log JSON with the implementation, the culprit was CRLF line endings. The tool runs on Jenkins, so the diff text had \r\n in it, and matched_files paths ended up with a trailing \r. "src/foo.py\r" != "src/foo.py" was rejecting all of them.

Lesson: distinguish "correct silence" from "over-filtering"

Looking at a quieter report, it's tempting to just go "great, it's clean now." But stopping for a beat here matters.

Don't blur these three together. You can't tune the tool unless you know which state you're in, so I started logging both the filter's input and every drop reason at INFO.

logger.info(
    "Grounding filter input: candidates=%d diff_files=%d",
    len(matches), len(diff_files)
)
# ...
logger.info(
    "Detection result dropped (ungrounded) pattern=%s reason=%s",
    match.pattern_id, reason
)

Fix 2: checklist caps and forbidden categories

Grounding cut down on fabrication. But then I hit PRs with 11-item checklists. The consolidation prompt says "4–6 items" and the model wasn't respecting it.

Two-layer fix again.

Prompt side: enumerate the forbidden categories explicitly.

- **Total checklist items must be 6 or fewer.** If you'd produce 7+,
  drop items in order of weakest PR-specificity.
- **Do not output generic items that aren't specific to this PR.**
  The following are forbidden:
  - "Ran the tests and confirmed all pass," "Updated the docs" —
    anything that would be true on any PR
  - "Don't over-rely," "Review carefully," "Pay attention" —
    self-soothing phrases with no concrete action
  - Legal approval, organizational policy compliance, compliance sign-off —
    macro concerns the tool and individual developer cannot judge
  - "References in other repos," "external clients" —
    speculation you can't ground in the diff

Code side: as a safety net for when the prompt is ignored, cap_checklist_items() enforces a hard limit at the end.

def cap_checklist_items(content: str, max_items: int) -> str:
    """Trim - [ ] items under the checklist heading to at most max_items."""
    # Find the checklist heading
    heading_match = re.search(r"^##\s*Checklist\s*$", content, re.MULTILINE)
    if not heading_match:
        return content
    # ... count items after the heading and drop the overflow

Don't rely on the prompt alone — have the code be the last gate. This is a principle I feel strongly about when you build tools that embed LLMs. The prompt is a request ("please output in this shape"); the code is a contract ("only this shape passes"). Without both, quality tracks the model's mood that day.

Fix 3: the real culprit behind JSON parse failures

After running for a while, one day I got "no matching patterns" again. But the log told a weirder story.

WARNING - Failed to parse detection response:
  Expecting ',' delimiter: line 20 column 116 (char 1284)

A JSON parse failure, dropped. Checking the saved prompt log, the reasoning field contained a string like this:

"reasoning": "The prompt format in recipe_generator.py is `\"pr={pr_number}:{pr_title}:..."

See it? Inside the reasoning string, there's a backtick-wrapped code snippet, and that snippet contains ". The LLM forgot to escape " inside the JSON string, breaking the string boundary.

This is a classic LLM-JSON failure mode. For a fix, I added JSON-writing rules to the system prompt.

## JSON string rules (strict)
- **Do not include `"` inside `reasoning`.**
  Use backticks (\` ... \`) or Japanese quotes (「...」) for code
  examples or string literals.
- **Do not include newlines (`\n`) or backslashes inside `reasoning`.**
  Write a single concise sentence in natural prose.
- Only produce strings that can be passed directly to `json.loads()`.
  Forgotten escapes get dropped downstream immediately.

Alongside that, I raised max_completion_tokens from 3000 to 8000. GPT-5-class models consume the completion budget with internal reasoning tokens too, so 3000 would sometimes get spent on reasoning and return empty, or cut the JSON off mid-stream.

That one came from a model-specific quirk — a trap you'd walk straight into from "standard OpenAI intuition."

Fix 4: raise the quality of the silence

When grounding is working well, you end up with PRs where the comment is just the single line "no matching patterns." That's a little too little information.

But if you tell the prompt "say something," the tail starts wagging: "no blind spots, but watch out for X anyway" — and the noise comes right back.

After thinking about it, I settled on branching to a dedicated prompt for the zero-detection case.

## Result: no matching blind spots detected

### Summary of changes in this PR
Summarize the diff in 1–2 sentences. Include PR-specific file names,
function names, and identifiers — be concrete.

### Notes on the judgment
In 1–2 sentences, explain why none of B1–B7 applied.

The constraints are tight.

Also, do not branch to this prompt on API errors or JSON parse failures. Asking the LLM to confabulate a summary in accident cases is dangerous. For those, fall back to the plain "no matching patterns" message.

def _build_no_match_output(outcome, diff_text, ...) -> str:
    """Build output for the zero-detection case. LLM summary on success, plain message on accidents."""
    if not outcome.successful:
        logger.info("Skipping no-match summary because detection failed")
        return "No matching patterns\n"
    # ... generate the silence explanation via LLM

To distinguish success from failure, I changed detection's return value from a (list, usage) tuple to a DetectionOutcome(matches, usage, successful) dataclass. An empty array has two flavors — "nothing detected" vs. "detection failed" — and encoding that distinction in the type system is the robust move.

Fix 5: constrain the verbs

After the fixes so far, noise had dropped a lot. But rereading the checklists, a new uneasy feeling surfaced.

- [ ] Confirmed whether chmod 777 is necessary, or 770 would suffice
- [ ] Confirmed the archive location and visibility are appropriate
- [ ] Confirmed that savellmcall_log has a masking mechanism

These are "code review," not "impact analysis."

The verb was decisive.

The tool's purpose is the former. The latter is for the reviewer to judge by reading the PR. A tool enumerating those points doesn't help — it actually crowds out the dialogue the author and reviewer should be having.

I wrote the constraint into the prompt.

### Verbs to use (tracing)
- **enumerated** / **confirmed every reference** / **verified affected downstream**
  / **confirmed the impact on downstream** / **traced the callers**
  / **listed the consumers**

### Verbs and forms NOT to use (code-review)
- "confirmed whether X is necessary," "confirmed whether X is valid,"
  "confirmed whether X is appropriate"
- "reviewed the implementation of X," "confirmed the absence of X"

### Forbidden item categories
- **Design-decision justification**: "Why this value?" "Were alternatives considered?"
- **Organizational policy fit**: "Does this meet audit requirements?"
  "Has legal approved?"
- **Implementation-gap pointers**: "Is there a masking mechanism?"
  (These are improvement proposals, not impact analysis.)

Next run, the checklists came out noticeably more honest.

- [ ] Enumerated — per caller — every generator that loads the template
and every downstream that consumes it (CI report aggregators,
doc-generation tools, etc.), and listed the consumers

The verbs are "enumerated" and "listed." Both "what changed" and "where it propagates" are present. That's the shape I wanted.

An unexpected trap: LLM self-reference

Right after putting that rule in, I ran into a delightfully dumb failure.

I ran the tool on a PR that edited the prompt template (.md files), and the checklist started talking about chmod 777.

That PR touched no chmod 777 anywhere.

Looking at the prompt log, I saw it. My rule had included this example:

- Bad: `- [ ] Confirmed whether chmod 777 is necessary` (re-litigates design)
- Good: `- [ ] Confirmed what becomes externally writable because of
  the PROMPT_LOG_DIR permission change (chmod 777)`

The LLM had mistaken this example for the PR's actual changes.

A meta-PR-specific trap. When you build a tool that rewrites its own prompts, you get this self-referential situation.

I sat back and laughed for a second. When you develop an AI tool, the LLM does not distinguish "what kind of text is this text I'm handling right now." That was the moment the assumption that it would distinguish collapsed in me.

I don't have a clean fix yet. What I've got so far:

But both introduce new example text that can leak back into a future diff — a hall of mirrors. This kind of self-reference happens fundamentally because "writing the prompts" and "being the subject that the prompts process" live in the same repo. I'm starting to suspect the structure itself needs rethinking.

The whole loop so far

Boiled down, the trial-and-error looked like this.

Every lap around the loop reveals the next uneasy feeling. This is clearly not the kind of work that ends when you write the doc. Both the prompt and the code have to be grown by using them.

What I'm leaving: token consumption

Quality has come up a lot, but the last thing left is token consumption.

Right now, the diff is sent at:

If a PR matches three patterns, the diff goes over the wire five times; with five patterns, seven. The bigger the diff, the more linearly the waste grows.

A few things I want to try next.

I'll split this out into a separate issue and pick it up in the next session.

Retrospective: principles that worked when embedding LLMs

Through this round of quality fixes, a few principles crystallized for me.

1. Constrain with the prompt and the code — both

The prompt is a request ("please output in this shape"); the code is a contract ("only this shape passes"). Without both, quality tracks the model's mood that day.

2. "Empty" has flavors

Zero detections and zero-after-failure are completely different states. Distinguishing them in the type system is the most reliable move.

3. Silence is value

Every noisy comment teaches reviewers "skim is fine." For PR-comment tools, staying quiet is when they're most trusted.

4. Verbs set the design

"Enumerate" and "confirm whether X is valid" imply totally different tool roles. Just constraining verbs in the prompt flips the character of the output.

5. Use what you're building on yourself

Traps like "the prompt-editing PR made the prompt blow up self-referentially" simply don't surface unless you dogfood.

Further reading: designing applications on foundation models

Grounding, the prompt-and-code two-layer discipline, token economy — the concerns raised in this post recur across applications built on top of foundation models. If you'd like to zoom out and pick up the concepts more systematically, this book pairs well with these notes.

Writing this up, the thing I keep coming back to is: the quality of an LLM-embedded tool is determined not by "the LLM's performance" but by "the design of how you live with the LLM." Less prompt engineering, more "prompt operations design," if that label makes sense.

And the process of honing it — that itself becomes the act of articulating "what do I want this tool to detect, and what do I want it to ignore?" and "what is this tool for in the first place?" As I wrote the code, fuzzy concepts in my head — like "the difference between investigation and review" — started to sharpen their edges. Making things and introspection are tangled up more than I'd thought.

Token reduction has to assume quality doesn't regress, so it'll need more care than the previous rounds. Continuing in the next post.