Seven Principles for Designing LLM-Embedded Tools: An 11-Day Decision Log from the CARE Project

What this series was actually doing

The starting point was 9 incidents from a single quarter. Lining them up and classifying root causes turned up something odd. Not a single incident was preventable by static analysis. There were almost no typos-as-bugs. The thread running through all 9 was this shape of miss: "Nobody noticed that this code propagates over there."

ORM class names don't match table names (ProjectUsage → project_usage), so grep on the table name turns up nothing. Multiple repositories mean no single search surface covers it. Batch jobs and cross-service references keep the dependency invisible from the app code. That type of cognitive blind spot.

There was no off-the-shelf solution. A cross-product × cross-language × includes-data-source-outside-code problem sits on the boundary of several tool categories; no single SaaS swoops in and fixes it.

So I started building. CARE (Context-Aware Review Engine) — a tool that reads a PR's changes, detects the cognitive blind spots reviewers tend to miss, and posts a confirmation checklist as a PR comment. Under the hood, it uses an LLM (Azure OpenAI's GPT-5-class models) in a three-stage pipeline — detection / generation / consolidation — to narrow down the "where should you look?" checklist.

The first design got parked partway through. The autonomous-agent approach that looked promising on a 9-incident sample had its ROI drop to borderline on the full 94-incident data. The day after I parked it, a new idea clicked: return the recipe for investigation, not the investigation result. That became the "investigation recipe" approach. From there came Phase 0 (get it running) → Phase 1 (quality of silence) → Phase 2 (token reduction and shipping decision), ending with "this is an initial version, not a prototype." The full arc — 8 posts over 11 days.

Each post lives in the dev-labo (development lab) category as a trial-and-error log. But finishing the run, the logs turned out to contain a set of principles worth extracting. This post is me narrowing them down to seven and systematizing them. For people who are about to build an LLM-embedded tool of their own — or who are stuck partway through — this is the one-stop version I wish I'd had. The full list of 8 posts sits at the end; read those for the episodes behind each principle.

1. Separate the cognitive problem from the execution problem

The single fact that ran through all 9 incidents was this.

"It was investigable. I just didn't recognize that it needed investigation."

Dependencies you could find with grep. ALTER-level column changes that propagate to a batch job in a totally different repo. Technically all of them were investigable problems.

The moment that distinction is clear, the countermeasure's design changes completely. "Investigation automation" tries to solve the execution problem; "a system that tells you what to investigate" solves the recognition problem. The latter is lighter, scales better, and is actually stronger against cognitive blind spots — like "this change's effect on the DB or external integrations" which you can't see from code alone.

The first question to ask when designing an LLM tool: is this a cognitive problem or an execution problem? Trying to solve both at once usually makes the tool too heavy for anyone to actually use. In fact, the CARE autonomous-agent design got parked precisely because it was trying to solve both. The post-pivot "investigation recipe" design committed to solving only the cognitive problem, and the result runs significantly lighter and faster against the same underlying issue.

LLMs can be used for either problem, which makes it easy to design greedily into both. Stopping to articulate "which one am I actually solving?" was the first move that mattered.

2. Draw the line between the theoretical ceiling and the real-data floor

While hammering out CARE's design, I hit the question: "can static analysis track change impact exhaustively at all?" The research answer was blunt. Complete automation is impossible in principle.

Precisely deciding a program's behavior statically reduces, via Rice's theorem, to an undecidable problem. Forward Slice's exhaustive impact tracking on a dynamic language, if pushed to "zero false negatives (Sound)," reports roughly the entire program as the impact scope. Even Google doesn't have a perfect inter-procedural analysis infrastructure; the industry standard is "Soundy" (nearly-sound) approximations.

Once you know about this kind of "perfection is impossible" theoretical ceiling, it looks like a binary: "give up" vs. "do it anyway." Both were wrong answers.

The right question was, "where between the ceiling and the floor do you draw the line?" The ceiling is Rice's theorem. The floor is "the 9 incidents that actually happened." Somewhere between those is "good enough." The material for drawing the line isn't theory — it's the real data you have in hand.

When I laid the 9 incidents out, the boundary between "capturable" and "structurally impossible" became concrete. Dependencies traceable from PR-originated changes: capturable. Pre-defined concern areas: capturable. Work outside the PR (data migrations, manual ops), runtime-input-dependent issues: structurally out of scope. Making that boundary explicit gave us a defensible "not perfect, but necessary-and-sufficient for the current problem."

When building an LLM tool, hitting a "limit" in a paper or a theoretical argument isn't a reason to stop the design. Your real data is the starting point for line-drawing. Theory only tells you where the ceiling is; the line your project needs sits much lower, shaped by the real data in front of you.

3. Don't let the LLM judge — let it gather evidence

The most important boundary I drew while designing the CARE agent was this: the LLM's role is evidence gathering, not judgment.

Concretely, the agent's inputs and outputs were locked down like this:

Why hold this boundary so hard? Because if you let the LLM judge too, hallucination gets laundered into "judgment." "This change is safe," the LLM declares — and then a miss happens. Accountability becomes murky. "The AI said it was safe" doesn't hold up as a review log.

Keep the LLM at "there's evidence here / this is unverified," and hallucination surfaces as "inventing evidence that isn't there" — which the grounding filter (_is_grounded() in Phase 1) can drop. The filter checks whether every matched_files entry actually exists in the diff, and whether reasoning contains concrete identifiers from the diff. Anything that fails is never treated as "evidence." In the prompt you ask "collect evidence"; in the code you enforce "nothing that isn't evidence passes." Together they make "never decide on the LLM's words alone" into a lived property of the system.

This isn't about underestimating what LLMs can do — it's a design call driven by the nature of the information the LLM handles. Don't be vague about how uncertain information is treated. This is the LLM-era version of the old "separate fact from interpretation" / "separate data from decisions" rule.

4. If the sample changes, the design changes

I'd hardened CARE's design around 9 incidents. Then the full set of 94 incidents showed up. Re-running the analysis on the same framework flipped several design premises at once.

Specifically:

Seeing this, I rebuilt the design philosophy. YAML-structured rules (with keys like trigger / question / scope / depth) got dropped in favor of Markdown narratives (free-form postmortem-style text). Reason: one-off patterns can't be pre-defined in YAML. Writing "where to notice" in prose overlaps naturally with the existing incident-response workflow. I also swapped up-front scope control for back-end guardrails (max tool calls, max token spend, timeout) — because tightening up-front kills the autonomy that's an agent's whole point.

The redesign then fed into a fresh ROI calculation, which led to parking the autonomous-agent approach altogether. The next day, the pivot to "investigation recipe" landed. So the 94-incident sample didn't just reshape the design — it reshaped the product direction itself.

The principle is simple. Don't finalize a design on a small sample. When data grows, be prepared to redesign. And conversely: don't over-polish your design on the first sample. A prototype you plan to break later is more data-update-friendly than one engineered to last.

This isn't "wait for all 94 before designing." It's "design with the assumption that the sample will grow, and deliberately keep zones of the design unset." LLM tools grow more through data than through construction — build an architecture that accommodates that property.

5. The prompt isn't a contract — the code is

This was the principle that hit hardest in Phase 1.

Write "checklists must be 6 items or fewer" into the prompt, and the model returns 11 items depending on its mood that day. Write "don't include " inside JSON reasoning," and it embeds " inside backtick-wrapped code snippets while trying to be helpful — breaking the JSON parse. Write "no generic items that aren't PR-specific," and "confirmed all tests pass" (true on every PR) sneaks in anyway.

The prompt is a request. On cooperative days, it goes through. On other days, it doesn't.

The fix: make the code the contract. Whatever the prompt asks for, the code enforces as "nothing else passes."

def cap_checklist_items(content: str, max_items: int) -> str:
    """Hard-truncate - [ ] items under the checklist heading to at most max_items."""

def _is_grounded(match, diff_files, diff_text) -> tuple[bool, str]:
    """Verify matched_files exist in diff and reasoning contains diff-derived identifiers."""

The prompt says "please do this." The code says "anything else is dropped." Both express the same spec.

The key is not relying on the prompt alone. LLM-embedded tool quality is proportional to how thick this code-side guard is. Prompt-only means quality rides directly on the model's mood that day — and you get inconsistent behavior across model versions when you swap.

Conversely, if the code enforces everything and the prompt describes nothing, the model's output shape and the code's accepted shape drift further apart, and drop rates balloon. Request (prompt) and contract (code) as a two-layer design is what actually worked in practice.

6. Noise and silence are asymmetric

This was the principle that, counterintuitively, landed hardest from building a PR-comment tool.

In Phase 1, I froze reading the first comment the tool posted. Seven items on the checklist — and mixed in among them were "obtained legal/security approval" and "confirmed no overreliance on test-covered cases." Things that could be said on any PR.

Reading them, I realized: every time the tool emits this kind of noise, reviewers accumulate an irreversible learning: "the comments from this tool can be skimmed." Once that happens, even a genuinely important pointer later goes unread.

Meanwhile, the time the tool stays silent isn't a negative — it's a value. A quiet tool is quietly signaling "nothing worth flagging," without eroding trust through noise.

Acting on this, Phase 1 added a whole stack of defenses:

Every one of these pointed at the same thing: raise the quality of the silence.

Generalized: bias the cost of the "emit vs. don't emit" call toward the emitter. Same structure in Slack notifications, alerts, or PR comments. Treating silence as an asset is a design principle that applies broadly — but it's especially valuable in LLM tools, because LLMs come with a built-in urge to produce text. The stronger the temptation to speak, the more valuable the discipline of staying silent.

7. "Done" and "drawing the line" are different calls

In Phase 2, I decided "this is an initial version, not a prototype anymore."

At that point, the tool wasn't done. The meta-PR hallucination was still unresolved — when the tool runs on a PR that edits its own prompt files, the LLM confuses the "bad example / good example" pairs inside the prompt for the PR's actual changes, and starts generating checklists from them. This is structural: "writing the prompts" and "being the subject the prompts process" happen in the same repo. A real fix has a big blast radius.

But I realized drawing the line can be decided on criteria separate from "done." Here are the five I used, in writing:

(5) was the essence. The decision to move from tuning alone to having others use it. Same word — "tuning" — but a design change in whose loop this is. Keep tuning solo, and the optimizations drift toward the desk. You might pick up uneasiness other developers never experience, and bend the design to fix it.

LLM tools have a large portion of their growth in "use and cultivate after shipping." Waiting for "done before shipping" means never shipping. Keep a set of criteria for drawing the line that is separate from done. That was the key to keeping the shipping decision from getting stuck.

Not an implementation-quality question. Reframe "drawing the line" as a design change about whose feedback feeds the next iteration. Done is about the future; drawing the line is about now. That was the last principle.

Closing: this was as much introspection as tool-building

Laying out the seven, I notice that more than half aren't about "how to handle LLMs" — they're about "how to make decisions myself." Separate cognition from execution. Draw lines between ceilings and floors. Rewrite designs as the sample grows. Separate "done" from "drawing the line." None of these are LLM-specific. They're LLM-flavored applications of older principles.

The most distinctive thing about building an LLM-embedded tool, for me, was that concepts in your head firm up as you write. "Cognition vs. execution." "Evidence vs. judgment." "Noise vs. silence." "Done vs. drawing the line." These distinctions crystallized bit by bit while writing prompts, writing code. Building things and introspection were tangled up more than I'd expected, across these 11 days.

My loop pauses here, and the next loop moves to other people's hands. If these seven principles make the landscape a little brighter for people walking the same road, that's the happiest kind of stopping point.

Further reading

For readers who want to go deeper on the principles in this post — from either the technical side or the design-judgment side — two books I kept coming back to while writing.

From the LLM / foundation-model side

Grounding, the prompt-and-code two-layer discipline, token economy — the concerns raised in this post are recurring ones whenever you build applications on top of foundation models. This book gives them a systematic treatment.

From the software-design-judgment side

The closing claim of this post was that more than half of the seven are LLM-flavored applications of older design principles. For tracing the judgment process of architecture decisions more broadly, this book widens the view.

The full 8-post series