You | TIELEC Blog

Introduction

I was thinking about something the other day.

Security cameras — you never really know how many you need to install before they start being effective. But you still have to set a budget. That feeling is oddly similar to project management.

This idea actually came from an earlier article where I compared SRE investment to police and fire departments — exploring why preventive work tends to be undervalued.

/en/tech/why-sre-investment-undervalued

There was something in that analogy I couldn't quite articulate.

In high-uncertainty projects, we've always tried to build plans that eliminate uncertainty. Detailed WBS documents, precise Gantt charts, exhaustive risk lists. But the more detailed the plan, the higher the cost when reality deviates from it.

So what do we do instead?

This article explores the idea that uncertainty should be structured, not eliminated — and what happens when you bring AI (LLMs) into that equation.

The Fundamental Shift in Thinking

When high-uncertainty projects get stuck, the root cause is often attachment to the plan.

Traditional waterfall thinking assumes: "If the plan is correct, execution will follow the plan." But reality doesn't work that way. Assumptions change, requirements shift, technical walls appear.

This creates a choice: spend your energy defending the plan, or spend it updating the plan while continuing to learn.

In high-uncertainty environments, the latter is overwhelmingly more effective.

The Agile/Scrum approach of "build something small in a short cycle and get feedback" targets exactly this. The question in a sprint retrospective shouldn't be "Did we finish the tasks?" but "What did we learn? How do we adjust the plan?"

The axis of progress reporting changes too. Instead of "completion rate," focus on "risk burn-down rate" — "How many critical assumptions have we validated?" and "What's the biggest remaining risk?" DORA metrics (deployment frequency, lead time, change failure rate, recovery time) are well-known indicators for development team health, but their real purpose is as a tool for seeing what the team is learning.

Another approach I've found personally effective is shifting from "risk management" to "assumption management."

"Predicting risks" tends to become just listing things that haven't happened yet. Instead, asking "What assumptions must hold for this project to succeed?" and then creating an impact × uncertainty matrix to systematically validate the most dangerous assumptions first — I call this "risk-driven prioritization."

The answer to "When will it be done?" changes too:

"We're currently estimating between month X and month Y. If we can validate assumption Z, we can narrow it to the end of month X."

Answer in ranges, not pinpoints. Communicate uncertainty honestly. When project managers stop "performing certainty," stakeholder trust often actually stabilizes.

Using AI as a "Structuring Tool"

Here's where it gets interesting.

Uncertainty management (agile approaches) and AI adoption have been discussed in separate contexts. But when combined, an intriguing structure emerges.

AI makes uncertainty visible. Humans use that visibility to make decisions.

The starting point is an idea I call LLM-based ticket difficulty assessment.

Why Ticket Difficulty Assessment Is Interesting

In development teams, planning proceeds while each person has a different perception of ticket (task) difficulty. Someone thinks "this will take 2 days," starts working, and it takes 5. Bugs appear. But in retrospect, there's no evidence beyond "it felt that way."

When you feed an LLM the ticket description, related components, dependencies, and historical data from similar tickets, it produces a difficulty assessment that gives the team a shared starting point.

Ticket created
　↓
LLM assesses difficulty & bug risk (with rationale)
　↓
Presented as reference during planning
　↓
Developers agree & adjust
　↓
Actual results (time, bugs) recorded
　↓
LLM assessment accuracy continuously improved

There's one critical design consideration here.

The system must not allow "human overrides" of the assessment.

If difficulty assessments are used for performance evaluation, developers gain an incentive to manipulate ticket descriptions. The moment "making it look harder improves my review" becomes true, the system collapses.

That's why it's important to design the system so it doesn't rely solely on ticket descriptions. Use signals that developers don't write: historical bug frequency of related components, number of files changed, code complexity metrics. And position it strictly as a tool for "team design improvement" and "decision support."

Maintaining organizational transparency about the system's purpose is what makes it sustainable.

What Happens When Difficulty Data "Connects"

When ticket difficulty is structured and accumulated as data, it doesn't stop at a single use case. It creates a web of possibilities.

graph TD
    A[Ticket Difficulty Assessment LLM] --> B[Assignment Optimization]
    A --> C[Bug Risk Management]
    A --> D[Growth Roadmaps]
    A --> E[Estimation Accuracy]
    B --> F[Organizational Learning Speed]
    C --> F
    D --> F
    E --> F

Assignment Optimization

Different people are motivated by different types of work. Some thrive on tackling hard problems; others prefer steadily completing smaller tasks.

By matching difficulty data with "each developer's preferred ticket profile," you can move assignment decisions from "experience and intuition" slightly closer to "evidence-based choices."

When this data is used for assignment rather than evaluation, developers have an incentive to answer honestly — because it serves their own interests.

Furthermore, if someone says they prefer high-difficulty tickets but their post-assignment bug rate is high, a skill-preference mismatch becomes visible. Using that as material for 1-on-1s becomes a natural next step.

Development and Organizational Design

The idea is to define engineering grades through "ticket difficulty distribution."

Junior : Can reliably handle difficulty 1-2
Mid    : Can independently handle difficulty 3-4, challenge difficulty 5
Senior : Can reliably handle difficulty 4-5 while making design decisions

Intentionally assigning "slightly stretching" tickets to build experience, and tracking how someone's difficulty distribution has changed over six months — this makes the otherwise invisible aspects of professional growth a little more visible.

Estimation Accuracy

In many teams, estimates only account for implementation time. Testing is a rough guess based on experience, and unexpected issues are handled with vague "buffer" padding.

When difficulty data and historical results accumulate, judgments like this become possible:

Difficulty 3 ticket (based on historical data)
  Implementation : Average 2 days
  Testing        : Average 1 day (50% of implementation)
  Rework         : 30% probability, average 0.5 day additional
  → Estimate     : Plan for a 3-4 day range

The ad-hoc rule of thumb "add generous buffer" transforms into structural rules based on difficulty bands.

The Gap This Concept Is Trying to Fill

Agile preaches "adapt quickly," but doesn't specify the data design for what you're adapting to. DORA metrics propose data, but don't cover individual preferences or growth integration.

"Structuring uncertainty with AI" tries to bridge that gap.

Don't eliminate uncertainty — structure it. Use AI for the structuring. Then feed the accumulated data back into human decisions: assignments, development, and estimation.

You could call it a design philosophy for the division of labor: AI makes uncertainty visible, and humans use that visibility to decide.

What I Haven't Fully Articulated Yet

Honestly, this concept is still at the "thought experiment" stage, with parts that haven't been implemented or validated.

In particular, how to continuously improve LLM assessment accuracy, and how to organizationally maintain the stance that "assessments are just reference points" — these are things that can only be figured out by trying.

But the direction — "uncertainty can't be eliminated, so structure it" — is something I can say with confidence.

You can't know how many security cameras you need for them to be effective. But by recording "where you placed them" and "what changed," each subsequent decision gets a little more accurate.

I think project management works the same way.