SRE Work: From Problem Discovery to Permanent Solution and Results

Introduction

We've been tracking the Jenkins EFS issue through three articles:

And today, we have results from our solution.

Honestly, I didn't expect such clear results. When I saw the graph, I felt a sense of "ah, my hypothesis was correct."

This article reports on post-solution results and explores the SRE workflow process that emerged from this entire response.

Post-Solution Results: Dramatic Improvement

In Part 3, we enabled Shared Library caching. Here's the EFS throughput utilization graph afterward:

Before (left side, 06:00-12:00):

After (12:00 onward):

The 3-hour spikes are from Shared Library cache refresh checks (Refresh time: 180 minutes). In other words, it's working as expected.

Complete Record of Throughput Mode Transitions

Let's organize the five throughput mode transitions we went through during this response.

Timeline

Details and Costs of Each Mode

Why We Changed from Elastic Throughput

Actually, we discovered that Elastic Throughput "costs more than expected."

One day of usage was ~$8. That's about $240/month (~35,000 yen).

In contrast, Provisioned 10 MiB/s costs only ~$72/month (~10,000 yen). Given our current usage pattern (average throughput a few %, max 60%), 10 MiB/s is sufficient.

However, this is just a validation period setting. We plan to eventually return to Burst mode.

Hypothesis Validation and Revision

Let's validate the hypothesis we posed in Part 3.

Original Hypothesis

Did the change to disposable agent mode (1/13) massively increase metadata IOPS?

This hypothesis was partially correct, but not the main culprit.

The Actual Culprit

Shared Library caching was disabled (existed from before)

Simply enabling caching brought throughput utilization down to nearly 0%. This means that Shared Library's full fetch on every build was overwhelmingly consuming metadata IOPS.

Impact of Disposable Agent Mode

So, was disposable agent mode unrelated?

Not quite. The change to disposable agent mode was one factor that accelerated Burst Credit depletion.

These three factors combined caused rapid Burst Credit depletion from 1/13, with symptoms surfacing two weeks later on 1/26.

With only the Shared Library cache disabled, credit might have decreased more slowly. The disposable agent mode change was likely the final push that tipped things over.

The SRE Workflow

Reflecting on this entire response, we can see a clear process in SRE work.

1. Problem Discovery (1/27 morning)

At this stage, understanding "what's happening" is crucial.

2. Emergency Response (1/27 morning)

Emergency decisions prioritize "avoiding worst-case scenarios." Costs can be explained later, but losses from development stoppage are irreversible.

3. Impact Minimization (1/27 afternoon)

In parallel with emergency measures, implement prevention strategies.

4. Root Cause Investigation (1/27-1/30)

Honestly, I got stuck here. When I found tmp_pack_*, I thought "this is the cause," but it was actually just part of the symptoms.

By reviewing graphs chronologically, we reached the true root cause.

5. Permanent Solution (1/30)

Once the root cause is identified, the solution is simple.

6. Effect Measurement (1/30-)

Results prove whether the solution was correct.

7. Retrospective & Knowledge Sharing (this article)

This is surprisingly critical. Don't just solve the problem and move on—verbalize "why it happened" and "how we decided," and apply it next time.

Remaining Issues and Future Plans

Short-term Issues

1. Return to Burst Mode

We're currently running on Provisioned 10 MiB/s, but plan to eventually return to Burst mode.

Before returning, check:

2. Enhanced Monitoring

This is a complete lesson learned. With proper monitoring, this problem could have been detected earlier.

Alerts to configure:

3. Continued Cleanup Job Operation

We created the periodic cleanup job, but need to regularly verify that tmp_pack_* files are actually being deleted.

Long-term Considerations

1. Reconsider Disposable Agent Mode

We're currently continuing disposable agent mode, but the metadata IOPS impact is non-negligible.

Options to consider:

How to balance cost and performance—that's the next challenge.

2. Reconsider EFS Usage

There's also a fundamental question: is placing Jenkins' work directory on EFS optimal?

Alternatives:

However, this would be a major architectural change requiring careful consideration.

Summary: What Is SRE Work?

Looking back, arranging this entire response chronologically revealed some insights.

Technical Learnings

Process Learnings

But what's more important is "how we decided."

Emergency Decision-Making:

Investigation Approach:

Accountability:

These three are the greatest learnings from this response.

When in Doubt, Choose the Harder Path

Finally, on a personal note.

Not knowing about Elastic Throughput was clearly a knowledge gap. The $64 difference is significant.

But I believe the decision itself—"ensuring an environment to continue investigation"—was correct. I regret not knowing, but the decision process can be applied next time.

When in doubt, choose the harder path, the one that leads to growth—this is a principle I always value.

In this response, the easier choice would have been "stop Jenkins and investigate over the weekend." But by choosing the harder path of "continuing operation while investigating," I experienced the complete process of emergency response, cost decisions, and root cause identification.

The $69 tuition might have been expensive. But I believe I gained more than that in learning.

Related Articles

This article is a continuation of the following three articles:

Related Books

For those interested in learning more about infrastructure operations and troubleshooting: