The Work After Troubleshooting — A $69 Lesson on Provisioned Throughput

Introduction

Recently, I published an article titled "The Tale of Jenkins Gradually Failing to Clone Git — Fighting EFS Metadata IOPS." It described how EFS metadata IOPS exhaustion caused Jenkins to stop, and how I identified and resolved the issue in about 3 hours of investigation.

Technically, the problem was solved, but that wasn't the end of the story.

The next day, when I opened AWS Cost Explorer, I froze at the number that jumped out at me.

"$69.00"

Honestly, I thought... maybe I messed up.

This article is about cost judgment during emergency response and the accountability that follows. It's the "aftermath" of the troubleshooting article.

What Happened

As I wrote in the previous article, EFS metadata IOPS exhaustion caused Jenkins to effectively stop. And this happened on a Monday morning — the worst possible timing.

To keep Jenkins running while investigating, I made an emergency decision to change the EFS throughput mode to Provisioned Throughput at 300 MiB/s.

This decision resulted in:

Technically, it was successful. However...

The Cost Reality

The next day, after reverting to Elastic Throughput, I became curious about the cost.

I knew that Provisioned Throughput was "expensive." But I thought, for just 26 hours, it wouldn't be that bad.

When I opened Cost Explorer:

Approximately $69

In 26 hours.

If I had left it running for a month, it would have been about $2,160 (over ¥300,000). I was relieved I changed it back quickly, but at the same time, the question arose: "Wasn't there a better option?"

Understanding EFS Pricing

I needed to properly organize the EFS pricing structure. The truth is, during the emergency response, I didn't know about Elastic Throughput as an option.

EFS has three throughput modes:

1. Bursting Throughput (Original Setting)

Pricing: Storage charges only

Throughput is determined by storage capacity. There's a baseline of 50 MiB/s per TiB, with a credit system that accumulates during low load and can be used during high load.

Advantages: No additional charges

Disadvantages: Performance degradation when credits are depleted (the problem we had)

2. Provisioned Throughput (Emergency Response Setting)

Pricing: Storage charges + throughput charges

In the Tokyo region, it's approximately $7.2/month per MiB/s.

For 300 MiB/s:

For 26 hours of use:

The actual measurement was $69, so the baseline throughput was probably deducted.

Advantages: Stable performance, no credits needed

Disadvantages: Very expensive, charged even when not in use

3. Elastic Throughput (The Option I Didn't Know About)

Pricing: Storage charges + actual throughput used

In the Tokyo region:

Advantages: Pay only for what you use, auto-scaling

Disadvantages: Budget management is difficult if usage is unpredictable

The Better Option

Let me recalculate.

Assuming we used about 50GB of throughput during the investigation:

Difference: Approximately $64~$66 (about ¥9,000~¥9,500)

If I had chosen Elastic Throughput, the cost would have been about 1/20th.

Honestly, I was stuck here. In an emergency, I should have calmly estimated costs, but due to lack of knowledge, Elastic Throughput never came to mind as an option.

Explaining to Stakeholders

This cost requires an explanation to stakeholders.

However, rather than just ending with "I made a mistake," I needed to properly organize the background of the decision and the cost-benefit analysis.

Background of the Decision

During the emergency response, I was thinking about the following points:

Situation:

Option Consideration:

Reasoning:

Cost-Benefit Analysis

Assuming a development team of 10 people at ¥5,000/hour:

Losses Avoided:

Costs Incurred:

Cost-Benefit: Approximately 15x ROI

Looking at the numbers, the decision itself appears to have been appropriate. However, it's also true that if I had known about Elastic Throughput, I could have achieved the same effect at a much lower cost.

Creating the Report

I organized the above content and created a report for stakeholders.

The report included:

I wrote honestly that "the decision was appropriate, but it was not the optimal solution due to lack of knowledge."

Lessons and Reflection

Technical Lessons

Reflection: Why Didn't I Know About Elastic Throughput?

Being honest in my reflection, I only understood the EFS pricing structure "vaguely."

I had this rough knowledge, but I hadn't investigated "how expensive" or "are there other options?"

Elastic Throughput was announced in 2022. In other words, my knowledge wasn't outdated — I simply hadn't researched it properly.

Especially in emergencies, take a moment to research. I regret that I lacked this habit.

Organizational Improvements

To ensure this experience becomes organizational knowledge, not just personal learning:

These are in progress.

Conclusion

Even if you can technically resolve a problem, that's not the end.

I realized that these are all part of an engineer's "work."

The $69 this time was expensive tuition, but I believe I gained the following value:

If I encounter the same situation again, I will choose Elastic Throughput without hesitation.

And if this experience helps someone else, then the $69 was not wasted at all.

Related Articles

The previous troubleshooting article:

JenkinsがじわじわGitクローンできなくなっていった話 ― EFSメタデータIOPSとの闘い

ある月曜の朝、Jenkinsが突然重くなり、Gitクローンが失敗するようになりました。504エラーも頻発。調査の結果、EFSのメタデータIOPSが枯渇していたことが判明。さらに、Git一時ファイルが蓄積され続けていたことが真犯人でした。約3...

Tech

SRE

References