Introduction

Recently, I published an article titled "The Tale of Jenkins Gradually Failing to Clone Git — Fighting EFS Metadata IOPS." It described how EFS metadata IOPS exhaustion caused Jenkins to stop, and how I identified and resolved the issue in about 3 hours of investigation.

Technically, the problem was solved, but that wasn't the end of the story.

The next day, when I opened AWS Cost Explorer, I froze at the number that jumped out at me.

"$69.00"

Honestly, I thought... maybe I messed up.

This article is about cost judgment during emergency response and the accountability that follows. It's the "aftermath" of the troubleshooting article.

What Happened

As I wrote in the previous article, EFS metadata IOPS exhaustion caused Jenkins to effectively stop. And this happened on a Monday morning — the worst possible timing.

To keep Jenkins running while investigating, I made an emergency decision to change the EFS throughput mode to Provisioned Throughput at 300 MiB/s.

This decision resulted in:

Technically, it was successful. However...

The Cost Reality

The next day, after reverting to Elastic Throughput, I became curious about the cost.

I knew that Provisioned Throughput was "expensive." But I thought, for just 26 hours, it wouldn't be that bad.

When I opened Cost Explorer:

Approximately $69

In 26 hours.

If I had left it running for a month, it would have been about $2,160 (over ¥300,000). I was relieved I changed it back quickly, but at the same time, the question arose: "Wasn't there a better option?"

Understanding EFS Pricing

I needed to properly organize the EFS pricing structure. The truth is, during the emergency response, I didn't know about Elastic Throughput as an option.

EFS has three throughput modes:

1. Bursting Throughput (Original Setting)

Pricing: Storage charges only

Throughput is determined by storage capacity. There's a baseline of 50 MiB/s per TiB, with a credit system that accumulates during low load and can be used during high load.

Advantages: No additional charges
Disadvantages: Performance degradation when credits are depleted (the problem we had)

2. Provisioned Throughput (Emergency Response Setting)

Pricing: Storage charges + throughput charges

In the Tokyo region, it's approximately $7.2/month per MiB/s.

For 300 MiB/s:

For 26 hours of use:

The actual measurement was $69, so the baseline throughput was probably deducted.

Advantages: Stable performance, no credits needed
Disadvantages: Very expensive, charged even when not in use

3. Elastic Throughput (The Option I Didn't Know About)

Pricing: Storage charges + actual throughput used

In the Tokyo region:

Advantages: Pay only for what you use, auto-scaling
Disadvantages: Budget management is difficult if usage is unpredictable

The Better Option

Let me recalculate.

Assuming we used about 50GB of throughput during the investigation:

Mode 26-hour Cost Notes
Bursting $5.6 (monthly portion) Investigation impossible due to credit depletion
Provisioned $69 Actually selected
Elastic $3.5~$5 Didn't know about it

Difference: Approximately $64$66 (about ¥9,000¥9,500)

If I had chosen Elastic Throughput, the cost would have been about 1/20th.

Honestly, I was stuck here. In an emergency, I should have calmly estimated costs, but due to lack of knowledge, Elastic Throughput never came to mind as an option.

Explaining to Stakeholders

This cost requires an explanation to stakeholders.

However, rather than just ending with "I made a mistake," I needed to properly organize the background of the decision and the cost-benefit analysis.

Background of the Decision

During the emergency response, I was thinking about the following points:

Situation:

Option Consideration:

Option Advantages Disadvantages Decision
Continue Bursting Mode No cost increase Investigation difficult due to credit depletion ❌ Not viable
Provisioned 300 MiB/s Guaranteed performance
Investigation possible		 High cost ✅ Adopted
Elastic Throughput Usage-based billing, affordable
Auto-scaling		 Not considered due to lack of knowledge -

Reasoning:

  1. Minimize impact on the entire development team due to system outage
  2. Stable I/O performance needed for investigation work (find, du, etc.)
  3. Expected to be able to change settings the next day
  4. Business continuity as top priority

Cost-Benefit Analysis

Assuming a development team of 10 people at ¥5,000/hour:

Losses Avoided:

Costs Incurred:

Cost-Benefit: Approximately 15x ROI

Looking at the numbers, the decision itself appears to have been appropriate. However, it's also true that if I had known about Elastic Throughput, I could have achieved the same effect at a much lower cost.

Creating the Report

I organized the above content and created a report for stakeholders.

The report included:

I wrote honestly that "the decision was appropriate, but it was not the optimal solution due to lack of knowledge."

Lessons and Reflection

Technical Lessons

  1. Effectiveness of Elastic Throughput

    • Optimal for spike responses
    • Predictable costs with usage-based billing
    • Will use as the first choice in the future

  2. Cost Estimation, Especially in Emergencies

    • "Just getting it working" is not enough
    • Develop a habit of comparing options and cost impact
    • Question whether there might be options you don't know about

  3. Strengthening Monitoring Systems

    • Regular monitoring of EFS metrics
    • Monitor not just storage capacity but also file count and IOPS
    • Preventive maintenance through early detection

Reflection: Why Didn't I Know About Elastic Throughput?

Being honest in my reflection, I only understood the EFS pricing structure "vaguely."

I had this rough knowledge, but I hadn't investigated "how expensive" or "are there other options?"

Elastic Throughput was announced in 2022. In other words, my knowledge wasn't outdated — I simply hadn't researched it properly.

Especially in emergencies, take a moment to research. I regret that I lacked this habit.

Organizational Improvements

To ensure this experience becomes organizational knowledge, not just personal learning:

  1. Knowledge Sharing

    • Publish this article internally and externally
    • Consider holding AWS cost optimization study sessions

  2. Documentation

    • Develop troubleshooting procedures
    • Formalize decision-making flows including cost estimation

  3. Monitoring & Alerting

    • Visualize EFS usage
    • Set alerts for throughput utilization exceeding 75%

These are in progress.

Conclusion

Even if you can technically resolve a problem, that's not the end.

I realized that these are all part of an engineer's "work."

The $69 this time was expensive tuition, but I believe I gained the following value:

  1. Deep understanding of EFS pricing structure
  2. Improved emergency decision-making process
  3. Organization-wide knowledge sharing

If I encounter the same situation again, I will choose Elastic Throughput without hesitation.

And if this experience helps someone else, then the $69 was not wasted at all.

Related Articles

The previous troubleshooting article:

/en/tech/sre/jenkins-efs-metadata-iops-issue

References