Jenkins & EFS Follow-up: The Problem Surfaced on 1/26, But the Root Cause Was on 1/13
In my previous articles, "How Jenkins Gradually Lost the Ability to Clone Git Repositories" and "The Aftermath: Spending $10,000 on Provisioned Throughput", I documented our EFS metadata IOPS exhaustion issue and the response.
On Monday morning, Jenkins became sluggish. After investigation, we discovered approximately 15GB of accumulated Git temporary files (tmp_pack_*). As an emergency measure, we switched to provisioned throughput at 300 MiB/s, later migrated to Elastic Throughput, and created a periodic cleanup job.
"That should solve it," I thought.
But actually, it wasn't over yet.
What the Graph Reveals About the True Beginning
While reviewing EFS Burst Credit Balance graphs after the incident, I noticed a clear downward trend starting around 1/13.
The problem became visible on 1/26-1/27, but the root cause occurred two weeks earlier, around 1/13.
This means the tmp_pack_* accumulation described in the previous article was a symptom of deterioration, not the true root cause.
Honestly, I was stuck here. I had some ideas about what changed around 1/13, but nothing definitive.
Identifying Changes Around 1/13
I could think of two potential changes:
1. Agent Architecture Change
Around 1/13, we made a significant change to how Jenkins agents operated.
Previous approach: Shared agents
- EC2 instances: c5.large, etc.
- Shared across multiple jobs
- Workspace reuse
git pullto fetch only deltas
New approach: Ephemeral agents
- EC2 instances: t3.small, etc.
- One agent per job
- Destroyed after use
- Full
git cloneevery time
This was a cost optimization change, but we hadn't considered the impact on metadata IOPS.
2. Post-Holiday Development Acceleration
After the New Year holidays, development activity across teams accelerated, potentially increasing overall Jenkins load.
Hypothesis: Ephemeral Agents Drastically Increased Metadata IOPS
My hypothesis was that the shift to ephemeral agents dramatically increased metadata IOPS.
Let's do a simple calculation:
Build count: 50/day (estimated)
Files created per clone: 5,000
Shared agent approach:
Clone only on first use = 5,000 metadata operations
Ephemeral agent approach:
50 builds × 5,000 files = 250,000 metadata operations/dayA 50x increase in metadata IOPS.
If development accelerated post-holidays, this number would be even higher. Our cost optimization was potentially creating a different problem.
The Cache Directory Mystery
During the investigation, I noticed the /mnt/efs/jenkins/caches/ directory.
/mnt/efs/jenkins/caches/git-3e9b32912840757a720f39230c221f0eThere were many such directories. What are these?
These are bare repositories where Jenkins' Git plugin caches repositories.
How Git Caching Works
Jenkins' Git plugin has an optimization mechanism:
- Cache remote repositories as bare repositories in
/mnt/efs/jenkins/caches/git-{hash}/ - Each job's workspace clones from this cache using
git clone --reference - Hash values are generated from repository URL + branch, etc.
A new question arose: Did switching to ephemeral agents disable this cache?
- Shared agents: Cache is effectively utilized
- Ephemeral agents: Each new agent can't benefit from the cache, resulting in full clones every time
Revisiting the tmp_pack_* Location
Let me review where we found the tmp_pack_* files from the previous article:
jobs/sample-job/jobs/sample-pipeline/builds/104/libs/
└── 335abf.../root/.git/objects/pack/
└── tmp_pack_WqmOyE ← 100-300MBThese exist in per-build directories. Specifically:
jobs/sample-job/jobs/sample-pipeline/
└── builds/
├── 104/
│ └── libs/.../tmp_pack_WqmOyE
├── 105/
│ └── libs/.../tmp_pack_XYZ123
└── 106/
└── libs/.../tmp_pack_ABC456Every build checks out the Pipeline Shared Library again, leaving tmp_pack_* files behind.
This raises a new question: Why is the Shared Library being checked out for every build?
Normally, Shared Libraries should also be cached.
Root Cause Discovery: Cache Setting Was OFF
When checking Jenkins configuration, I found the critical misconfiguration.
In the Shared Library settings, Cache fetched versions on controller for quick retrieval was unchecked.
This meant:
- Shared Library cache was completely disabled
- Full fetch from remote repository every build
- Temporary files generated in
.git/objects/pack/ - Metadata IOPS explosion
The dots connected:
- Shift to ephemeral agents (1/13) → Full clone every time on agent side
- Disabled Shared Library cache (pre-existing) → Full fetch every time on controller side
- Post-holiday development acceleration → Increased build count
These three factors combined to cause a massive increase in metadata IOPS. Over two weeks, the Burst Credit depleted, and symptoms became apparent on 1/26.
Solution: Enable the Cache
I immediately changed the settings:
- Turned ON
Cache fetched versions on controller for quick retrieval - Set
Refresh time in minutesto 180 minutes
How We Decided the Refresh Time
This is more critical than it seems. The options were:
- 60-120 minutes: For active development periods. Changes reflected quickly but moderate IOPS reduction
- 180 minutes (3 hours): Balanced approach. ~8 update checks per day
- 360 minutes (6 hours): For stable operations. 4 update checks per day
- 1440 minutes (24 hours): Maximum IOPS savings. For low change frequency
We chose 180 minutes (3 hours). The reasons:
- Approximately 8 update checks per day (9:00, 12:00, 15:00, 18:00...)
- Shared Library changes typically don't need to be reflected within half a day
- Significant IOPS reduction (every build → every 3 hours)
- Manual cache clear available for urgent changes
Actually, Jenkins has a "force refresh" feature, so urgent changes aren't a problem. However, since I might forget this feature exists, I documented it in our operations guide.
Measuring the Effect After Configuration Change
After the configuration change, we plan to monitor effectiveness on this timeline:
Short term (24-48 hours)
- Are new
tmp_pack_*files no longer being generated? - Has EFS metadata IOPS decreased?
Medium term (1 week)
- Is Burst Credit Balance trending upward?
- Is build performance stable?
Long term (1 month)
- Is credit being maintained stably?
- Has the problem recurred?
Reflections and Lessons
The Root Cause Was Compound
This issue wasn't caused by a single factor, but by multiple factors converging:
- Disabled Shared Library cache (pre-existing)
- Shift to ephemeral agents (around 1/13)
- Post-holiday development acceleration (increased build count)
Each factor alone might not have caused a major problem, but together they pushed metadata IOPS past the critical threshold.
Symptom Timing Differs from Root Cause
- Symptoms appeared: 1/26-1/27
- Root cause: Around 1/13
- Credit depletion: Progressed over 2 weeks
I deeply felt the importance of looking at graphs chronologically. Addressing only the immediate symptoms leads to superficial fixes.
Architecture Change Risks
The shift to ephemeral agents was intended for cost optimization. While EC2 costs did decrease, it created cost and performance problems elsewhere.
When changing architecture:
- Evaluate performance impact in advance
- Set up monitoring metrics before making changes
- Continuously track metrics after changes
I felt the importance of these practices anew.
Characteristics of Metadata IOPS
I learned important characteristics about EFS metadata IOPS:
- Mass creation/deletion of small files is critical
- File count matters more than storage capacity
- Credit management is key in burst mode
- Credit depletion progresses gradually
Especially with cases like .git/objects/ containing "many small files," behavior differs completely from normal file I/O.
The Next Question
With this response, we enabled Shared Library caching. However, the ephemeral agent approach continues.
Can agent-side Git caching be effectively utilized with ephemeral agents?
This requires more investigation. Possibilities include:
- Share Git cache on EFS across all agents
- Extend agent lifecycle slightly to reuse across multiple jobs
- Place cache in S3 and sync on startup
How to balance cost and performance remains the next challenge.
Related Articles
This article is a follow-up to:
JenkinsがじわじわGitクローンできなくなっていった話 ― EFSメタデータIOPSとの闘い
トラブル解決後の仕事 ― プロビジョンドスループットで1万円使った反省
Recommended Reading
For those interested in learning more about infrastructure operations and troubleshooting:
Was this article helpful?
If this article helped you organize your thoughts
a coffee-sized support would be much appreciated.
※ This is separate from tipping, but—
If you'd like to organize similar themes in your own context, I also offer dialogue sessions as a form of thought organization.
Recommended for you