Jenkins & EFS Follow-up: The Problem Surfaced on 1/26, But the Root Cause Was on 1/13

In my previous articles, "How Jenkins Gradually Lost the Ability to Clone Git Repositories" and "The Aftermath: Spending $10,000 on Provisioned Throughput", I documented our EFS metadata IOPS exhaustion issue and the response.

On Monday morning, Jenkins became sluggish. After investigation, we discovered approximately 15GB of accumulated Git temporary files (tmp_pack_*). As an emergency measure, we switched to provisioned throughput at 300 MiB/s, later migrated to Elastic Throughput, and created a periodic cleanup job.

"That should solve it," I thought.

But actually, it wasn't over yet.

What the Graph Reveals About the True Beginning

While reviewing EFS Burst Credit Balance graphs after the incident, I noticed a clear downward trend starting around 1/13.

EFS Burst Credit Balance trend - decline begins on 1/13, reaches minimum on 1/26

The problem became visible on 1/26-1/27, but the root cause occurred two weeks earlier, around 1/13.

This means the tmp_pack_* accumulation described in the previous article was a symptom of deterioration, not the true root cause.

Honestly, I was stuck here. I had some ideas about what changed around 1/13, but nothing definitive.

Identifying Changes Around 1/13

I could think of two potential changes:

1. Agent Architecture Change

Around 1/13, we made a significant change to how Jenkins agents operated.

Previous approach: Shared agents

EC2 instances: c5.large, etc.
Shared across multiple jobs
Workspace reuse
git pull to fetch only deltas

New approach: Ephemeral agents

EC2 instances: t3.small, etc.
One agent per job
Destroyed after use
Full git clone every time

This was a cost optimization change, but we hadn't considered the impact on metadata IOPS.

2. Post-Holiday Development Acceleration

After the New Year holidays, development activity across teams accelerated, potentially increasing overall Jenkins load.

Hypothesis: Ephemeral Agents Drastically Increased Metadata IOPS

My hypothesis was that the shift to ephemeral agents dramatically increased metadata IOPS.

Let's do a simple calculation:

Build count: 50/day (estimated)
Files created per clone: 5,000

Shared agent approach:
  Clone only on first use = 5,000 metadata operations

Ephemeral agent approach:
  50 builds × 5,000 files = 250,000 metadata operations/day

A 50x increase in metadata IOPS.

If development accelerated post-holidays, this number would be even higher. Our cost optimization was potentially creating a different problem.

The Cache Directory Mystery

During the investigation, I noticed the /mnt/efs/jenkins/caches/ directory.

/mnt/efs/jenkins/caches/git-3e9b32912840757a720f39230c221f0e

There were many such directories. What are these?

These are bare repositories where Jenkins' Git plugin caches repositories.

How Git Caching Works

Jenkins' Git plugin has an optimization mechanism:

Cache remote repositories as bare repositories in /mnt/efs/jenkins/caches/git-{hash}/
Each job's workspace clones from this cache using git clone --reference
Hash values are generated from repository URL + branch, etc.

A new question arose: Did switching to ephemeral agents disable this cache?

Shared agents: Cache is effectively utilized
Ephemeral agents: Each new agent can't benefit from the cache, resulting in full clones every time

Revisiting the tmp_pack_* Location

Let me review where we found the tmp_pack_* files from the previous article:

jobs/sample-job/jobs/sample-pipeline/builds/104/libs/
  └── 335abf.../root/.git/objects/pack/
      └── tmp_pack_WqmOyE  ← 100-300MB

These exist in per-build directories. Specifically:

jobs/sample-job/jobs/sample-pipeline/
└── builds/
    ├── 104/
    │   └── libs/.../tmp_pack_WqmOyE
    ├── 105/
    │   └── libs/.../tmp_pack_XYZ123
    └── 106/
        └── libs/.../tmp_pack_ABC456

Every build checks out the Pipeline Shared Library again, leaving tmp_pack_* files behind.

This raises a new question: Why is the Shared Library being checked out for every build?

Normally, Shared Libraries should also be cached.

Root Cause Discovery: Cache Setting Was OFF

When checking Jenkins configuration, I found the critical misconfiguration.

In the Shared Library settings, Cache fetched versions on controller for quick retrieval was unchecked.

This meant:

Shared Library cache was completely disabled
Full fetch from remote repository every build
Temporary files generated in .git/objects/pack/
Metadata IOPS explosion

The dots connected:

Shift to ephemeral agents (1/13) → Full clone every time on agent side
Disabled Shared Library cache (pre-existing) → Full fetch every time on controller side
Post-holiday development acceleration → Increased build count

These three factors combined to cause a massive increase in metadata IOPS. Over two weeks, the Burst Credit depleted, and symptoms became apparent on 1/26.

Solution: Enable the Cache

I immediately changed the settings:

Turned ON Cache fetched versions on controller for quick retrieval
Set Refresh time in minutes to 180 minutes

How We Decided the Refresh Time

This is more critical than it seems. The options were:

60-120 minutes: For active development periods. Changes reflected quickly but moderate IOPS reduction
180 minutes (3 hours): Balanced approach. ~8 update checks per day
360 minutes (6 hours): For stable operations. 4 update checks per day
1440 minutes (24 hours): Maximum IOPS savings. For low change frequency

We chose 180 minutes (3 hours). The reasons:

Approximately 8 update checks per day (9:00, 12:00, 15:00, 18:00...)
Shared Library changes typically don't need to be reflected within half a day
Significant IOPS reduction (every build → every 3 hours)
Manual cache clear available for urgent changes

Actually, Jenkins has a "force refresh" feature, so urgent changes aren't a problem. However, since I might forget this feature exists, I documented it in our operations guide.

Measuring the Effect After Configuration Change

After the configuration change, we plan to monitor effectiveness on this timeline:

Short term (24-48 hours)

Are new tmp_pack_* files no longer being generated?
Has EFS metadata IOPS decreased?

Medium term (1 week)

Is Burst Credit Balance trending upward?
Is build performance stable?

Long term (1 month)

Is credit being maintained stably?
Has the problem recurred?

Reflections and Lessons

The Root Cause Was Compound

This issue wasn't caused by a single factor, but by multiple factors converging:

Disabled Shared Library cache (pre-existing)
Shift to ephemeral agents (around 1/13)
Post-holiday development acceleration (increased build count)

Each factor alone might not have caused a major problem, but together they pushed metadata IOPS past the critical threshold.

Symptom Timing Differs from Root Cause

Symptoms appeared: 1/26-1/27
Root cause: Around 1/13
Credit depletion: Progressed over 2 weeks

I deeply felt the importance of looking at graphs chronologically. Addressing only the immediate symptoms leads to superficial fixes.

Architecture Change Risks

The shift to ephemeral agents was intended for cost optimization. While EC2 costs did decrease, it created cost and performance problems elsewhere.

When changing architecture:

Evaluate performance impact in advance
Set up monitoring metrics before making changes
Continuously track metrics after changes

I felt the importance of these practices anew.

Characteristics of Metadata IOPS

I learned important characteristics about EFS metadata IOPS:

Mass creation/deletion of small files is critical
File count matters more than storage capacity
Credit management is key in burst mode
Credit depletion progresses gradually

Especially with cases like .git/objects/ containing "many small files," behavior differs completely from normal file I/O.

The Next Question

With this response, we enabled Shared Library caching. However, the ephemeral agent approach continues.

Can agent-side Git caching be effectively utilized with ephemeral agents?

This requires more investigation. Possibilities include:

Share Git cache on EFS across all agents
Extend agent lifecycle slightly to reuse across multiple jobs
Place cache in S3 and sync on startup

How to balance cost and performance remains the next challenge.

This article is a follow-up to:

/en/tech/sre/jenkins-efs-metadata-iops-issue

/en/tech/sre/efs-provisioned-throughput-cost-reflection