What Looked Like a Simple t4g.medium → t4g.large Swap — and How It Turned Into Confronting Jenkins' Structural Weaknesses

Background: CPU credits "effectively not working"

One day, while staring at an AWS cost report, something felt off.

Our Jenkins Controller EC2 instance (t4g.medium) was supposedly getting a $24/month discount from a Savings Plan, yet we were paying $19/month in CPU credit charges. The net savings came out to just $5.

"This Savings Plan is barely doing anything," I thought.

Looking at CloudWatch, average CPU utilization was 53%, and the CPU credit balance was constantly drained. T-family instances work on a model where "you consume credits when you exceed the baseline, and once they run out, you're billed in Unlimited mode." With a t4g.medium baseline of 40% and steady 53% usage, credit charges keep accruing indefinitely.

At first, I thought "I just need to move up one instance size." Technically, that's true.

But I had no idea that this supposedly simple change would end up forcing me to face the structural weaknesses in our Jenkins operations.

The problem: what we needed to solve

Putting it in order, the problems we were solving were:

The goal was to "drive credit charges to zero," but I also added a follow-on goal: "be able to make the next sizing decision quantitatively." Next time the same problem hits, I didn't want to fall back on gut feel.

Setup: the environment

Initial decision: t4g.large + JavaMelody

I decided to push two changes at the same time.

1. Scale up t4g.medium → t4g.large

Comparing per-vCPU baselines within the t4g family:

Since average 53% was constantly above the medium baseline of 40%, moving up to t4g.large would put baseline (60%) above usage (53%), zeroing out credit charges in theory. And since we stayed in the same t4g family, our existing Savings Plan continued to apply as-is.

Cost-wise, the delta was about $5/month. RAM doubles from 4 GiB to 8 GiB, giving the JVM heap more breathing room as well. "One size up within the same family" turned out to be an overwhelmingly favorable risk-vs-reward trade.

I also considered moving to non-burstable types like m7g.large or c7g.large, but that would waste our current t4g-only Savings Plan. The decision was to defer until SP expiration, then re-evaluate.

2. Introduce JavaMelody (the monitoring plugin)

But part of me was thinking: "Sure, t4g.large fixes the numbers, but is that really the right answer?"

Just upsizing without knowing what the Controller was actually doing with CPU is symptomatic treatment. It's "the battery's empty, so let's get a bigger battery" — without ever asking why consumption is so heavy.

So I decided to install JavaMelody first, get visibility into per-thread CPU, GC time, and HTTP request distribution, then proceed with the upgrade.

JavaMelody adds an overhead of about +1–3% CPU and a few tens to ~100 MB of memory. With CPU already pegged at 53% and credits constantly drained, I was a little worried that the +1–3% would slightly increase credit charges — but the value of root-cause visibility was overwhelmingly larger.

3. Don't rush to buy more Savings Plan capacity

This was the part I agonized over the most. I was worried that "moving to t4g.large would exceed our SP capacity," but a closer look showed our existing SP had spare capacity that could likely absorb the extra cost.

Savings Plans are 1- or 3-year commitment products. Buying in a panic backfires. I decided to watch for at least one month, ideally two, before deciding. Increasing commitment without data is gambling.

Implementation: a cascade of four issues

This should have been "tweak Pulumi config, run Ansible." In reality, four problems chained together.

Walking through them in order.

Issue 1: Pulumi passphrase mismatch — couldn't even start the deploy

The instant I ran the Phase 1 SSM init, this error appeared:

error: getting stack configuration: get stack secrets manager: incorrect passphrase

Pulumi stores its state file encrypted, so deploys need the passphrase to decrypt it. This project resolved the passphrase in the following priority:

Digging in, the cause was that a Pulumi.dev.yaml generated for testing had been committed. The file only contained encryptionsalt, and that was clashing with the existing stack's encryption salt.

# pulumi/jenkins-ssm-init/Pulumi.dev.yaml (the file that had crept in)
encryptionsalt: v1:xxxxxxxxx:...

This salt had been auto-generated locally by a pulumi command — a different one — so even with the correct passphrase from SSM, decryption failed.

Removing the file (git rm) let Pulumi regenerate a new salt and encrypt with the SSM passphrase, fixing the issue.

Takeaway: when an AI workflow agent runs operations like pulumi stack init in the background, byproducts like this can get committed. Adding a check for stray Pulumi.<stack>.yaml files at issue-creation or PR time felt worth establishing as a workflow rule.

Issue 2: chown -R on EFS hangs for 17+ minutes

When I ran the Phase 3 EFS mount and Jenkins install, Ansible timed out at 10 minutes.

Investigating, the hang was inside controller-mount-efs.sh at chown -R jenkins:jenkins /mnt/efs/jenkins. Process state was STAT: D (waiting on NFS I/O). EFS held 4.7 GB of data with a high file count, so a recursive chown over NFS was going more than 15 minutes without finishing.

The relevant code was just three lines:

chown -R jenkins:jenkins $JENKINS_HOME_DIR     # ← hangs here
find $JENKINS_HOME_DIR -type d -exec chmod 755 {} \;
find $JENKINS_HOME_DIR -type f -exec chmod 644 {} \;

The idempotency check only verified "is EFS mounted?" — ownership and permissions were re-applied recursively every time. Combined with NFS round-trip latency, this was a deploy-blocking time bomb.

I added idempotency: "the first run is unavoidable, but on second-and-later runs, if ownership is already correct, skip the recursion."

# Always set ownership on required top-level directories deterministically.
# Handles new subdirectories that mkdir created (root-owned).
# One-deep, so it's fast even on EFS.
chown jenkins:jenkins "$JENKINS_HOME_DIR" 2>/dev/null || true
chown jenkins:jenkins "$JENKINS_HOME_DIR"/{plugins,jobs,secrets,nodes,logs,init.groovy.d} 2>/dev/null || true

# Make the recursive pass idempotent: skip when the top is already correct.
CURRENT_OWNER=$(stat -c '%U:%G' "$JENKINS_HOME_DIR" 2>/dev/null || echo "")
CURRENT_MODE=$(stat -c '%a' "$JENKINS_HOME_DIR" 2>/dev/null || echo "")

if [ "$CURRENT_OWNER" = "jenkins:jenkins" ] && [ "$CURRENT_MODE" = "755" ] && [ "${FORCE_CHOWN:-false}" != "true" ]; then
    log "Ownership and permissions are already correct (owner=$CURRENT_OWNER, mode=$CURRENT_MODE)"
    log "Skipping recursive pass to preserve EFS performance"
else
    log "Setting ownership... this can take a while on EFS"
    chown -R jenkins:jenkins "$JENKINS_HOME_DIR"
    find "$JENKINS_HOME_DIR" -type d -exec chmod 755 {} \;
    find "$JENKINS_HOME_DIR" -type f -exec chmod 644 {} \;
fi

I deliberately left a FORCE_CHOWN=true escape hatch for recovery scenarios when "I just want everything fixed" is the answer.

The reasoning behind the check: this script is the only place that sets ownership, so if the top directory is jenkins:jenkins, we can assume everything underneath is too. Not perfect, but edge cases can fall through to FORCE_CHOWN. That was the trade-off.

After this patch, EFS mount processing went from 17 minutes to about 1 minute — a 17x improvement. That one felt good.

Issue 3: JCasC init failed on Jenkins 2.555.1

EFS resolved, Jenkins should be coming up... but /login kept returning HTTP 500.

The logs showed:

io.jenkins.plugins.casc.UnknownAttributesException:
  Invalid configuration elements for type: DefaultCrumbIssuer : excludeClientIPFromCrumb

The JCasC (Configuration as Code) template had an excludeClientIPFromCrumb attribute, but the deployed Jenkins 2.555.1 had removed it. So JCasC initialization failed → BootFailure → every request got a 500.

Looking it up, excludeClientIPFromCrumb was removed around Jenkins 2.452, and modern Jenkins now defaults to "exclude the client IP from the crumb." So removing the attribute kept the same behavior we had with true. That was the saving grace.

# Before
crumbIssuer:
  standard:
    excludeClientIPFromCrumb: true   # ← Jenkins 2.555.1 doesn't recognize this

# After (first attempt)
crumbIssuer:
  standard:

But then the issue chained.

Issue 4: Just leaving standard: produced a different JCasC error

Thinking "all I need to do is delete the line for excludeClientIPFromCrumb," I merged the PR and re-deployed — only to hit yet another error:

Single entry map expected

When standard: has no value after it, YAML parses it as {'standard': None}. JCasC doesn't interpret that as "an empty-config standard CrumbIssuer" — it fails to recognize it and errors out at init.

The correct form is explicitly using an empty map {}.

# Final form
crumbIssuer:
  standard: {}   # The empty map makes JCasC correctly recognize the standard CrumbIssuer

The YAML parsing differences:

Honestly, this was a trap. I thought I was "just deleting one attribute," and missed that the type at the YAML semantic level had changed. For systems that template JCasC YAML, "empty-content keys" are something to watch out for.

Stepping back: facing the deeper question

Once the cascade of issues settled, I paused to reflect.

"Doesn't this mean every time Jenkins gets upgraded, something will go out of sync, and we'll be patching it after the fact?"

That question was hitting the real issue. What happened wasn't bad luck — it was a structural weakness in our operational design surfacing.

What was broken

The point: there's no artifact representing "a verified, working combination." Jenkins core, plugins, and JCasC are all downloaded/installed/expanded at runtime — so every deploy is the first time a new combination gets exercised, and that "first time" happens in production.

Pinning to LTS reduces "combination variance," but it doesn't change the fundamental structure of "first verification happens in prod."

Detour: Would Fargate be cheaper?

The thought "what about moving to ECS Fargate?" did cross my mind. But running the numbers, it didn't help.

Fargate's pricing includes "management overhead," so for a 24/7 always-on workload like a Controller, EC2 is unconditionally cheaper. Fargate's economics only work out for workloads that can autoscale or use Spot.

That said, Fargate also eliminates some operational costs (AMI build maintenance, OS patching, no CPU credit concept at all, etc.). Translated to monthly labor, that's maybe 0.5–1 hour/month — a few thousand to ten-thousand yen of savings — but probably not enough to overcome the raw compute cost difference.

Conclusion: "if cost is the goal, moving to Fargate doesn't help." Fargate is worth it when you're willing to pay for "complete liberation from OS management." That's the framing I landed on.

Next steps: addressing the structural weakness "within realistic bounds"

The ideal answer would be to build Jenkins core + all plugins + JCasC into a single Docker image, and deploy that — an "immutable Jenkins." That makes "the working combination" exist as a real Docker-image artifact, shifts verification left to build time, and means production only ever sees "verified" artifacts.

But realistically, rebuilding the current EC2 + EFS structure end-to-end is major surgery. There's a lot of surrounding capital — AMI Builder, EC2 Fleet, ECS Fargate agents, etc.

So I prioritized realistic landing points:

The judgment call: rather than "do everything on paper," "only address things once you actually feel the pain in operations" — that avoids over-investment.

The cost of LTS-pinning is "security patches lag," but you can absorb that by frequently picking up LTS minor updates (no breaking changes), planning LTS-line (major) updates quarterly, and handling CVEs through a separate exception flow. The Jenkins community at large operates this way, so there's a well-established playbook.

Result

Eventually, every phase completed and Jenkins came up healthy.

And the issues/PRs that came out:

CPU effect verification is scheduled for two weeks out (via CloudWatch + Cost Explorer). If average CPU lands at 30–45% and credit charges disappear, Phase 3 is done. From there, we move to the Savings Plan increase decision.

Wrap-up: what I learned

I started this thinking it was "a simple t4g.medium → t4g.large swap." By the time it was over, I had run into structural weaknesses in our Jenkins operations and the absence of an upgrade strategy.

Three things stuck with me.

First, we had been letting "simple changes turn out to not be simple, and only in production" sit unaddressed. The moment we were pulling the latest tag, deploys were a version-roulette. For a team our size, that's a clearly excessive risk — and it's the kind of structure you don't see until you've taken a hit once.

Second, "idempotency reveals its time cost in production." EFS chown -R finishes in seconds during dev with test data and the problem stays invisible — but when 4.7 GB has accumulated in production, it hangs for 17 minutes. "Works" and "works at production scale" are different things, and the gap is especially wide once external storage like NFS is in the picture.

Third, before jumping to Fargate or a giant refactor, first identify the actual weak point in your current architecture. Containerizing Jenkins (Dockerizing it) is the right answer in principle — but doing all of it now? No. LTS-pinning + a Runbook stops the "unexpected stumbles" first. You operate with that, then look at where pain still remains and address that next. The right answer on paper and the right answer that's implementable in reality are different things — that's a lesson I keep relearning, and keep forgetting.

What was supposed to be "cost optimization" turned into a structural inventory of our operations. It took longer than I'd planned, but for work where there's a "terrain you only see by walking it," I'd guess that estimating it up-front is largely meaningless — and pressing forward without retreating is, in the end, faster.

Next, I'm picking up Issue #576 (LTS pin + upgrade Runbook). Now is the only time we're going to be able to write that Runbook with the locations of the mines we just stepped on still fresh in mind.

Related Books

Here are recommended books for those who want to learn more about Jenkins operations, AWS infrastructure operations, and SRE-style structural design like in this case.