I restarted Jenkins, and it didn't come back up.

This was expected. To be precise, I had already decided: "I know it will break eventually, and I'll fix it when it does." Recovery went fine in the end, but I want to document the reasoning behind that decision.

These kinds of decisions rarely get written about. "It broke, we fixed it" gets documented, but "why did we leave it broken in the first place" usually doesn't. I'm writing this partly to organize my own thinking.

The Situation

Our Jenkins is managed with JCasC (Jenkins Configuration as Code). A jenkins.yaml file in JENKINS_HOME is loaded at startup to restore configuration. We use EC2 Spot Fleet as agents, and the Fleet request ID is written into that YAML file.

jenkins.yaml doesn't auto-update when you change settings through the GUI. On restart, it always restores the values from jenkins.yaml as the source of truth. Spot Fleet is an AWS resource — when it gets recreated, the ID changes. This means that unless jenkins.yaml is updated every time the Fleet is recreated, Jenkins will break on restart.

I was fully aware of this structure.

There was a pipeline job that updates jenkins.yaml, and running it would have solved the problem. But that job was temporarily broken due to a separate bug.

The details of that bug are documented in this issue:

🔗 https://github.com/tielec/infrastructure-as-code/issues/560

In short, a design inconsistency in the amazon-ecs plugin caused JCasC's check-configuration (dry-run validation) to fail. The actual startup and UI-based configuration application worked fine — only the CLI dry-run was rejected. While I was deferring the fix for this plugin bug, the jenkins.yaml update job remained stuck.

Why I Left It Alone

To be honest: because I decided "I'll fix it when it breaks."

The basis for that confidence was accumulated experience troubleshooting Jenkins. Plugin initialization failures, JCasC configuration errors, shared storage mount issues — having worked through all of these, I had built up a sense of where to look in logs and how to narrow down causes.

My read was: "If it's a Spot Fleet ID problem, I can recover quickly."

This isn't really a right-or-wrong thing — it's about reading the situation and making a call. But I need to be honest: "I can fix it" is the flip side of "only I can fix it." That's something to stay aware of.

How I Actually Fixed It

After the restart, Jenkins wouldn't start. Logs showed the ec2-fleet plugin failing to initialize during JCasC application. The YAML syntax was correct and all values were populated. That pointed to the referenced AWS resource being the issue.

# Check Spot Fleet request ID status
aws ec2 describe-spot-fleet-requests \
  --spot-fleet-request-ids sfr-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \
  --query "SpotFleetRequestConfigs[].{ID:SpotFleetRequestId,State:SpotFleetRequestState}" \
  --output table

Expired. I replaced the ID in jenkins.yaml with the currently active one and restarted. Recovery complete.

Reflecting on the Decision

"Running something while knowing it will break" is a choice that invites debate. "That's negligence" or "known issues should be fixed immediately" — these positions are valid in principle. I agree that's the ideal.

But in practice, there are situations where it's difficult to fix every issue before it causes a problem. This time, I chose "fix when broken." I think it's less about right or wrong and more about how you read the situation at the time.

Looking back, this decision only held up because I had enough Jenkins experience to recover quickly. With less experience, the same structural problem could have taken much longer to resolve — or the root cause might not have been found at all. "Fix when broken" only works when you have reasonable grounds to believe you can fix it. That's something to keep in mind.

And even if I can fix it, if that knowledge only lives in my head, there's a limit to how long that's sustainable. That's part of why I'm writing this down.

Recommended Resources

The following books are helpful for operational decision-making and Jenkins troubleshooting.

[📩 敆擁ăƒȘンク: moshimo-book-sre-google]

[📩 敆擁ăƒȘンク: moshimo-book-jenkins-jissen]