I Noticed 40 Minutes Later

I executed a CloudFormation stack update and moved on to other work.

40 minutes later, I casually opened the AWS console and saw EC2 instances running—10 at a time, 50 in total.

"What?"

At first, I couldn't understand what was happening. Why are there so many instances running?

The next moment, I understood.

"Oh no, I did it."

Rather than panic, a calm recognition of "this is bad" came first. I've been through enough experiences that this doesn't make me panic. But still, it's bad. I need to stop this and fix it quickly.

Then, a slowly spreading regret.

"Why didn't I check?"

What worried me was, of course, the cost. Even though they're spot instances, 50 of them for 40 minutes. How much is this going to cost?

What Happened

The task was to add multiple instance size patterns (t-nano, t-small, t-medium, t-large, c-xlarge) to a SpotFleet used as Jenkins Agents.

I was modifying CloudFormation templates while having a conversation with Claude Code. The diff was large—LaunchTemplates increased, SpotFleet resources split into 5, and parameters were added.

In the CloudFormation template, I defined 5 SpotFleet resources, each with a configuration like this:

TargetCapacity: !Ref MaxTargetCapacity

The MaxTargetCapacity parameter was set to 10. In other words, each SpotFleet was configured to launch 10 instances.

Originally, in this architecture, the Jenkins EC2 Fleet Plugin was designed to dynamically manage SpotFleet's TargetCapacity.

In other words, the CloudFormation template shouldn't have set an initial TargetCapacity. The correct setting was:

TargetCapacity: 0

I knew about TargetCapacity. But I overlooked it.

After executing the stack update, I moved on to other work. CloudFormation takes time. Doing other work during that time is natural.

40 minutes later, I checked.

50 instances were running.

The Accumulation of Small Mistakes

Looking back at this incident, I realized something.

This wasn't a single mistake.

It was the result of slipping through multiple checkpoints.

The Checkpoints That Were Bypassed

1. Design Phase: Default to the "Safe Side"

When designing the template, I hadn't set a default value of TargetCapacity: 0. The design philosophy of "defaulting to the safe side" was missing.

If I had set the default to 0 here, nothing would have launched in the first place.

2. Review Phase: Verify Important Parameters in AI-Generated Code

The diff was too large, and I didn't have the mental energy to review it carefully. I thought "it should be fine" and let it pass.

If I had noticed the TargetCapacity setting here, this could have been prevented.

3. Post-Execution Phase: Set Up Stack Update Completion Notifications

I hadn't set up notifications for stack update completion. With EventBridge and SNS, you can automatically notify when a stack update completes.

If I had notifications, I wouldn't have forgotten to check.

4. Verification Phase: Make Immediate Confirmation a Habit After Notifications

After executing the stack update, I should have checked immediately, but CloudFormation takes time, so I moved on to other work during that period.

If I had checked here, I would have noticed much sooner.

Any One of These Would Have Prevented It

Among these checkpoints, if even one had worked, the 50-instance launch could have been prevented.

But all of them were bypassed.

The accumulation of small mistakes became a big problem.

This is always how these incidents happen. They don't occur from a single mistake. They only happen when multiple checkpoints are bypassed.

Emergency Response: Discovering the "Amazing" Side of AI-Driven Development

The moment I confirmed 50 instances were running, several options came to mind.

  1. Modify the CloudFormation template and execute another stack update
  2. Use AWS CLI to directly set SpotFleet's TargetCapacity to 0
  3. Manually terminate instances from the console

"Stop it first."

This was the top priority.

Here, I asked Claude Code for help.

"50 instances have launched. First, I want to set SpotFleet's TargetCapacity to 0 and terminate the running instances."

Claude Code carefully organized the execution conditions before responding.

First, it checked all running SpotFleet requests. Then it accurately generated commands to set each SpotFleet's TargetCapacity to 0:

aws ec2 modify-spot-fleet-request \
  --spot-fleet-request-id sfr-xxx \
  --target-capacity 0

It executed for all 5 SpotFleets with the exact IDs specified.

Next, it retrieved all running instance IDs and generated a command to terminate them all at once:

aws ec2 terminate-instances \
  --instance-ids i-xxx i-yyy i-zzz ...

This process was surprisingly fast.

If I had tried to do it myself, I would have needed to check SpotFleet IDs, write commands, execute them, check instance IDs, write again, execute again... Mistakes might have occurred along the way.

But Claude Code wrote the commands accurately and organized the execution conditions before executing. Faster than a human.

The recovery speed was incredibly fast.

They stopped.

The Dual Nature of AI-Driven Development

Through this experience, I learned both the "scary" and "amazing" sides of AI-driven development at the same time.

Scary:

Amazing:

AI-driven development causes failures and helps with recovery.

Fixing the Template

After the emergency response, the next step was fixing the CloudFormation template.

TargetCapacity: 0  # Let Jenkins EC2 Fleet Plugin manage this

I also added a comment. Without explicitly stating "why it's 0," I might make the same mistake again.

I committed the fix and executed the stack update again. This time, I watched the screen and confirmed. Until I confirmed that no instances launched.

They didn't launch.

I was relieved.

What About the Cost?

After calming down, I calculated the cost I was worried about.

Operating Status:

Breakdown (us-west-2, Spot pricing):

Estimated Cost: About $1.41 USD

Thanks to Spot pricing, it was much cheaper than I expected.

The moment I thought "it only cost $1.41," the tension left my shoulders. But at the same time, a question arose: "Is this okay?"

Because it only cost $1.41, maybe the lesson will fade.

What if I had noticed even later? What if they were on-demand instances? What if it was a production environment?

I don't want to think about it, but I have to.

Moving Forward: Adding More Checkpoints

This incident was the result of slipping through multiple checkpoints.

So what should I do next?

Add more checkpoints.

Specific Checkpoints

1. Design Phase: Default to the "Safe Side"

Parameters like TargetCapacity, MinSize, MaxSize should default to 0 or minimum values. "Doing nothing" should be the safe state.

2. Review Phase: Always Verify Important Parameters

The larger the diff, the more carefully you should review. Especially verify parameters related to capacity and size.

3. Post-Execution Phase: Set Up Stack Update Completion Notifications

Use EventBridge and SNS to automatically notify when stack updates complete. Slack, email, or phone notifications work too.

4. Verification Phase: Confirm Immediately When Notifications Arrive

CloudFormation takes time, so it's understandable to move on to other work. But when notifications come, check immediately. Make it a habit.

If One Fails, Others Can Cover

The point of adding checkpoints is to ensure that if one fails, others can provide coverage.

Lessons from this incident:

And one more thing.

Knowing something and practicing it are different.

I knew about TargetCapacity. But I overlooked it. There's always a gap between knowledge and practice.

What fills that gap is checkpoints.

The idea of "adding more checkpoints" discussed in this article aligns with the fundamental philosophy of SRE (Site Reliability Engineering). This book, based on Google's experience, systematically explains system design that assumes failures will occur and methods to cover human errors with mechanisms.

[📦 商品リンク: moshimo-book-sre-google]

When working with cloud infrastructure, foundational knowledge of servers and networks is essential. To master SpotFleet and CloudFormation, I recommend building a solid foundation with this book.

[📦 商品リンク: moshimo-book-server-infra]

This book provides a comprehensive guide to AWS operations, from basics to practical know-how. Understanding proper EC2 and SpotFleet configuration helps prevent incidents like this one.

[📦 商品リンク: moshimo-book-aws-operations]

I Should Do Better Next Time (But Will I Really?)

I'm relieved it only cost $1.41. But is that okay?

No, I want to use this experience and connect it to what's next.

I'll add more checkpoints.

If one fails, others can cover.

I should do better next time.

But will I really?

Won't I repeat the same thing once I've forgotten about it?

Even if I decide to add checkpoints, actually setting them up is tedious. I might postpone the EventBridge setup and then forget about it. Even when notifications come, I might think "I'll check later" and it becomes another 40 minutes.

Humans are creatures that forget. And creatures that let their guard down thinking "it should be fine."

Whether I can apply the lessons from this incident ultimately depends on my future self.

How much can I trust my future self?

I still don't have an answer to that question.