Production Deploy Day: Creating the Canary SG, Updating ECS SGs, and Running Pulumi

In the previous article, I wrote about the design for incrementally migrating Canary security group (SG) management from Pulumi to Terraform.

This is the follow-up: a record of the day I actually deployed it to production.

Unlike the design phase, the actual deploy day came with a few unexpected small snags. The .yml vs .yaml extension story in particular felt worth writing down.

Premise: the overall flow

The work breaks down into a few phases.

Phase 1: Manually create the Canary SG (no ingress, egress 80/443 only)
Phase 2: Add an ingress rule from the Canary SG to several ECS SGs
Phase 3: Replace the canarySecurityGroupId placeholder in Pulumi.prod.yml with the real ID, commit & push
Phase 4: Merge the PR into dev
Phase 5: Run preview → deploy on Jenkins
Phase 6: Verify Canary behavior (RUNNING/PASSED, approve SNS subscriptions)
Phase 7: Flip snsEnabled to true and other follow-up work (separate PR)

This record mainly covers Phases 1–3 and a few do-overs that happened along the way.

Before starting: simulate using read-only commands

Before doing anything destructive, I ran read-only commands as a pre-flight check.

This is the step that reduces "things you only notice once you're in production."

What I checked:

# Profile check
aws sts get-caller-identity --profile <prod-profile>

# VPC existence check
aws ec2 describe-vpcs --vpc-ids <vpc-id> --profile <prod-profile> --region us-west-2

# Canary SG duplicate check
aws ec2 describe-security-groups \
  --filters "Name=group-name,Values=<canary-sg-name>" \
            "Name=vpc-id,Values=<vpc-id>" \
  --profile <prod-profile> --region us-west-2

# Enumerate ECS SGs
for NAME in <ecs-sg-name-1> <ecs-sg-name-2> ...; do
  SG_ID=$(aws ec2 describe-security-groups \
    --filters "Name=group-name,Values=$NAME" \
    --query 'SecurityGroups[0].GroupId' --output text \
    --profile <prod-profile> --region us-west-2)
  echo "$NAME = $SG_ID"
done

The result: "Canary SG not yet created, all ECS SGs present, Pulumi.prod.yml still has the placeholder." With that confirmed, I felt comfortable moving on to Phase 1.

The simulation itself takes less than five minutes, but the psychological effect of being able to say "OK, I'm cleared to execute" is huge.

Phase 1: Create the Canary SG

# Create the SG
MSYS_NO_PATHCONV=1 aws ec2 create-security-group \
  --group-name <canary-sg-name> \
  --description "Security group for Synthetics Canary" \
  --vpc-id <vpc-id> \
  --tag-specifications 'ResourceType=security-group,Tags=[{Key=Name,Value=<canary-sg-name>}]' \
  --profile <prod-profile> --region us-west-2

# Remove the default egress (allow all)
MSYS_NO_PATHCONV=1 aws ec2 revoke-security-group-egress \
  --group-id <canary-sg-id> \
  --ip-permissions '[{"IpProtocol":"-1","IpRanges":[{"CidrIp":"0.0.0.0/0"}]}]' \
  --profile <prod-profile> --region us-west-2

# Allow HTTP (80)
MSYS_NO_PATHCONV=1 aws ec2 authorize-security-group-egress \
  --group-id <canary-sg-id> \
  --ip-permissions '[{"IpProtocol":"tcp","FromPort":80,"ToPort":80,"IpRanges":[{"CidrIp":"0.0.0.0/0"}]}]' \
  --profile <prod-profile> --region us-west-2

# Allow HTTPS (443)
MSYS_NO_PATHCONV=1 aws ec2 authorize-security-group-egress \
  --group-id <canary-sg-id> \
  --ip-permissions '[{"IpProtocol":"tcp","FromPort":443,"ToPort":443,"IpRanges":[{"CidrIp":"0.0.0.0/0"}]}]' \
  --profile <prod-profile> --region us-west-2

MSYS_NO_PATHCONV=1 is required because of a Git Bash quirk.

Git Bash tries to convert Windows-style paths automatically, which means strings like 0.0.0.0/0 get rewritten too. Setting this environment variable disables the conversion.

After creation, ingress was empty and egress only allowed HTTP/HTTPS — exactly what I wanted. Onward.

Phase 2: Add ingress rules to existing ECS SGs

With the Canary SG ID locked in, I added a "TCP/80 from the Canary SG" ingress rule to each of the ECS SGs being monitored.

MSYS_NO_PATHCONV=1 aws ec2 authorize-security-group-ingress \
  --group-id <ecs-sg-id> \
  --ip-permissions '[{
    "IpProtocol":"tcp",
    "FromPort":80,
    "ToPort":80,
    "UserIdGroupPairs":[{
      "GroupId":"<canary-sg-id>",
      "Description":"Allow from Synthetics Canary SG"
    }]
  }]' \
  --profile <prod-profile> --region us-west-2

I repeated this for each target ECS SG (six this time).

Finally, a reverse-lookup check. The "find SGs that allow ingress from the Canary SG" filter is handy here.

MSYS_NO_PATHCONV=1 aws ec2 describe-security-groups \
  --filters "Name=ip-permission.group-id,Values=<canary-sg-id>" \
  --profile <prod-profile> --region us-west-2 \
  --query 'SecurityGroups[*].GroupName'

All six matched. On to Phase 3.

Phase 3: Update Pulumi.prod.yml and commit

Replace the placeholder in Pulumi.prod.yml with the Canary SG ID created in Phase 1.

# Before
cm-iac-monitoring:canarySecurityGroupId: sg-xxxxxxxxxxxxxxxxx

# After
cm-iac-monitoring:canarySecurityGroupId: <newly created Canary SG ID>

Commit, push, and switch the PR to Ready for review.

The do-over: .yml vs .yaml

After running the Jenkins preview in Phase 5, I got this error:

error: Missing required configuration variable 'cm-iac-monitoring:vpcSubnetIds'
    please set a value using the command `pulumi config set cm-iac-monitoring:vpcSubnetIds <value>`

"Wait, vpcSubnetIds should be in Pulumi.prod.yml…" — and the cause turned out to be a file rename.

In the middle of the work, someone said "isn't .yaml more correct than .yml?" and I had renamed Pulumi.prod.yml to Pulumi.prod.yaml. That was the cause of the error.

How Pulumi finds stack config files

Pulumi looks up stack config files using the same extension as the project file (Pulumi.yml).

✅ Pulumi.yml + Pulumi.prod.yml     → works
✅ Pulumi.yaml + Pulumi.prod.yaml   → works
❌ Pulumi.yml + Pulumi.prod.yaml    → config is not loaded, error

The project file in this repo was Pulumi.yml (.yml), so the stack file had to be .yml too. Renaming it to .yaml made Pulumi lose its config, which surfaced as a "missing vpcSubnetIds" error.

State stores resource information, not config values. So the assumption that "it's in state, so we're fine" doesn't hold here — that was another lesson confirmed.

.yml vs .yaml — why both exist

While we're here, a quick aside on why both .yml and .yaml are in the wild.

.yaml is what the YAML spec (yaml.org) recommends. .yml is a leftover from the old Windows 8.3 filename limit (FAT16 era — extensions could only be three characters). It's the same backstory as .htm vs .html, or .jpg vs .jpeg.

The 8.3 limit is long gone, but established projects and conventions linger, so both extensions continue to coexist. For new projects, .yaml is the spec-correct choice, but rewriting an existing project carries a real cost. I felt that cost firsthand today.

The follow-up: enabling snsEnabled

After the deploy, I flipped each endpoint's snsEnabled from false to true. I had set it to false for the initial deploy on purpose (verify behavior first), then enabled it once I was confident things were working.

# Before
- name: <endpoint-name>
  ...
  snsEnabled: false

# After
- name: <endpoint-name>
  ...
  snsEnabled: true

Applied to every endpoint, then commit & push.

Retrospective

A few things I felt all over again from the day's work.

Don't skip the pre-flight simulation. Spending 5–10 minutes on read-only checks completely changes the psychological state of the operator: you can say "I verified everything before starting." It prevents that mid-operation moment of "wait, does this SG even exist?"

Match the extension on Pulumi stack config files. This will catch you if you don't know it. The error message comes through as "config is missing," so it's not obvious that the cause is the filename.

Estimate the blast radius before changing naming conventions. The .yml → .yaml change looks like a simple rename, but for tools like Pulumi that rely on extension matching, it changes behavior. Even if it's only under monitoring this time, if other places follow the same pattern you have to change them all at once or risk extra confusion.

I used Claude Code for parts of the work today. I had it pre-read the AWS operation steps and walk through each phase with me. Compared to typing commands by hand, it was nice to be confirming "what is this command doing right now" and "what should I be checking" together as a set.

The next step is to migrate the SG management I still did by hand today over to Terraform. That will be another article.

Further reading

If you want a systematic look at AWS operations and troubleshooting, the following book is a useful reference. It's helpful for organizing the kind of pre-flight checks and CLI workflows discussed here.