From a Pulumi Bug to Terraform: 3 Pitfalls in Migrating Between IaC Tools

Background and motivation

In a repository that manages our synthetic monitoring (CloudWatch Synthetics Canary) configuration, pulumi refresh suddenly started failing one day.

The cause was a regression in the Pulumi AWS Provider: running refresh against SecurityGroupIngressRule resources errored out. It was a known issue with no quick fix in sight.

I decided that continuing to manage things in Pulumi was no longer viable, and chose to move the security group (SG) for the Canary over to Terraform.

I assumed the migration itself would be relatively simple. In practice, I walked into three pitfalls. This is the record of that journey.

Architecture overview

Here is what the setup looked like before the migration.

Pulumi created the Canary SG and added an ingress rule to each ECS service SG that allowed TCP/80 from the Canary. After the migration, things look like this.

Only SG management is handed over to Terraform. Pulumi is left to focus on the Canary resources themselves.

Pitfall 1: Mixing inline rules with aws_security_group_rule

There are two ways to define ingress/egress rules for a security group in Terraform:

The existing code used the inline style, so I matched that for the new Canary SG.

resource "aws_security_group" "canary_sg" {
  name        = "your-app-synthetics-canary-sg-${var.env}"
  description = "Security group for CloudWatch Synthetics Canary"
  vpc_id      = var.vpc_id

  ingress = []  # The Canary SG itself accepts no inbound traffic

  egress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

I also added inline ingress rules to each ECS service SG.

There is something to be careful about here: you must not mix inline rules with separate rule resources (like aws_security_group_rule).

The Terraform AWS Provider can fall into a "plan/apply infinite loop" where rules disappear and reappear on every apply when the two styles are mixed. The original Pulumi-side issue (rules disappearing on apply) was very likely caused, in part, by exactly this kind of mixing.

I went with the rule "match the existing inline style and stay consistent." Eventually consolidating on aws_vpc_security_group_ingress_rule is an option, but I treated that as a separate piece of work and deliberately kept it out of scope.

Why I split the work into two PRs

At first I bundled the Terraform and Pulumi changes into a single PR.

That was a mistake. The Terraform apply and the Pulumi deploy have a strict ordering dependency.

If you break that order, Pulumi tries to destroy an SG that no longer exists in its world, or you end up with old and new rules coexisting on AWS. Both are dangerous.

A reviewer also explicitly asked me to split the Terraform and Pulumi changes into two PRs, so I broke things up like this:

PR-B can't be merged until PR-A's apply has completed, so I opened it as a Draft. Once PR-A had been applied and the actual SG ID was known, I wrote that ID into the config and then flipped PR-B to Ready for review.

Reviewing the Terraform plan

To validate PR-A in the dev environment, I ran terraform plan locally.

terraform plan -var-file ./tfvars/secrets_dev.tfvars

Summary of the result:

Plan: 1 to add, 6 to change, 0 to destroy.

A destroy count of zero is the important part. I confirmed that nothing was being torn down before moving on to apply.

I pasted the plan output directly into the PR comment as well. Leaving "this is what I verified" as a log makes it easier for reviewers to make a call, and it helps me when I look back at the work later.

Pitfall 2: IAM eventual consistency

After PR-A was merged and applied, I ran the Pulumi deploy for PR-B and hit this error:

CREATE_FAILED: The provided execution role does not have permissions
to call CreateNetworkInterface on EC2
(AWSLambda; InvalidParameterValueException)

Of the five Canaries, only one failed.

The cause is AWS IAM eventual consistency.

I had cleared the Pulumi state once and recreated everything from scratch, which meant the Canary creation kicked off immediately after the IAM role and policy were created. IAM changes don't propagate instantly — there is a delay of tens of seconds. The fact that four out of five Canaries succeeded and only one failed is a textbook symptom of this timing gap.

Fix: Insert an explicit wait with @pulumiverse/time

In Terraform there's a well-known pattern of inserting a time_sleep resource. Pulumi has an equivalent package, @pulumiverse/time.

import * as time from "@pulumiverse/time";

// Assign the IAM RolePolicy to a variable so we can reference it in dependsOn
const canaryRolePolicy = new aws.iam.RolePolicy(`${namePrefix}-canary-policy-${environment}`, {
  name: `${namePrefix}-canary-policy-${environment}`,
  role: canaryRole.name,
  policy: /* ... */,
});

// Wait for IAM propagation (eventual consistency mitigation).
// Creating a Canary right after the IAM role/policy can fail with
// "CreateNetworkInterface permission not found".
// A 30-second sleep stabilizes things.
const canaryIamPropagation = new time.Sleep(
  `${namePrefix}-canary-iam-propagation`,
  { createDuration: "30s" },
  { dependsOn: [canaryRolePolicy] },
);

// Add dependsOn to each Canary
new aws.synthetics.Canary(
  canaryName,
  { /* ... */ },
  {
    replaceOnChanges: ["runtimeVersion"],
    deleteBeforeReplace: true,
    dependsOn: [canaryIamPropagation],  // <- add this
  },
);

The key point is assigning canaryRolePolicy to a variable. The original code had new aws.iam.RolePolicy(...) as a bare statement, so there was no handle to reference from dependsOn.

After this fix, typecheck and tests all passed. New creations cost an extra 30 seconds, but updates to existing resources don't trigger the sleep. That's an acceptable tradeoff.

For the record: this sleep pattern is not directly recommended by AWS. It's a de facto standard workaround in the Terraform and Pulumi communities. AWS docs do state that IAM is eventually consistent, but they do not prescribe specific countermeasures like a sleep duration.

Pitfall 3: Leftover ERROR-state Canaries causing state drift

After adding the IAM propagation fix and re-running pulumi up, I hit a different error:

ConflictException: Canary is in a state that can't be modified: ERROR

When the previous pulumi up failed to create the Canary, the Canary was left behind on AWS in an ERROR state. Pulumi state still considered it "present," so this run treated it as an update — but AWS refuses to modify an ERROR-state Canary, hence the error.

The IAM propagation fix itself was fine. The leftover from the previous failure was getting in the way.

Fix: Delete via AWS CLI, then pulumi refresh to sync state

First, delete the ERROR-state Canary directly with the AWS CLI.

# Delete the ERROR-state Canary
aws synthetics delete-canary \
  --name your-app-canary-service-dev \
  --region us-west-2

Next, sync Pulumi state with the actual state on AWS. Because resources that have been deleted are removed automatically from Pulumi state on refresh, you don't need to run pulumi state delete by hand.

# Run via Jenkins or CI
pulumi refresh  # Sync state with the real resources
pulumi preview  # Check the diff
pulumi up       # Apply (the Canary is recreated, with the IAM propagation fix in place)

That resolved the issue and the dev deploy completed.

Retrospective: what today taught me

Migrating between IaC tools is all about ordering

You must respect the Terraform apply → Pulumi deploy order, and as a precondition for that, you need to split the work into separate PRs. Bundling both changes into one PR makes review harder and raises the risk of applying things in the wrong order.

Don't mix Terraform inline ingress with separate rule resources

This was an indirect cause of the original Pulumi-side problem. Pick one style and stick with it. When in doubt, follow the existing convention in the codebase.

AWS IAM eventual consistency happens in production too

Inserting a 30-second wait via @pulumiverse/time's time.Sleep is the realistic countermeasure today. It was the right call to assume "this will happen in prod too" and put the fix in place during the dev rollout.

Document the cleanup procedure for ERROR-state resources

This time I had the flexibility to handle it because it was the dev environment. If the same thing happens in production, it could turn into a late-night incident. I was reminded how important it is to write down "what to do when things fail" ahead of time.

Current status and next steps

For production, the plan is to follow the same flow: create the SG, update the config file, run Terraform apply, then run Pulumi deploy. Thanks to the trial and error in dev, I have a pretty good map of the pitfalls before stepping into prod.

Further reading

If you want to systematically learn the operational pitfalls and best practices around AWS, the following book is a useful reference. It helps put situations like the IAM behavior and resource management discussed here into a broader context.

For those who want to dive deeper into Terraform itself, this book is also a great resource.