Stuck on CloudFormation Cross-Account IAM Role Order

Conclusion: Trust Policies Validate Principal ARN Existence

Let me start with the conclusion.

When building cross-account AssumeRole configurations with CloudFormation, the trust policy on the target account side validates whether the referenced IAM role ARN actually exists. If you try to create a stack while the AssumeRole source role doesn't exist, it will fail with Invalid principal in policy.

The correct creation order is as follows:

graph TD
    A[1. Create/update IAM role in CI account] --> B[2. Confirm role ARN]
    B --> C[3. Create cross-account role in target account A]
    B --> D[3. Create cross-account role in target account B]
    B --> E[3. Create cross-account role in target account C]
    C --> F[4. Add role ARNs to CI-side AssumeRole permissions]
    D --> F
    E --> F

This might seem obvious. However, when working across multiple accounts, it's easy to lose track of this order. That's exactly what happened to me.

Context: What I Was Trying to Do

The goal was to set up monitoring of resources across multiple AWS accounts from a CI server (Jenkins).

[CI Account]                      [Target Accounts A, B, C]
  Jenkins Agent                     Monitoring Role
  (IAM Role)   --- AssumeRole --->  (Cross-Account Role)

Let me clarify "AssumeRole" and "Trust Policy" first.

In AWS, one account's IAM role can "assume" another account's IAM role for cross-account access. The role being assumed needs a configuration that defines who is allowed to assume it. This is the Trust Policy.

So there are two players:

• IAM Policy (Permission Policy): Set on the CI account side. Defines "what resources this role can access and what actions it can perform." In this case, "permission to AssumeRole into target account roles"
• Trust Policy: Set on the target account's role. Defines "who is allowed to assume this role." In this case, "allow AssumeRole from the CI account's Jenkins role"

Both are required for cross-account AssumeRole to work. They're like two wheels on a bicycle.

I prepared two CloudFormation templates:

1. CI Account Side: IAM role for Jenkins agent (with IAM policy granting AssumeRole permissions)
2. Target Account Side: Monitoring role (with trust policy referencing the CI account role ARN)

Deploy to each account and it should work—or so I thought.

Issue #1: Name Conflict with Existing Roles

Conclusion: CloudFormation cannot create IAM roles with the same name from different stacks within the same account and region.

What Happened: Stack creation rolled back.

Resource already exists in stack arn:aws:cloudformation:...

Cause: A previously created stack for a different purpose already had an IAM role with the same name. Naming conventions weren't standardized—some existing roles used {purpose}-role-{environment}, others used {stack-name}-role-{environment}. This ambiguity was the root cause.

Resolution: Changed the role name prefix as a workaround. At this point, I realized "we need to standardize naming conventions," which led to further unification work later.

Issue #2: Invalid principal in policy

Conclusion: Stack creation fails if the IAM role ARN specified in the trust policy doesn't exist.

What Happened: After resolving the naming conflict and redeploying, a different error appeared:

Invalid principal in policy

Cause: I was trying to create a trust policy on the target account side that referenced the new role name before actually changing the role name on the CI account side. The role ARN being referenced didn't exist yet, so it was rejected.

I got stuck here. I checked the ARN multiple times for typos, but the problem wasn't the correctness of the ARN—it was that the role itself didn't exist.

sequenceDiagram
    participant CI as CI Account
    participant Target as Target Account

    Note over Target: ❌ Failure Pattern (Role Not Created)
    Target->>CI: Reference CI role ARN in trust policy
    CI-->>Target: That role doesn't exist yet
    Target->>Target: Invalid principal → Rollback

    Note over CI,Target: ✅ Success Pattern (Role Exists First)
    CI->>CI: 1. Create agent role in CI account
    Target->>CI: 2. Reference CI role ARN in trust policy
    CI-->>Target: Role exists → Validation OK
    Target->>Target: Monitoring role created successfully

Key Point: I discovered an important asymmetry here:

• Trust Policy: Validates the existence of the ARN specified in the principal. Fails if the target doesn't exist
• IAM Policy (Permission Policy): Can use ARN patterns (wildcards like arn:aws:iam::*:role/monitoring-*). Can be set even if the target doesn't exist

In other words, the dependency direction is asymmetric. The CI-side policy can be written first, but the target-side trust policy can't be written until the CI role exists. This was the biggest learning from this work.

Bonus: Double Replacement in Bulk sed Replacement of 25 Files

After standardizing the role name, there were 25 files referencing it: configuration files, Jenkinsfiles, Groovy scripts, documentation, etc. I used sed for bulk replacement, but there was a small trap here too.

Since the replacement target string contained part of the new prefix, double replacement occurred.

# Bulk replacement
find path/to/jobs -name "Jenkinsfile" -type f \
  -exec sed -i "s/old-role-name/new-role-name/g" {} \;

# Expected: old-role-name → new-role-name
# Actual: old-role-name → new-new-role-name (double replacement)

The fix was brute force—another sed to correct the double-replaced strings.

find path/to/jobs -name "Jenkinsfile" -type f \
  -exec sed -i "s/new-new-role-name/new-role-name/g" {} \;

I should have verified targets with grep first and used a pattern closer to exact match. When rushing, you lose time on these mundane issues.

Summary

Here are the key learnings:

• Cross-account AssumeRole trust policies validate the existence of the referenced role. Wrong creation order results in Invalid principal errors
• Trust policies and IAM policies have asymmetric dependency directions. IAM policies can use wildcards and don't depend on target existence, but trust policies validate existence
• For multi-account setups, draw dependency diagrams before starting work. Managing in your head leads to losing track of order when switching between accounts
• Standardize naming conventions during initial design. Fixing later only increases the blast radius

Especially the second point about asymmetry—while it's documented, you don't truly internalize it until you actually get stuck. Hope this helps someone.

Reference Documentation

• AWS IAM: Creating a role to delegate permissions to an IAM user
• AWS IAM: IAM role trust policies
• AWS CloudFormation: AWS::IAM::Role
• AWS IAM: Tutorial: Delegate access across AWS accounts using IAM roles

Reference Books

For those who want to learn more about AWS operations and infrastructure automation, here are some recommended books:

[📦 商品リンク: moshimo-book-aws-operations]

[📦 商品リンク: moshimo-book-server-infra]