Investigating an IAM User Nobody Remembers Creating — And Hitting CloudTrail's Retention Wall

One day I noticed an IAM user in our company's AWS account that nobody could account for — no one knew who created it, when, or why.

You know the type. It's just there. No tags, no documentation, no matching Slack conversations.

Caught between "leaving it is scary" and "deleting it is scary," I decided to investigate first. Here's the investigation log, worked through with Claude Code, organized with lessons learned.

The situation

First goal: find out who created it.

Investigation 1: check CloudTrail

For finding who created an IAM user, looking up the CreateUser event in CloudTrail is the standard move.

# Check user info
aws iam get-user --user-name external-service-user

# Search CloudTrail by username
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=external-service-user \
  --max-results 50

Result: "Events": []. Empty.

Tags were empty too. Zero information about the creator.

Investigation 2: the CloudTrail retention wall

I looked into why no events came back.

# Check CloudTrail log settings
aws cloudtrail list-trails --region ap-northeast-1

# Check CloudWatch Logs integration
aws cloudtrail get-trail --name <trail-arn>

What I found: the log retention wall.

The user was created about 7 months ago. The 90-day CloudTrail event history was obviously expired. CloudWatch Logs at 77 days was also expired.

The logs should exist in S3, but they're stored in a Log Archive account under AWS Organizations — and my current IAM role didn't have access.

AccessDenied: User is not authorized to perform s3:ListBucket

A painful error. "The logs exist, but you can't see them."

An important design lesson

Even with CloudTrail enabled, underestimating retention design will leave you stuck. That's one of this investigation's takeaways.

In this setup, Control Tower's centralized log management (Log Archive account) was in place, but the investigating account didn't have permissions to view that S3 — that was the sticking point.

Investigation 3: reverse-trace through access key usage

I couldn't identify the creator. But investigation continues.

Even without knowing "who created it," I can check "where is this key being used from right now."

aws iam get-access-key-last-used --access-key-id <access-key-id>

What came back:

It's active. And recent.

Then searching CloudTrail by access key ID:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=<access-key-id>

This returned interesting information.

Caller details:

The User-Agent reveals a lot:

Looking up the source IPs through IPinfo pointed to a specific cloud provider's Tokyo region.

The calling pattern was clear

Going through all CloudTrail entries, the pattern was unmistakable:

Every entry across 90 days followed this pattern. A scheduled batch job runs every Monday and has been failing with AccessDenied the entire time.

No actual data access is happening because there are no permissions — but the key is still in active use.

The full picture of the access keys

Summarizing what the investigation revealed:

The fact that a second key was issued is notable. Likely "the first one didn't work so they created another" — but the issuer is also beyond the 90-day window and can't be identified.

Recommended response: don't delete immediately

The tough judgment here: "I want to delete this, but there's a running batch job that uses it."

Honestly, I never found out what this key was originally created for. But the decision was "gradual response" rather than "immediate deletion."

Recommended steps

Step 1: Inactivate key 1 (unused for 7 months)
         → Low risk, can execute immediately

Step 2: Contact the external service's responsible person
         → Which system is using this?
         → Is the GetBucketAcl failure expected?

Step 3: Issue a new key and share via secure method (password manager, etc.)

Step 4: After confirming switchover, Inactivate key 2 → monitor → Delete

As "minimum actions if nothing is urgent," I proposed inactivating the unused key and reaching out to the responsible party.

AWS services useful for key management

AWS services for IAM key management, organized during this investigation.

IAM Credential Report (great for inventory)

# Generate report
aws iam generate-credential-report

# Retrieve and decode
aws iam get-credential-report \
  --output text --query Content | base64 -d > credential_report.csv

Outputs a CSV listing all IAM users and key states across the account. Key creation dates, last-used dates, and MFA status all visible at once. The most convenient tool for inventory.

Note: cached for 4 hours.

IAM Access Analyzer - Unused Access

AWS Config Rules

Enable managed rules for detection:

Can be combined with SNS notifications or auto-remediation.

Long-term: IAM Roles + AssumeRole

For "external service accessing AWS" scenarios like this one, IAM roles + AssumeRole is more appropriate than long-lived access keys.

No need to hand secret keys to the other party. The "what if the key is leaked" problem ceases to exist. External IDs enable cross-account control.

Lessons learned

What I was able to organize through this investigation:

Build CloudTrail retention into your design

Relying solely on the 90-day CloudTrail event history won't support investigations six months later. You need long-term S3 storage configuration and access permission design for that storage.

User-Agent carries surprisingly rich information

It can include .NET versions, OS build numbers, and more. It becomes a clue for estimating what system is making the calls.

Don't leave "keys nobody remembers" sitting around

IAM keys with unknown creators, recipients, and purposes are problematic from both a leak-risk and management perspective. Regular Credential Report inventory and IAM Access Analyzer usage reduce these "unnoticed" states.

The importance of "verify before deleting"

This time, I paused when I discovered "there's a batch job running every Monday." If I'd deleted immediately, it could have impacted the external service's jobs. Checking impact scope first sounds obvious but gets skipped when you're rushing.

In the end, "who created it" remained unknown. The CloudTrail wall couldn't be overcome.

But "what's happening" and "what to do next" became clear. The work of organizing "unknown" into "organized unknown" is unglamorous, but it's very much SRE work.

Further reading

If you want to go deeper on AWS operational design and practical IAM/CloudTrail usage, this book is a good starting point.