5 Pitfalls in CloudWatch SDK Migration: A Debug Chronicle of LLM Model Updates and Metrics Retrieval

We migrated our in-house CloudWatch monitoring system from a shell script-based Jenkinsfile to a new Python SDK-based Jenkinsfile.

"If we switch to SDK, it'll be easier to maintain and we can write tests." That's what we thought when we started the migration, but honestly, that's where the real journey began. As soon as we fixed one problem, another appeared—a classic "whack-a-mole" debugging session.

In this article, I'll document the 5 pitfalls we encountered during migration and the debugging process for each, in chronological order.

Context: What We Were Building

We have a monitoring pipeline that periodically fetches CPU metrics from CloudWatch, analyzes them with an LLM (Azure OpenAI), and automatically generates Markdown reports.

CloudWatch → Fetch Metrics → Threshold Check → Generate Graph Images → LLM Analysis → Output Report

Originally, the Jenkinsfile directly invoked AWS CLI to fetch metrics, processed them with jq, and passed the data to the LLM. This migration consolidated everything into Python SDK for better testability.

Pitfall 1: max_tokens No Longer Supported

This was the first error we encountered after migration:

Error code: 400 - {'error': {'message': "Unsupported value: 'max_tokens' is not
supported with this model. Use 'max_completion_tokens' instead.", ...}}

When we updated the LLM model to a newer version, the max_tokens parameter was deprecated, and we needed to use max_completion_tokens instead.

The fix itself was straightforward:

# Before
response = client.chat.completions.create(
    model=config.deployment_name,
    messages=messages,
    max_tokens=config.max_tokens,        # ← Deprecated
)

# After
response = client.chat.completions.create(
    model=config.deployment_name,
    messages=messages,
    max_completion_tokens=config.max_completion_tokens,  # ← New parameter
)

We updated the dataclass definition and configuration file (YAML) accordingly. This one was easy to fix.

Lesson Learned: Always check API parameter compatibility when upgrading models. It was written in the release notes, but we overlooked it.

Pitfall 2: temperature Also Unsupported

Fixed the first issue and re-ran. Another 400 error appeared:

Unsupported value: 'temperature' does not support 0.7 with this model.
Only the default (1) value is supported.

The new model didn't support the temperature parameter at all—it only accepted the default value (1).

We removed the temperature parameter from the API call:

response = client.chat.completions.create(
    model=config.deployment_name,
    messages=messages,
    max_completion_tokens=config.max_completion_tokens,
    # Don't specify temperature (use model default)
)

Lesson Learned: After fixing one parameter, check if other parameters work with the same model. If you think "it's fixed" and move on, you might miss similar issues.

Pitfall 3: Empty Metrics JSON—The Surprising Culprit

This is where the real challenge began.

The LLM errors were resolved, but now daily_metrics.json and weekly_metrics.json fetched from CloudWatch were empty. Zero datapoints.

{
  "datapoints": [],
  "statistics": {
    "max_cpu": 0.0,
    "avg_cpu": 0.0,
    "points": 0
  }
}

Graph images (via GetMetricWidgetImage API) were generated successfully, but statistics data (via GetMetricStatistics API) was empty. Same credentials, same dimension specifications—why did only one work?

Investigation 1: Verify Metrics Exist

First, we confirmed whether the data actually existed in CloudWatch:

aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=AutoScalingGroupName,Value=<ASG name> \
  --start-time 2026-02-08T00:00:00Z \
  --end-time 2026-02-09T12:00:00Z \
  --period 3600 \
  --statistics Maximum Average \
  --region us-west-2

Result: 34 datapoints retrieved. The data exists.

Investigation 2: Suspecting IAM Permissions

It worked with admin permissions but not via AssumeRole. That pointed to IAM permissions.

When fetching CloudWatch metrics using Auto Scaling Group dimensions, AWS might verify the ASG's existence behind the scenes. We added these permissions to the IAM role:

# Auto Scaling read permissions
- autoscaling:DescribeAutoScalingGroups
- autoscaling:DescribeAutoScalingInstances

# EC2 read permissions
- ec2:DescribeInstances
- ec2:DescribeInstanceStatus
- ec2:DescribeTags

We updated the CloudFormation stack and propagated the permissions to cross-account roles.

...But still empty.

Investigation 3: Comparing with Old Jenkinsfile

We changed our approach here. "The old Jenkinsfile worked, but the new SDK doesn't." If so, we should look at the differences.

The old Jenkinsfile directly invoked AWS CLI. The new SDK uses boto3's CloudWatch client. Reading through the boto3 client initialization code, we found this:

client = boto3.client("cloudwatch", **client_kwargs)

# Event hooks added for compatibility with test framework (moto)
client.meta.events.register(
    "before-parameter-build.cloudwatch.GetMetricStatistics",
    self._normalize_cloudwatch_params
)
client.meta.events.register(
    "before-call.cloudwatch.GetMetricStatistics",
    self._normalize_cloudwatch_request_body
)

Surprisingly, this was the key.

These event hooks were added for compatibility with the test mock framework (moto). They intercept and rewrite parameters before boto3 issues API requests.

While they work correctly in test environments, in production they unexpectedly modified CloudWatch API request parameters, resulting in empty responses.

# Fix: Remove event hooks
client = boto3.client("cloudwatch", **client_kwargs)
# No hook registration. Let boto3 make standard API calls

This also explains why GetMetricWidgetImage worked. The event hooks were only registered for GetMetricStatistics, so GetMetricWidgetImage wasn't affected.

Lesson Learned: Test-specific code affecting production is easy to overlook. "Works in test environment" doesn't guarantee "works in production." Low-level interventions like boto3 event hooks require special attention.

Pitfall 4: Prompt Too Large for LLM

Metrics were now being fetched successfully, and datapoints were properly populated in JSON. But now the LLM analysis returned empty responses:

OpenAI API call failed (3 attempts):
Empty response returned from OpenAI API.

No error code. The API call succeeded, but the response content was empty.

We calculated the data size:

Weekly datapoints: 168 points → 26,590 characters
Daily datapoints:  288 points → 45,247 characters
Total: 71,837 characters ≒ approximately 18,000 tokens

We were sending all datapoints as raw JSON. Combined with the prompt template, this likely exceeded the model's input limit.

The old Jenkinsfile only sent statistical summaries (max, avg, points). The new SDK implementation was sending all datapoints—that was the issue.

# Before: Send all datapoints (~18,000 tokens)
"weekly_metrics": json.dumps(weekly_metrics, ensure_ascii=False, indent=2)

# After: Send only statistical summary (a few hundred tokens)
weekly_summary = {
    "statistics": weekly_metrics.get("statistics", {}),
    "period": weekly_metrics.get("period", {}),
    "status": weekly_metrics.get("status", ""),
    "warnings": weekly_metrics.get("warnings", []),
    "datapoints_count": len(weekly_metrics.get("datapoints", []))
}
"weekly_metrics": json.dumps(weekly_summary, ensure_ascii=False, indent=2)

Lesson Learned: Design data passed to LLMs with "necessary granularity for analysis" in mind. Sending raw data wholesale isn't just token-inefficient—it risks hitting the model's input limit.

Pitfall 5 (Bonus): Why Not Analyze Images Too?

At this point, the monitoring pipeline was working. But then I realized something:

"We're generating graph images, but we're only sending statistical values to the LLM. Shouldn't we have it analyze the images too?"

The Vision functionality could capture trends and spike patterns that statistical values alone might miss.

# Build user message
user_content = [{"type": "text", "text": prompt}]

# Add graph images
for image_path in image_paths:
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode("utf-8")
    user_content.append({
        "type": "image_url",
        "image_url": {"url": f"data:image/png;base64,{image_data}"}
    })

messages = [
    {"role": "system", "content": "You are a CloudWatch metrics analysis expert "
     "with the ability to analyze both numerical data and visual graphs."},
    {"role": "user", "content": user_content},
]

By sending weekly and daily graph images together with statistical summaries, we enabled analysis from both numerical data and visual pattern perspectives.

Retrospective: The Complete Flow

As we fixed one issue, the next became visible. By solving these 5 problems sequentially, we ended up with a better monitoring pipeline than we started with.

Summary of Decision Points

The third issue—"Empty metrics JSON"—is where our initial hypothesis (insufficient IAM permissions) was wrong. It was reasonable to suspect IAM when it worked with admin permissions but not via AssumeRole, but the actual cause was at a lower layer.

Switching to diff comparison based on "it worked in the old version" was ultimately the right call.

Conclusion

Migration work seems simple at first glance—"just convert something that works into a different form." But in reality, these small compatibility issues accumulate.

The "test code breaking production" pattern was particularly striking in this case. Event hooks added for moto test compatibility modified production CloudWatch API calls. Writing tests is a good practice, but we need to be careful that test-specific code doesn't intrude into production code paths.

We spent several hours thinking "no choice but to push forward" as we tackled each issue, but we ultimately reached a better state by adding image analysis. Sometimes the detour is worth it.

Reference Books

For those who want to learn more about AWS operations and monitoring design systematically, the following books are highly recommended:

Reading these alongside your actual operational experience will help you build a better toolkit for handling issues like the ones we encountered.