We migrated our in-house CloudWatch monitoring system from a shell script-based Jenkinsfile to a new Python SDK-based Jenkinsfile.
"If we switch to SDK, it'll be easier to maintain and we can write tests." That's what we thought when we started the migration, but honestly, that's where the real journey began. As soon as we fixed one problem, another appeared—a classic "whack-a-mole" debugging session.
In this article, I'll document the 5 pitfalls we encountered during migration and the debugging process for each, in chronological order.
Context: What We Were Building
We have a monitoring pipeline that periodically fetches CPU metrics from CloudWatch, analyzes them with an LLM (Azure OpenAI), and automatically generates Markdown reports.
CloudWatch → Fetch Metrics → Threshold Check → Generate Graph Images → LLM Analysis → Output Report
Originally, the Jenkinsfile directly invoked AWS CLI to fetch metrics, processed them with jq, and passed the data to the LLM. This migration consolidated everything into Python SDK for better testability.
Pitfall 1: max_tokens No Longer Supported
This was the first error we encountered after migration:
Error code: 400 - {'error': {'message': "Unsupported value: 'max_tokens' is not
supported with this model. Use 'max_completion_tokens' instead.", ...}}
When we updated the LLM model to a newer version, the max_tokens parameter was deprecated, and we needed to use max_completion_tokens instead.
The fix itself was straightforward:
# Before
response = client.chat.completions.create(
model=config.deployment_name,
messages=messages,
max_tokens=config.max_tokens, # ← Deprecated
)
# After
response = client.chat.completions.create(
model=config.deployment_name,
messages=messages,
max_completion_tokens=config.max_completion_tokens, # ← New parameter
)
We updated the dataclass definition and configuration file (YAML) accordingly. This one was easy to fix.
Lesson Learned: Always check API parameter compatibility when upgrading models. It was written in the release notes, but we overlooked it.
Pitfall 2: temperature Also Unsupported
Fixed the first issue and re-ran. Another 400 error appeared:
Unsupported value: 'temperature' does not support 0.7 with this model.
Only the default (1) value is supported.
The new model didn't support the temperature parameter at all—it only accepted the default value (1).
We removed the temperature parameter from the API call:
response = client.chat.completions.create(
model=config.deployment_name,
messages=messages,
max_completion_tokens=config.max_completion_tokens,
# Don't specify temperature (use model default)
)
Lesson Learned: After fixing one parameter, check if other parameters work with the same model. If you think "it's fixed" and move on, you might miss similar issues.
Pitfall 3: Empty Metrics JSON—The Surprising Culprit
This is where the real challenge began.
The LLM errors were resolved, but now daily_metrics.json and weekly_metrics.json fetched from CloudWatch were empty. Zero datapoints.
{
"datapoints": [],
"statistics": {
"max_cpu": 0.0,
"avg_cpu": 0.0,
"points": 0
}
}
Graph images (via GetMetricWidgetImage API) were generated successfully, but statistics data (via GetMetricStatistics API) was empty. Same credentials, same dimension specifications—why did only one work?
Investigation 1: Verify Metrics Exist
First, we confirmed whether the data actually existed in CloudWatch:
aws cloudwatch get-metric-statistics \
--namespace AWS/EC2 \
--metric-name CPUUtilization \
--dimensions Name=AutoScalingGroupName,Value=<ASG name> \
--start-time 2026-02-08T00:00:00Z \
--end-time 2026-02-09T12:00:00Z \
--period 3600 \
--statistics Maximum Average \
--region us-west-2
Result: 34 datapoints retrieved. The data exists.
Investigation 2: Suspecting IAM Permissions
It worked with admin permissions but not via AssumeRole. That pointed to IAM permissions.
When fetching CloudWatch metrics using Auto Scaling Group dimensions, AWS might verify the ASG's existence behind the scenes. We added these permissions to the IAM role:
# Auto Scaling read permissions
- autoscaling:DescribeAutoScalingGroups
- autoscaling:DescribeAutoScalingInstances
# EC2 read permissions
- ec2:DescribeInstances
- ec2:DescribeInstanceStatus
- ec2:DescribeTags
We updated the CloudFormation stack and propagated the permissions to cross-account roles.
...But still empty.
Investigation 3: Comparing with Old Jenkinsfile
We changed our approach here. "The old Jenkinsfile worked, but the new SDK doesn't." If so, we should look at the differences.
The old Jenkinsfile directly invoked AWS CLI. The new SDK uses boto3's CloudWatch client. Reading through the boto3 client initialization code, we found this:
client = boto3.client("cloudwatch", **client_kwargs)
# Event hooks added for compatibility with test framework (moto)
client.meta.events.register(
"before-parameter-build.cloudwatch.GetMetricStatistics",
self._normalize_cloudwatch_params
)
client.meta.events.register(
"before-call.cloudwatch.GetMetricStatistics",
self._normalize_cloudwatch_request_body
)
Surprisingly, this was the key.
These event hooks were added for compatibility with the test mock framework (moto). They intercept and rewrite parameters before boto3 issues API requests.
While they work correctly in test environments, in production they unexpectedly modified CloudWatch API request parameters, resulting in empty responses.
# Fix: Remove event hooks
client = boto3.client("cloudwatch", **client_kwargs)
# No hook registration. Let boto3 make standard API calls
This also explains why GetMetricWidgetImage worked. The event hooks were only registered for GetMetricStatistics, so GetMetricWidgetImage wasn't affected.
Lesson Learned: Test-specific code affecting production is easy to overlook. "Works in test environment" doesn't guarantee "works in production." Low-level interventions like boto3 event hooks require special attention.
Pitfall 4: Prompt Too Large for LLM
Metrics were now being fetched successfully, and datapoints were properly populated in JSON. But now the LLM analysis returned empty responses:
OpenAI API call failed (3 attempts):
Empty response returned from OpenAI API.
No error code. The API call succeeded, but the response content was empty.
We calculated the data size:
Weekly datapoints: 168 points → 26,590 characters
Daily datapoints: 288 points → 45,247 characters
Total: 71,837 characters ≒ approximately 18,000 tokens
We were sending all datapoints as raw JSON. Combined with the prompt template, this likely exceeded the model's input limit.
The old Jenkinsfile only sent statistical summaries (max, avg, points). The new SDK implementation was sending all datapoints—that was the issue.
# Before: Send all datapoints (~18,000 tokens)
"weekly_metrics": json.dumps(weekly_metrics, ensure_ascii=False, indent=2)
# After: Send only statistical summary (a few hundred tokens)
weekly_summary = {
"statistics": weekly_metrics.get("statistics", {}),
"period": weekly_metrics.get("period", {}),
"status": weekly_metrics.get("status", ""),
"warnings": weekly_metrics.get("warnings", []),
"datapoints_count": len(weekly_metrics.get("datapoints", []))
}
"weekly_metrics": json.dumps(weekly_summary, ensure_ascii=False, indent=2)
Lesson Learned: Design data passed to LLMs with "necessary granularity for analysis" in mind. Sending raw data wholesale isn't just token-inefficient—it risks hitting the model's input limit.
Pitfall 5 (Bonus): Why Not Analyze Images Too?
At this point, the monitoring pipeline was working. But then I realized something:
"We're generating graph images, but we're only sending statistical values to the LLM. Shouldn't we have it analyze the images too?"
The Vision functionality could capture trends and spike patterns that statistical values alone might miss.
# Build user message
user_content = [{"type": "text", "text": prompt}]
# Add graph images
for image_path in image_paths:
with open(image_path, "rb") as f:
image_data = base64.b64encode(f.read()).decode("utf-8")
user_content.append({
"type": "image_url",
"image_url": {"url": f"data:image/png;base64,{image_data}"}
})
messages = [
{"role": "system", "content": "You are a CloudWatch metrics analysis expert "
"with the ability to analyze both numerical data and visual graphs."},
{"role": "user", "content": user_content},
]
By sending weekly and daily graph images together with statistical summaries, we enabled analysis from both numerical data and visual pattern perspectives.
Retrospective: The Complete Flow
graph TD
A[SDK Migration Complete] --> B[Pitfall 1: max_tokens not supported]
B --> C[Pitfall 2: temperature not supported]
C --> D[Pitfall 3: Empty Metrics JSON]
D --> D1[Hypothesis 1: Insufficient IAM permissions]
D1 --> D2[Added permissions → No change]
D2 --> D3[Hypothesis 2: Diff with old Jenkinsfile]
D3 --> D4[Root cause: Test event hooks]
D4 --> E[Pitfall 4: Empty LLM Response]
E --> E1[Root cause: Prompt at 18,000 tokens]
E1 --> F[Pitfall 5: Add image analysis]
F --> G[Complete]
As we fixed one issue, the next became visible. By solving these 5 problems sequentially, we ended up with a better monitoring pipeline than we started with.
Summary of Decision Points
| Issue | Initial Hypothesis | Actual Cause | Lesson Learned |
|---|---|---|---|
max_tokens error |
Parameter name changed | ✅ Correct | Check API parameter compatibility on model updates |
temperature error |
Same as above | ✅ Correct | After fixing one, check others |
| Empty metrics JSON | Insufficient IAM permissions | ❌ Test event hooks | Test code impacting production is easy to miss |
| Empty LLM response | Model malfunction? | ❌ Prompt size exceeded | Design LLM inputs with necessary granularity |
The third issue—"Empty metrics JSON"—is where our initial hypothesis (insufficient IAM permissions) was wrong. It was reasonable to suspect IAM when it worked with admin permissions but not via AssumeRole, but the actual cause was at a lower layer.
Switching to diff comparison based on "it worked in the old version" was ultimately the right call.
Conclusion
Migration work seems simple at first glance—"just convert something that works into a different form." But in reality, these small compatibility issues accumulate.
The "test code breaking production" pattern was particularly striking in this case. Event hooks added for moto test compatibility modified production CloudWatch API calls. Writing tests is a good practice, but we need to be careful that test-specific code doesn't intrude into production code paths.
We spent several hours thinking "no choice but to push forward" as we tackled each issue, but we ultimately reached a better state by adding image analysis. Sometimes the detour is worth it.
Reference Books
For those who want to learn more about AWS operations and monitoring design systematically, the following books are highly recommended:
[📦 商品リンク: moshimo-book-aws-operations]
[📦 商品リンク: moshimo-book-monitoring-intro]
[📦 商品リンク: moshimo-book-sre-google]
Reading these alongside your actual operational experience will help you build a better toolkit for handling issues like the ones we encountered.