How Our Monitoring Tech Selection Pivoted Three Times

Starting Point: Monitoring That Only Works If You Remember to Check

Incident detection had been entirely dependent on user reports. Basic uptime monitoring — whether HTTP responses were coming back, whether error rates were spiking — wasn't even in place for several of our services.

The decision to "add monitoring" itself wasn't hard. The question was how to implement it.

I set one constraint before starting the research: we eventually want to see internal metrics (RDS, ECS, SQS, etc.) on the same dashboard. "Start with uptime monitoring" is the right approach, but the choice we make in Step 1 might constrain Steps 2 and beyond. I kept that in mind throughout the evaluation.

Round 1: SaaS Pricing Wasn't What It Seemed

I started by listing uptime monitoring SaaS candidates: UptimeRobot, Hyperping, Instatus, and Better Stack. The advertised prices all looked reasonable. But when I dug into the pricing pages with the condition "a team of 10 people," the picture changed.

The Gap Between Listed Prices and Reality

Hyperping is $74/month for 1-5 users, but jumps to $199/month the moment you hit 6 users. For a 10-person team, you're immediately on the top-tier plan.

Better Stack is trickier — the $29/month starting price is just the baseline. Each "Responder" seat that can receive on-call notifications incurs additional charges. A configuration where all 10 members can receive alerts comes out to $290-$340/month. You might think "read-only access is cheaper," but then some people can't receive alerts.

Instatus offers unlimited members at $20/month on the Pro plan, which seems like a steal. However, it's more of a status page product, and its monitoring capabilities are less robust compared to the others. If SSO becomes a requirement, the price jumps to $300/month.

UptimeRobot has additional seats at $1.5/month, making it the most realistic option at $40-45 for 10 users.

Here's the comparison for 10 users, 50 monitors, and 1 status page:

Pricing pages often say "simple pricing," but when you calculate based on actual team usage, it's anything but simple.

Round 2: "Extend Our Existing Jenkins" as an Option

With SaaS pricing established, I estimated the cost of the other axis: extending Jenkins.

We already have a Jenkins master running. It's used for a separate workflow — fetching daily cost and API usage data, generating custom dashboards, and sending notifications to Teams. We could reuse that same Jenkins instance.

The agent runs on spot instances (t3.medium, Tokyo region) at $0.018/hour. Running 24 monitoring jobs per day with 5-10 minutes per execution puts the monthly cost at roughly $1-2. Essentially zero for running costs.

Comparing TCO Including Development Costs

Comparing running costs alone, "Jenkins wins by a landslide." But there's a catch. SaaS is easier to set up, requiring less development effort than a Jenkins extension. Dividing the $40/month difference by the development cost difference, it would take 5-8 years to recoup.

However, that's only looking at Phase 1.

When you factor in Phase 2 — adding internal metrics monitoring — the equation changes.

If you choose an uptime monitoring SaaS, you hit a wall in Phase 2: "internal metrics can't be added." You end up building a Jenkins dashboard infrastructure anyway, and the development effort you saved in Phase 1 comes back as technical debt.

Looking at the total across all phases:

Development effort is roughly equivalent, but running costs differ by an order of magnitude. These numbers led to the decision: "Let's go with Jenkins extension."

Writing the Project Plan: The First Version Was Too Heavy

With the direction set, I started writing a project plan. The first version was over-engineered. Threshold definition documents, notification design documents, page design documents, job design documents, dashboard design documents... it stacked deliverable documents for each phase.

This didn't match the reality of one skilled engineer working solo. Technical development work would likely finish faster than even the optimistic estimates. What actually takes time is cross-team interviews, consensus on thresholds, and the context-switching cost of parallel workstreams.

I pivoted to "build something working first, improve through operation" and rewrote the plan. Deliverables were narrowed to three: "the code itself," "endpoint/threshold list (spreadsheet)," and "operations notes." The effort estimates compressed significantly.

Estimates Should Be Ranges

Single-point estimates tend to be imprecise. So I switched to optimistic-pessimistic ranges.

Range widths vary by task type: narrow for configuration/meetings, medium for design/definitions, wide for development/implementation. Each task also notes "why it might vary" (e.g., "heavily depends on current state of each service," "threshold consensus tends to take time").

The result: Phase 0-3 totals came out to 37-66 person-days optimistic-pessimistic, with a schedule of 3-4.5 months.

Round 3: Holes Revealed by PM Review

With the plan written, I ran a self-review from a senior PM perspective. Several concerns surfaced.

Critical Gap 1: No Success Metrics

The qualitative goal "detect incidents faster" was stated, but there was no number like "detect within 5 minutes." This makes it impossible to judge success or failure at project completion. Ironic — the project aims to "move away from gut-feel operations," yet the success criteria is gut-feel.

Critical Gap 2: 1-Hour Monitoring Interval Might Not Beat Humans

The plan stated "every hour," but hourly monitoring means up to 59 minutes of blindness in the worst case. If users report issues within 30 minutes on average, this monitoring might not improve on the status quo.

There was no rationale for "why 1 hour," and no consideration of different intervals based on service criticality.

Critical Gap 3: Weak Jenkins SPOF Mitigation

The plan mentioned "cron / separate health check route for monitoring," but "what separate route?" remained unanswered. Who notices and recovers when Jenkins goes down? How do we handle monitoring data gaps during extended Jenkins outages?

The structural weakness of placing a "watchdog" on the same infrastructure it's supposed to watch wasn't adequately addressed.

These two points — monitoring interval and SPOF — were issues that shook the tech selection decision itself.

Round 4: CloudWatch Synthetics Enters the Picture

Facing "we need shorter intervals" and "we need to solve the Jenkins SPOF," I took a fresh look at Amazon CloudWatch Synthetics.

What Is CloudWatch Synthetics?

It's AWS's native uptime monitoring service. It periodically runs Canaries (Node.js/Python scripts) to monitor endpoint availability and response times. Results are recorded as CloudWatch metrics, and notifications flow through CloudWatch Alarms → SNS → Teams Webhook.

The key features: minimum 1-minute intervals, and as an AWS managed service, no SPOF like Jenkins.

Cost Estimation

Each Canary run costs $0.0012. For our actual configuration:

Comparable to or cheaper than UptimeRobot's $40-45, depending on interval design.

Why It Could Be the Best of Both Worlds

It potentially resolves the weaknesses of both uptime SaaS and Jenkins extension simultaneously.

The bigger win is Phase 2 simplicity. RDS and ECS metrics already exist in CloudWatch. After starting uptime monitoring with Synthetics, moving to Phase 2 might only require adding CloudWatch Alarms configurations. Total development effort looks lightest compared to building custom scripts in Jenkins.

Two Approaches Under Consideration

With Synthetics, there are two main paths forward.

Approach D-1: Synthetics as the Core

Consolidate both uptime and internal monitoring in CloudWatch. Dashboard via CloudWatch Dashboard, status page as a separate S3 static site. The biggest advantage: Phase 2 becomes mostly configuration work.

Approach D-2: Synthetics + Jenkins Hybrid

Uptime monitoring goes to Synthetics, while dashboard generation and notifications leverage existing Jenkins. Jenkins already has HTML auto-generation capabilities, so those assets aren't wasted. The trade-off: accepting dual-system management of "monitoring in Synthetics, dashboards in Jenkins."

Open Questions Remain

CloudWatch Dashboard works well as an engineering operations screen, but whether it can serve as a "status page that developers and users can view together" — the original goal — is uncertain without actually building it. Canaries are also Lambda-based scripts, so the development and deployment workflow differs from Jenkins shell scripts. Depending on the team's AWS experience, learning costs may apply.

What the Gantt Chart Process Revealed

When adding a Gantt chart to the plan and organizing dependencies, something interesting emerged.

Total effort is 37-66 person-days optimistic-pessimistic, but the critical path is 43 days. That means 23 days of parallelizable work exist. Conversely, every day of delay on the critical path delays the whole project by one day.

Bottlenecks by phase:

Phase 2's "RDS freeze detection" is more of a research task than a development task since the approach isn't established. It needs a timebox — "investigate for N days and judge feasibility" — and descoping should remain an option.

Also, structuring the plan with a "zoom-in" approach — summary Gantt showing "4 phases, 9 weeks" at a glance, then a flowchart of all task dependencies, then the detailed Gantt — made it much more readable. Presenting 23 tasks in a Gantt chart immediately overwhelms readers with details before they grasp the big picture.

The Lesson: Don't Over-Detail Before Deciding

When choices expanded to A through D (uptime SaaS, integrated SaaS, Jenkins extension, CloudWatch Synthetics), I restructured the plan itself.

The original plan was built around Jenkins extension, with detailed task breakdowns per phase. But when the decision on which option to choose hasn't been made, building detailed designs for a specific option is wasted effort if it gets overturned.

I rewrote the plan to focus on "rough cost comparison across options and Phase 0 validation approach." Detailed phase plans would be created after the option is chosen. The detailed plan for Option C remained as an appendix but was no longer the decision-making centerpiece.

Estimation Accuracy and When to Refine

Current rough estimates are ±40-50% accurate. Running Phase 0 with one actual service will reveal:

Accuracy should improve to ±15-20% after Phase 0 completion. Detailed planning is designed for that timing.

The Tone of Decision Documents

Through multiple rewrites of the plan, another insight emerged.

The first version used definitive language: "The recommended option is Jenkins extension" and "SaaS is excluded at this point." Logically correct. But it makes it hard for readers to feel they "thought it through themselves."

What I changed was the tone. "Here's what the research found — how should we decide?" Present facts and data, create a process-of-elimination flow. This lets readers arrive at "D is the right choice" on their own.

Decisions that stick tend to be ones people reach themselves, rather than ones pushed upon them.

Answering "Which One Do You Recommend?"

Even if the document is written neutrally, you'll be asked for your opinion verbally. I prepared my stance in advance.

"I believe Option D (CloudWatch Synthetics) is the most rational choice. Three reasons: no SPOF since it's AWS managed, lightest total development effort because Phase 2 can leverage existing CloudWatch metrics, and low TCO. However, there's one thing to verify: whether CloudWatch Dashboard's expressiveness meets our requirements. We won't know until we actually build it. So I recommend prototyping with Option D in Phase 0 — if the dashboard is acceptable, proceed with D; if not, fall back to Option C (Jenkins)."

Three points: narrow the recommendation to one, state the weakness yourself, and preemptively present the fallback. "Each option A-D has pros and cons" adds no value when someone asks for your opinion.

Where We Are Now

We're still before Phase 0 (approach confirmation and PoC). This article itself is a "record of the selection process," not an implementation log.

After actually working with CloudWatch Synthetics, I plan to write another update.

Recommended Reading

For those who want to dive deeper into monitoring infrastructure design and tech selection, these books are helpful references.

Systematic Knowledge of Monitoring Design

SRE Principles and Practice

AWS Operations Fundamentals