When \"It's the Reseller Account\" Was Only Half Right — Redesigning Cost-Discrepancy Detection Around Savings Plans

Introduction

In the previous post, I wrote about how the AWS cost monitoring dashboard showed values about 2x different from the AWS Console. After eliminating hypotheses one by one, I'd concluded the cause was a structural fact: "the target account was a member under an AWS reseller."

This article is the continuation. Since the correlation "discrepancy = under reseller payer" had looked clean during the investigation, I implemented an automatic warning feature on the dashboard using that detection axis. I added organizations:DescribeOrganization permissions to the cross-account role in CloudFormation, rolled it out to all 7 monitored accounts, wrote the code, wrote the tests, and submitted a PR.

"Tomorrow morning's report should now show a yellow-bordered warning," I thought, and ran the Jenkins job — and something unexpected happened.

Right after I implemented it, my own premise broke under counter-examples.

Honestly, this is a little embarrassing to write, but the experience of "I implemented it, ran it, and my premise itself was wrong" turned out to be the lesson I most wanted to leave behind. So I'm recording the process and the redesign that followed.

Recap of what I implemented

The hypothesis from the previous investigation was:

When the monitored account is under an AWS reseller's payer, the Console's "Cost and usage" current-month value diverges from the Cost Explorer API value.

Based on this, I added the following to the cost-monitoring job:

The detection logic looked roughly like this (simplified):

def _fetch_payer_info(self, config):
    target_account = config.get("prod_account")
    # ...AssumeRole, then call Organizations API...
    response = org_client.describe_organization()
    organization = response.get("Organization") or {}
    payer_account_id = str(organization.get("MasterAccountId") or "")
    is_reseller_account = (
        bool(payer_account_id)
        and payer_account_id != str(target_account)
    )
    return {
        "payer_account_id": payer_account_id,
        "organization_id": organization.get("Id"),
        "is_reseller_account": is_reseller_account,
        "reseller_name": KNOWN_RESELLER_PAYERS.get(payer_account_id),
    }

In the CloudFormation template, I added a dedicated policy document to the cross-account role:

- PolicyName: !Sub "${RoleNamePrefix}-organizations-policy-${Environment}"
  PolicyDocument:
    Version: '2012-10-17'
    Statement:
      - Effect: Allow
        Action:
          - organizations:DescribeOrganization
        Resource: '*'

I rolled this out to all 7 monitored accounts via Change-set, after confirming Replacement: False / RequiresRecreation: Never. 3 accounts + 4 accounts, total 7, all UPDATE_COMPLETE.

So far, smooth sailing. The plan was: verify with Jenkins, see the warning render correctly, merge the PR, close the loop.

The premise broke during verification

After the CloudFormation update finished and Jenkins ran, the first thing I noticed was a small piece of friction:

=== Account A (prod) ===
{
    "MasterAccountId": "<reseller's payer ID>",
    "Id": "o-xxxxxxxxxx",
    ...
}
=== Account B (prod) ===
{
    "MasterAccountId": "<reseller's payer ID>",  # ← same
    ...
}
=== Account C (prod) ===
{
    "MasterAccountId": "<reseller's payer ID>",  # ← !? same
    ...
}

Account C — which I'd previously labeled as "directly contracted" — was actually under the same AWS reseller as A and B.

The grounds I'd written in the previous article:

| Account | Console discrepancy | Master Account |

But this time, when I actually called describe-organization, Account C was under the same payer too. Last time I had taken another team's operational understanding ("Account C has a separate contract") at face value, and never verified it against the API.

That alone would just be "huh, the operational understanding was off" — but there's a more important implication:

Account C is under the reseller, but no Console discrepancy is observed.

In other words, the correlation "discrepancy = under reseller payer" from the previous article doesn't fully hold. Within reseller-managed accounts, some show discrepancy and some don't.

I paused and re-organized the detection axis:

The implementation would warn "this is under a reseller" for Account C as well — a False Positive. The warning fires even though there's no actual discrepancy with the Console, which would just confuse readers.

I had three options at this point:

The lowest-impact option was A, so I went with A for the moment. The wording was already softened to "may diverge (the actual cost is this value)," so I just corrected the operational doc's classification table to "discrepancy presence is mixed even within reseller-managed accounts."

If I had merged it as-is here, someone (maybe future me) would probably have been tripped up by it eventually.

The same counter-example appeared in dev too — that's when I realized this was a design problem

After patching the operational doc with option A, I checked describe-organization behavior on the dev-environment accounts (Accounts D / E).

They were all under the same payer. And yet, in operation, Accounts D / E don't show any discrepancy. Adding to Account C, two more counter-examples piled up.

The premise had broken in 3 out of 5 accounts. That's no longer "soften the wording and accept the compromise." Most of the warnings would be False Positives.

The decisive moment came from a heuristic on the operations side:

"Accounts D / E don't use Savings Plans. Maybe that's why there's no discrepancy?"

When I heard it, it landed. The accounts with discrepancy — A and B — are environments that have purchased Savings Plans before. Accounts C / D / E are smaller-scale environments where SP purchases never made sense.

"Wait — maybe the detection axis isn't 'is it under a reseller?' but 'is Savings Plans applied?'"

In the previous article's verification, I had observed UnblendedCost == AmortizedCost for Account A, and from that I'd dismissed any "SP/RI effect." But on reflection, that was only checking consistency as seen from the member account side, and it doesn't necessarily verify the case where the SP is owned on the payer side and assigned to the member.

That's when I finally settled in: "this is a moment to raise the quality of the hypothesis."

Verifying with the API — confirming the new hypothesis

Changing the design on heuristics alone is risky, so I went to observe directly via the CE API:

aws ce get-cost-and-usage \
  --time-period Start=2026-05-01,End=2026-05-08 \
  --granularity MONTHLY \
  --metrics UnblendedCost \
  --filter '{
    "Dimensions": {
      "Key": "RECORD_TYPE",
      "Values": [
        "SavingsPlanCoveredUsage",
        "SavingsPlanNegation",
        "SavingsPlanRecurringFee"
      ]
    }
  }'

The 3 record types SavingsPlanCoveredUsage / SavingsPlanNegation / SavingsPlanRecurringFee only carry non-zero values on member accounts where SP is applied. Without SP, they're all zero — or at most rounding-error level (e.g. $1e-12).

Running the same query against all 7 accounts:

"SP applied = discrepancy exists" matched perfectly.

That was the moment when "ah, the previous investigation was only half right" finally landed. The previous conclusion "reseller is the cause" was a correlation that only held because the accounts I'd happened to observe (A and B) both had SP applied — the actual causation was elsewhere.

Being under a reseller was neither a necessary nor a sufficient condition for Console discrepancy. SP applicability is a far higher-precision signal.

Redesigning the detection axis

From here, the design rebuild. There was a lot to do, but what to do was clear.

Switch the detection axis from is_reseller_account to savings_plan_info.active. Change the wording from "this account is under a reseller payer" to "Savings Plans are applied, so the value may diverge." Rename the CSS class from reseller-notice to cost-discrepancy-notice. Rewrite all the tests.

One thing I hesitated on was the payer_info field. It could be deleted — it's no longer part of the detection — but reseller-status itself has operational value (when reconciling with the accounting side's management units, who is the payer is something you eventually want to know). So I decided to remove it from detection but keep it as operational information.

# New detection: SP-applicability based
def _fetch_savings_plans_usage(self, config, context):
    """Fetch SP usage from Cost Explorer. Returns None on failure."""
    # AssumeRole credential selection (also handles prod-only / dev-only configs)
    target_account = config.get("prod_account") or config.get("dev_account")
    if not target_account:
        return None
    
    # ...client construction omitted...
    
    response = ce_client.get_cost_and_usage(
        TimePeriod={"Start": period_start, "End": period_end},
        Granularity="MONTHLY",
        Metrics=["UnblendedCost"],
        Filter={
            "Dimensions": {
                "Key": "RECORD_TYPE",
                "Values": SAVINGS_PLAN_RECORD_TYPES,
            }
        },
    )
    
    total = sum(
        float(item["Total"]["UnblendedCost"]["Amount"])
        for item in response.get("ResultsByTime", [])
    )
    
    return {
        "active": abs(total) > SAVINGS_PLAN_ACTIVE_THRESHOLD_USD,
        "monthly_amount": round(total, 6),
        "currency": "USD",
        "period": {"start": period_start, "end": period_end},
    }

The threshold of 0.01 USD is insurance against rounding errors like -$1e-12 showing up as false. When future accounts are added, I didn't want active=true to flicker due to rounding noise.

A small but important change went into _fetch_payer_info: I made it work for dev-only YAML configs. The original required prod_account, so a YAML without prod_account would early-return and payer_info wouldn't appear.

target_account = config.get("prod_account") or config.get("dev_account")
if not target_account:
    return None

# AssumeRole credentials switch on the corresponding prod/dev keys
if config.get("prod_account"):
    role_arn = config.get("role_arn")
    external_id = config.get("external_id")
else:
    role_arn = config.get("dev_role_arn")
    external_id = config.get("dev_external_id")

I'd already fixed this once, before the design pivot. It turned out the same structure was needed in the SP-detection feature too. Even though the design premise changed, the AssumeRole credential switch worked as a shared foundation — it ended up being a clean reuse.

Verification — the dashboard renders as expected

Running the redesigned code through Jenkins gave the expected result:

A perfect match with the observed discrepancy presence. The 3 False Positives that the reseller axis was producing are gone. result.json outputs both savings_plan_info and payer_info, and payer_info remains as operational information.

{
  "service": "AWS",
  "currency": "USD",
  "period": {"start": "2026-05-01", "end": "2026-05-07"},
  "current": {"cost": 1275.51},
  "savings_plan_info": {
    "active": true,
    "monthly_amount": 177.41,
    "currency": "USD",
    "period": {"start": "2026-05-01", "end": "2026-05-08"}
  },
  "payer_info": {
    "payer_account_id": "<masked>",
    "organization_id": "o-xxxxxxxxxx",
    "is_reseller_account": true,
    "reseller_name": "<masked>"
  },
  "status": "OK"
}

Now we were finally in "OK to merge" state. From the original implementation, the redesign added 4 commits, touched 7 files, and was about +550 lines of code.

Retracting your own premise inside a PR

In the end, I rewrote the PR body too. The original title and body said "show a warning on Reseller-managed accounts." The redesign changed it to "show a discrepancy warning on Savings-Plans-applied accounts."

I added this paragraph at the top of the PR body:

Originally, this was designed to detect Reseller-payer-managed accounts. After verification, the actual detection axis turned out to be whether Savings Plans are applied, and a redesign was performed (details below).

Writing this is not easy. Saying "the original design was wrong" in public is uncomfortable for anyone. But unless the PR body explicitly states "the premise was retracted mid-flight," anyone referencing this PR later might think it's "logic that detects reseller-managed accounts." That would set up future me, and others, to fall into the same trap.

I also showed in the body the verification table that explains why the detection is keyed on "SP applied / not" rather than directly on "discrepancy yes / no." Future readers asking "why SP detection? Couldn't this be more direct?" can trace back the path: "we tried reseller detection, hit operational damage, switched."

Lessons — pitfalls of hypothesis-driven thinking, and verifying before redesigning

A few lessons from this round:

Don't bake correlation into the design before checking its provenance

When the previous investigation found "reseller-managed = discrepancy," I didn't take seriously enough that it was only "3-of-3 in the sample." Hearing "3 out of 3" feels like "100%," but a sample size of 3 is too weak to back the precision of a correlation. The moment the sample grew to 5, then 7, the correlation broke.

"Don't bake a correlation into the implementation before you can vouch for its provenance" — that's the lesson that landed deepest this time. When the premise breaks after you've shipped, the cost of fixing the code is bigger than expected (it ripples through tests, docs, CFN, reviewer explanations, the PR title — all of it).

Don't take heuristics on faith — translate them into something API-reproducible

The decisive nudge for the redesign was the operational heuristic "Accounts D / E don't use SP, that's probably why they don't diverge." Without it I'd probably have stopped at wording adjustments.

But just porting the heuristic into the implementation directly is risky too, so I translated it into a CE API RECORD_TYPE-filter-based automated check before embedding it. Heuristic → observable via API → reproducible in code — being able to bridge those was the smoothest part this round. Heuristics are maps, but you don't paste the map onto the implementation.

Leave "premise retraction" in commit logs and PR body

I deliberately committed docs(monitoring): reflect that discrepancy is mixed even under reseller-managed accounts mid-flight. I'd considered making it a smaller "lint fix"-style message, but the fact that the premise broke is a marker that future readers will want to spot, so I made it visible.

I followed it with feat(monitoring): switch the discrepancy-warning detection axis to Savings-Plans-detection-based. Reading just the commit log, the story "initial design → counter-example breaks the premise → redesign" is traceable.

When future me hits the same problem, I want to be able to figure out "why this decision?" from just the commit log and PR body. Same motivation as troubleshooting.md from the previous investigation: leaving a "thinking map" rather than a "warehouse of conclusions" helped again here.

Closing

The previous conclusion "reseller is the cause" was, as it turns out, half right and half wrong. It is true that all the accounts with discrepancies were reseller-managed. But the inverse — that all reseller-managed accounts had discrepancies — was not.

Honestly, when I wrote the previous article, I thought "this case is settled." Hypothesis built, verified, documented, turned into knowledge. "Next time I hit this, it's a 30-minute solve."

But the implementation built on top of that "settlement" hit a counter-example the moment it ran, and the premise collapsed. Technically this is a "common pattern," but for me it was a memorable experience. The moment you feel a case is settled, the window for re-verification is already closing — that feeling stayed with me.

The good thing about hypothesis-driven thinking is the propulsion it gives — you can move ideas forward. But if you ride that propulsion all the way into implementation, you can't stop when the premise breaks. A hypothesis built on a still-small sample should be treated as "the entrance to the next round of verification," not as "the right conclusion." Hold it at that distance, and you can probably run longer.

Opportunities to write "that previous conclusion was only half right" while quoting your own previous article are best kept few — but when one does come up, I think the right move isn't "this is embarrassing, let's hide it." Recording the path itself probably helps someone, eventually.

Going forward, when adding a new monitoring target, the checklist will include both describe-organization to confirm the payer and a CE API SP-records check. Yet another item added to the checklist by friction.

Related Books

Here are recommended books for those who want to learn more about AWS operations, retracting design decisions, and PR/commit-log documentation practices like in this case.