Learning from Missing CLI Command: Incremental Debugging and Flexible Parser Design

Introduction

When running a monitoring script in a Jenkins pipeline, I got this error:

Error: No such command 'check'.
script returned exit code 2

Honestly, my first thought was "Wait, the check command should be there..." But upon investigation, it wasn't implemented at all. The journey from "implement command → another error occurs → fix parser" turned out to be surprisingly educational, so I'm documenting it here.

This may be somewhat scattered, but since it's based on actual work logs, I'll leave it as is.

The Problem

When trying to run our internal monitoring SDK from Jenkins, the following error occurred:

python3 -m monitoring.cli check --config configs/api-usage.yaml --output reports/result.json

Error: No such command 'check'.

The pipeline's Jenkinsfile was written assuming the check command existed, but this command didn't exist in the CLI module.

Investigation

First, I checked the available commands in the CLI module.

# Registered commands in cli.py
cli.add_command(batch_check)
cli.add_command(logs_check)
cli.add_command(cost_check)
cli.add_command(cloudwatch_check)
# ... many more
# But 'check' doesn't exist

The Jenkinsfile was calling the check command, but the implementation was missing. This was a pattern where documentation mentioned it, but implementation hadn't caught up.

Considering Solutions

The dilemma here was what to name the command.

Options:

Looking at existing commands, there was a pattern of xxx_check or xxx-check naming that explicitly described functionality. To maintain consistency, I decided to adopt api-usage-check.

CLI Command Implementation

I implemented the command using Click. Three key points:

1. Determine API Type from Config File

@click.command("api-usage-check")
@click.option(
    "--config", "-c",
    required=True,
    type=click.Path(exists=True, dir_okay=False, readable=True),
    help="Path to monitoring config file (YAML format)",
)
@click.option(
    "--output", "-o",
    required=True,
    type=click.Path(dir_okay=False),
    help="Path to output JSON file for results",
)
def api_usage_check(config: str, output: str) -> None:
    """Execute external API usage check."""
    loaded_config = load_config(config, required_keys=["api_type"])
    api_type = str(loaded_config.get("api_type", "")).lower()

    if api_type not in {"service_a", "service_b"}:
        raise click.ClickException(
            f"Unsupported api_type: {api_type}"
        )

2. Reuse Existing Monitor Class

Rather than writing new logic, I called the existing HTTPAPIMonitor. This avoids duplication of monitoring logic.

    monitor = HTTPAPIMonitor()
    result = monitor.check(loaded_config)

3. Standardize Output Format

While data available varies by API type, I standardized the output format.

    output_data: dict[str, Any] = {
        "status": result.status,
        "message": result.message,
        "timestamp": result.details.get("timestamp"),
    }

    # Add API-specific data
    if api_type == "service_a":
        api_data = result.details.get("service_a", {})
        output_data.update({
            "usage_total": api_data.get("usage_total"),
            "daily_usage": api_data.get("daily_usage"),
        })

Deployed Initial Fix, Then Another Error

After implementing the command and running it in Jenkins, a different error appeared:

ERROR - Error in check command: usage_history not included in response.

This is a trap.

The command itself now works, but now there's an error in the API response parsing. The parser was validating usage_history as a required field, but the actual API response didn't include this field.

Parser Investigation

Checking the existing parser revealed this validation logic:

def validate(self, response: Mapping[str, Any]) -> None:
    if "usage_history" not in response:
        raise MonitorError("usage_history not included in response.")

Looking at test code, it was written assuming the usage_history field exists. However, the actual API didn't return usage_history in some cases, instead returning fields like usageTotalForMonth and usageTotalPerDay at the top level.

Honestly, I got stuck here.

Possible causes:

I searched past Issues but couldn't find a clear answer. However, since it's not working in reality, I decided to make the parser more flexible.

Parser Fix Strategy

The dilemma was how flexible to make it.

Options:

I chose option 2. Reasons:

Parser Fix

Relaxed Validation Logic

def validate(self, response: Mapping[str, Any]) -> None:
    logger.debug("Starting response validation")
    if not isinstance(response, Mapping):
        raise MonitorError("Invalid API response format.")

    # Type-check only if usage_history exists
    usage_history = response.get("usage_history")
    if usage_history is not None and not isinstance(usage_history, list):
        raise MonitorError("usage_history must be array format.")

    # Warning log if missing
    if usage_history is None:
        logger.warning(
            "usage_history not included in response. "
            "Treating as empty array."
        )

Added Fallback Logic

def parse(self, response: Mapping[str, Any]) -> APIUsageData:
    usage_history_raw = response.get("usage_history", [])

    usage_total = 0
    daily_usage = 0

    # Get from usage_history
    for entry in usage_history_raw:
        if "usageTotal" in entry:
            usage_total = int(entry["usageTotal"])
        if "dailyUsage" in entry:
            daily_usage = int(entry["dailyUsage"])

    # If usage_history empty, check top level
    if not usage_history_raw:
        logger.debug("usage_history empty, checking top level")

        if "usageTotalForMonth" in response:
            try:
                usage_total = int(response["usageTotalForMonth"])
            except (TypeError, ValueError):
                logger.warning("Failed to convert usageTotalForMonth")

        if "usageTotalPerDay" in response:
            try:
                daily_usage = int(response["usageTotalPerDay"])
            except (TypeError, ValueError):
                logger.warning("Failed to convert usageTotalPerDay")

    return APIUsageData(
        usage_total=usage_total,
        daily_usage=daily_usage,
        usage_history=usage_history_raw,
    )

This is surprisingly crucial. Even when falling back, I catch conversion errors and output warning logs. This allows processing to continue with unexpected response formats while enabling later investigation if issues arise.

Test Fixes

The original test expected an error when usage_history was missing, but I modified it to match the new behavior.

def test_parse_handles_missing_usage_history() -> None:
    """Use top-level fields when usage_history is missing."""
    parser = APIResponseParser()

    result = parser.parse({
        "account_info": {},
        "usageTotalForMonth": 1000,
        "usageTotalPerDay": 50,
    })

    assert result.usage_total == 1000
    assert result.daily_usage == 50
    assert result.usage_history == []

Reflection

Let me organize what I learned from this work.

1. Trust Error Messages and First Understand Current State

When told "command doesn't exist," verify it really doesn't. This seems obvious, but assumptions like "it should be there" can lead to oversights.

2. Follow Existing Naming Conventions for Commands

Inconsistent command names confuse users. Choosing api-usage-check instead of check maintained consistency with other commands.

3. Parser Balance Between "Strict" and "Flexible" is Important

Rather than immediately erroring when required fields are missing, warn and continue processing if alternatives exist. However, log it for later investigation.

4. Watch for Gaps Between Actual API Responses and Tests

If tests don't reflect actual API responses, problems only surface in production. Ideally, create a mechanism to record actual responses and reflect them in tests.

Conclusion

What started as a simple "command doesn't exist" error led to parser design changes. It's common that fixing one error reveals another, but carefully pursuing "why does this error occur?" each time deepens understanding of the overall system.

Thinking "I just have to keep going," it turned out to be surprisingly educational work.