LLM Monitoring Pipeline Token Optimization: When Reducing Input Doesn

In a previous article, I wrote about five pitfalls encountered during CloudWatch monitoring SDK migration. At the end of that article, I mentioned "it might be better to analyze images together," and when we actually expanded to GCP API monitoring, we implemented exactly that Vision API integration in earnest.

And that's where I ran into another problem. This time, it was "tokens."

Input too large—failed. Reduced it, then output became insufficient. Honestly, when you integrate LLM into a monitoring pipeline, you end up battling these "hard-to-see constraints." This article documents two problems encountered during prompt optimization for GCP API monitoring and the troubleshooting process.

Background: What We Were Building

We built a monitoring pipeline that periodically retrieves traffic metrics from GCP APIs (Search Console, Analytics Admin, etc.) and automatically generates analysis reports using LLM (Azure OpenAI).

Built on the same monitoring SDK as the CloudWatch edition, the basic flow looks like this:

GCP API Metrics Retrieval
  → Statistical Analysis (anomaly detection, data quality checks)
  → Graph Image Generation (PNG)
  → LLM Analysis (Vision API with image + statistical summary)
  → Markdown Report Output

Learning from the CloudWatch edition, we designed it to integrate Vision API from the start. The goal was to have the LLM read trend patterns and spikes from graph images that aren't visible from statistics alone.

Problem 1: Prompt Too Large – Duplicate Transmission of `hourly_stats`

When we ran the pipeline, the LLM returned empty responses for some services.

Empty response returned from OpenAI API.
finish_reason='length'

finish_reason='length' is a sign that we hit the token limit. First, I investigated the prompt size.

Saving the Prompt for Debugging

The first thing I did was add a mechanism to write the prompt being sent to the LLM to a file.

# For debugging: save the sent prompt to file
prompt_path = os.path.join(report_dir, f"{service_name}_prompt_sent.md")
with open(prompt_path, "w", encoding="utf-8") as f:
    f.write(prompt_text)

This is a small thing, but without it, you can't accurately understand what you're sending to the LLM. Debugging based on assumptions like "probably about this size" leads to detours.

What I Noticed When Examining the Saved Prompt

When I opened the written prompt file, it exceeded 400 lines. Looking through the contents, I found that hourly_stats (hourly statistical data for 24 hours) was duplicated in 2 places.

1. hourly_stats is entirely included in raw_data (JSON)
2. The same data is formatted and included in analysis_text (human-readable text)

In other words, we were sending the same information twice in different formats. Furthermore, the same time-series data was visualized and sent in the graph image (Vision API), so we were effectively sending it three times.

graph TD
    A[hourly_stats] --> B[raw_data JSON]
    A --> C[analysis_text Text]
    A --> D[Graph Image PNG]
    B --> E[LLM Prompt]
    C --> E
    D --> E
    E --> F[Sending same info 3 times]

Fix: Send Summary Only

I excluded hourly_stats from both raw_data and analysis_text.

# raw_data: Summary statistics only (hourly_stats excluded)
summary_data = {
    "service": service_name,
    "metric": metric,
    "data_quality": asdict(analysis.data_quality),
    "overall_stats": analysis.overall_stats,
    "anomalies_count": len(analysis.anomalies)
}

# analysis_text: Generate without hourly_stats
def build_analysis_results_text(analysis):
    payload = {
        "data_quality": asdict(analysis.data_quality),
        "anomalies": [asdict(a) for a in analysis.anomalies],
        "overall_stats": analysis.overall_stats,
    }
    return json.dumps(payload, ensure_ascii=False, indent=2)

This reduced about 240 lines (24 hours of hourly_stats). Vision API can read detailed time-series patterns from the graph image. It's sufficient to send only overall statistics and anomaly detection results via text.

Decision Point: Rather than "what to cut," work backward from "what we want the LLM to decide." Sending all data doesn't necessarily improve accuracy; it can become noise. The combination of statistical summary + graph image is a well-balanced design that covers both numerical accuracy and visual patterns.

Problem 2: Input Reduction Didn't Help – The Output Token Trap

After significantly reducing the prompt and re-running, Search Console metric analysis succeeded.

searchconsole → Response received successfully: length=1843

But Analytics Admin still failed with the same error.

analyticsadmin → finish_reason='length', content=''

For a moment I thought, "Is the prompt still too long?" But thinking calmly, Search Console succeeded. Why does only one fail when using the same prompt template?

Diagnosis: Input or Output?

finish_reason='length' means "output token limit reached." This is a different problem from input being too large.

This is a trap. When you hear "token problem," you instinctively think "let's reduce input." But LLM token constraints have two dimensions.

Input tokens: Prompt (system prompt + user message + images)
Output tokens: Length of LLM-generated response (limited by max_completion_tokens)

Checking our configuration, max_completion_tokens: 2500. While Search Console's response fit in 1843 tokens, Analytics Admin exceeded 2500 and was cut off mid-generation.

Why Only Analytics Admin Had Longer Responses

I compared the data characteristics.

Analytics Admin metrics had more than 10 missing_hours, indicating significant data gaps. When the LLM detects data quality issues, it tries to generate more detailed analysis comments (speculation on gap causes, impact scope, recommended actions). As a result, the response became too long to fit in 2500 tokens.

In other words, services with worse data quality cause the LLM to use more tokens for explanation. This makes sense when you think about it, but it fails when you have a fixed upper limit in the configuration.

Fix: Increase max_completion_tokens

openai:
  deployment_name: "gpt-5-mini"
  api_version: "2024-02-15-preview"
  max_completion_tokens: 4000  # 2500 → 4000
  max_retries: 3

Increasing from 2500 to 4000 solved the problem. Analysis reports were now successfully generated for all services.

Decision Point: The value of 4000 was determined as "current max response length + margin." Setting it too large increases cost and latency. On the other hand, setting it at the bare minimum causes the same problem again when services are added in the future. For now, we increased it by less than double and will monitor actual usage to adjust.

Overall Flow

graph TD
    A[GCP API Monitoring Implementation] --> B["finish_reason='length' Error"]
    B --> C[Hypothesis: Input prompt too large]
    C --> D[Investigation: Write prompt to file]
    D --> E[Discovery: hourly_stats sent 3 times]
    E --> F[Fix: Send summary only]
    F --> G[Search Console → Success]
    F --> H["Analytics Admin → Still failing"]
    H --> I[Reconsider: Output problem, not input?]
    I --> J[Cause: max_completion_tokens = 2500 insufficient]
    J --> K["Fix: Increase to 4000"]
    K --> L[All services working normally]

Summary of Decision Points

Problem	Initial Hypothesis	Actual Cause	Learning
Empty response (1st)	Prompt too long	✅ Triple transmission of `hourly_stats`	Work backward from "what we want LLM to decide"
Empty response (2nd)	Prompt still too long?	❌ Output token limit	`finish_reason='length'` is an output-side problem

When I encountered the "input reduction didn't help" situation in the second problem, not sticking to the same direction (input reduction) worked out well. From the fact that Search Console succeeded, I could pivot: "input size is the same but results differ → cause must be elsewhere."

Token Design When Integrating LLM into Monitoring Pipelines

Based on this experience, here are key points to consider for token design when integrating LLM into pipelines.

Input Design:

Don't send the same information in multiple formats (tends to become triple with JSON + text + image)
Work backward from "what we want the LLM to decide" to select minimal necessary data
If using Vision API, let graph images handle time-series data and send only statistical summaries via text

Output Design:

Set max_completion_tokens considering data characteristics of each service
Worse data quality causes LLM to generate longer explanations
Set with margin while monitoring actual usage

Debugging Infrastructure:

Always include a mechanism to write the prompt being sent to LLM to a file
Always log finish_reason ('stop' = normal completion, 'length' = token limit)

Conclusion

Following the "whack-a-mole debugging" in the CloudWatch edition, this time it was the "token swamp." As usual, solving one problem reveals the next.

However, I think it was progress that I could stay conscious of "am I looking at an input problem or output problem right now?" during debugging. Systems using LLM require debugging skills on a different axis from traditional software.

When reducing input doesn't help, suspect the output. It seems obvious, but it's surprisingly easy to overlook when you're in the middle of it. I hope this article helps someone who falls into the same swamp.

Reference Books

Here are some reference books for LLM application development and monitoring design.

[📦 商品リンク: moshimo-card-tpujl]

For monitoring pipeline design, the following O'Reilly book is a great reference.

[📦 商品リンク: moshimo-book-monitoring-intro]