Migrating the Docker Agent to an ECR Image Worked — But Three Bugs and One Misdiagnosis Landed the Same Day

In the previous post, I wrote about using docker buildx for multi-architecture builds and why --provenance=false --sbom=false was required to make things work with Jenkins.

This is the follow-up. I sat down to check whether the new setup was actually healthy, and ended up finding three more problems. On top of that, my first diagnosis of one of them turned out to be flat-out wrong.

Where we left off

Last time I was chasing "an image built on arm64 doesn't run on the x86_64 node." docker buildx fixed the architecture half, and discovering that buildx's default attestation metadata doesn't play nicely with Jenkins led to the --provenance=false --sbom=false workaround.

Alongside that, I was also rolling out another improvement: switching the Jenkins Docker agent from a dockerfile directive to an ECR image reference. Instead of rebuilding the Dockerfile every time a job kicks off, we now pull a pre-built image directly.

The ECR migration paid off — the numbers are clear

I compared metrics from before and after the agent change.

"Packets RX −85%" is the headline number. Before the change, every job kickoff pulled down apt and npm dependencies from scratch. That entire chunk of work is gone, and the metrics show it.

The CPU credit balance nearly doubling also matters: on a burstable instance, the risk of exhausting credits is effectively eliminated now.

I tried to verify it was healthy — it wasn't

My celebration was short-lived. When I ran the full pipeline as a sanity check, it failed.

It wasn't a "but it was working two minutes ago" situation — unrelated problems had just happened to pile up at the same moment. Digging in, I found three independent bugs.

Bug 1 (first diagnosis): Is Claude Code CLI missing from the Docker image?

This one was the most surprising.

The Claude Agent (our agent that uses Claude Code as a backend) was dying with exit code 1 within two seconds across every phase. No error message at all.

"Did the OAuth token expire?" I reissued the token and tried again. Same behavior. It was perfectly reproducible, so I decided it had to be a code problem and started digging.

My first diagnosis went like this.

The SDK we use (@anthropic-ai/claude-agent-sdk) spawns the actual Claude Code CLI (cli.js) internally. But looking at package.json, the SDK itself was listed while the CLI package (@anthropic-ai/claude-code) was not. So maybe the SDK couldn't find the CLI and exited 1 right after spawning?

The proposed fix was to add this to the Dockerfile:

RUN npm install -g @anthropic-ai/claude-code@latest

Except — as you'll see — this was a misdiagnosis.

Bug 2: False-positive auth failures (the code misreads itself)

This one was more subtle.

The Codex Agent (our agent that uses OpenAI Codex) was actually working. The logs showed type=turn.completed, and the output files were being generated successfully.

And yet, right after that, the pipeline declared "Codex Agent authentication failed" and fell back to Claude.

When I dug into the code, the auth-failure detection logic looked like this:

// auth failure detection
const authFailed = messages.some((line) => {
  const normalized = line.toLowerCase();
  return (
    normalized.includes('invalid bearer token') ||
    normalized.includes('authentication_error') ||
    normalized.includes('please run /login')
  );
});

messages is every line the agent emitted. That was the problem.

The test_implementation phase is all about reading and writing test files. When the Codex Agent uses cat or grep on a test file, the contents of that file end up inside messages.

And our own source code contains the literal string authentication_error — in the auth detection logic itself, in its tests, and in troubleshooting docs.

The moment Codex read one of those files, authentication_error flowed into messages, and the pipeline marked a perfectly healthy run as an auth failure.

It's a feedback loop where the code reads its own keywords and misjudges itself. The reason it only reproduced in test_implementation is that only that phase touches the offending files.

Bug 3: The auth-validation job was failing to parse JSON

This one's the least exciting, but it was actively getting in the way of debugging the others.

To investigate "Claude Agent is broken," I tried running the credential-validation job. It was also failing — with Invalid JSON String.

net.sf.json.JSONException: Invalid JSON String

The readJSON step was failing to parse what the CLI had written to stdout.

The likely cause: the CLI is leaking log lines into stdout. With --output json, the expected contract is that JSON goes to stdout and logs go to stderr. But depending on the logger's config, [INFO] ... lines slip into stdout. Once a log line lands in front of the JSON, parsing breaks.

This bug meant I couldn't use credential validation to tell whether the Claude Agent problem was an auth problem or something else. One of my investigation tools was silently broken.

I stopped and looked at the diff

While sorting through the three bugs, something bothered me.

"The all-phases pipeline on main is the Dockerfile-based implementation, and over there Claude Code worked fine. What actually differs?"

I went back and looked at the branch diff. The Dockerfile changes were exactly two:

@anthropic-ai/claude-code was unchanged. package.json was unchanged.

So why did main work and this branch not?

Then I looked at the Jenkinsfile diff.

 agent {
-    dockerfile {
+    docker {
+        image 'your-registry/ai-workflow-agent:latest'
         label 'ec2-fleet-small'
-        dir '.'
-        filename 'Dockerfile'
-        args "-v ${WORKSPACE}:/workspace -w /workspace -e CLAUDE_DANGEROUSLY_SKIP_PERMISSIONS=1"
+        args "-u root:root -e CLAUDE_DANGEROUSLY_SKIP_PERMISSIONS=1"
     }
 }

-u root:root had been added.

Correcting the misdiagnosis: the real culprit was -u root:root

The Claude Code CLI is known to behave differently when run as root.

Inside the Dockerfile, the node user is granted sudo privileges. The intended design is "run as node, use sudo if you need it." Running directly as root with -u root:root was an unintended code path.

I dropped -u root:root and re-ran.

-args "-u root:root -e CLAUDE_DANGEROUSLY_SKIP_PERMISSIONS=1"
+args "-e CLAUDE_DANGEROUSLY_SKIP_PERMISSIONS=1"

Cut a new branch from develop, changed this one line, and ran the job.

The result was unambiguous.

[INFO ] [AGENT RESULT] status=success, turns=1, duration_ms=13040
[INFO ] Difficulty analysis complete: level=moderate, confidence=0.85

The spot that used to die instantly now completed in 13 seconds. Confirmed: -u root:root was the root cause of the Claude Code CLI startup failures.

The Bug 1 fix — "add @anthropic-ai/claude-code to the Dockerfile" — was unnecessary.

Watching the job run to completion

I kept watching the job without -u root:root and it finished with SUCCESS.

At the same time, Bug 2 (the auth false positive) also reproduced as predicted. The moment Codex read agent-executor.test.ts, it picked up the authentication_error literal and fell back right after turn.completed. But because the Claude Agent fallback was now healthy, the overall job still ran to completion.

Triaging the problems and what's left

Summary of the day:

✅ Confirmed
  ECR migration gave large CPU/network wins (packets −85%)

❌ → ✅ Resolved
  Claude Agent exit 1 across all phases
  → Cause: -u root:root added to Jenkinsfile
  → Fix: remove -u root:root (one-line change)

⚠️ Deferred to separate issues
  Bug 2: false-positive auth-failure detection
    → Source code keywords leak into `messages`
    → Real-world cost: we throw away Codex output and redo it with Claude (2x cost)
    → The workflow still completes, it's just inefficient

  Bug 3: auth-validation job JSON parse failure
    → Caused by log lines leaking into stdout (handled separately)

  Extra: setupNodeEnvironment() npm install + build is redundant after ECR migration
    → Every job still reinstalls dependencies, canceling part of the ECR perf win

And I cleaned up the issue tracker:

Reflection

"I ran the job to verify things and three more problems showed up" is a common pattern, but this time there was an extra twist: my first diagnosis was wrong.

When I started investigating Bug 1 with the "maybe the CLI isn't installed" theory, it felt like a reasonable story. But once I carefully walked the diff against main, the problem turned out to be -u root:root in the Jenkinsfile, not anything in the Dockerfile.

"exit 1 with no output" has a huge suspect list, which is what makes it hard. The lesson for me: before asking "is the environment itself sane?", ask "what's actually different from main?" first. I'd have reached the real cause faster.

Bug 2 (the false positive) was interesting structurally. The code reading its own keywords and misjudging itself is a clean example of how fragile naive keyword matching is. It reminded me that error-detection logic needs to decide, at design time, what it's even allowed to inspect.

Further reading

If you want to go deeper on Jenkins pipeline design, Docker agent behavior, or Claude Code usage, these books are good starting points.