When a CI-Built Docker Image Wouldn't Run on Another Server: The Trap Was in buildx Defaults

What happened

A CI job was failing with this error.

does not seem to be running inside a container
$ docker run -t -d ... your-registry/your-image:latest cat
ERROR: The container started but didn't run the expected command.

"The container started, but it didn't run the expected command."

My first guesses were a misconfigured ENTRYPOINT or a user-permissions issue. The actual cause was somewhere completely different.

The cause: CPU architecture mismatch

Docker images depend on CPU architecture. Broadly speaking, there's "arm64" (Apple Silicon, AWS Graviton, etc.) and "x86_64" (typical Intel/AMD servers), and an image built for one will not run on the other.

It's roughly the same intuition as Windows software not running on a Mac.

The setup looked like this.

So "verification passes but production breaks." The error log doesn't directly say "exec failed because of architecture," so at first I had no idea what was happening.

Solution direction: support both architectures with a single image

I considered a few options.

Option 1 is fast, but leaves a constraint: "this image only works on arm64 servers." Option 2 makes tag management messy.

I went with option 3. With docker buildx, you can pack both "for arm64" and "for x86_64" into a single image tag. The server pulls whichever variant matches its own architecture, automatically. The consumer-side Jenkinsfile needs no changes.

How buildx + QEMU works

There's a question lurking here: "how do you build an x86_64 image on an arm64 server?" That's where QEMU comes in.

QEMU is a CPU emulator that translates instructions from one architecture and executes them on another. When an x86_64 build runs on an arm64 server, QEMU translates instructions one by one in the background.

# This makes the arm64 node able to execute x86_64 binaries
docker run --privileged --rm tonistiigi/binfmt:qemu-v8.1.5 --install all

The downside is builds get slower. Three to ten times slower than native execution. In this case, a build that took about 7 minutes became 22 minutes. I chose "slow but it works."

The build command itself looks like this.

docker buildx build \
    --platform linux/amd64,linux/arm64 \
    --push \
    -t your-registry/your-image:latest \
    .

--platform specifies both, and --push sends straight to the registry. ECR ends up with both an "arm64 variant" and an "x86_64 variant" under the same tag.

"Fixed!" — except it still didn't work

This was the biggest trap of the whole episode.

The push to ECR succeeded. Inspecting the manifest showed both arm64 and x86_64 entries. But the production job kept failing with the exact same error.

Investigating, the manifest in ECR wasn't quite what I expected.

{
  "manifests": [
    { "platform": { "architecture": "amd64", "os": "linux" } },
    { "platform": { "architecture": "arm64", "os": "linux" } },
    { "platform": { "architecture": "unknown", "os": "unknown" } },
    { "platform": { "architecture": "unknown", "os": "unknown" } }
  ]
}

The architecture: unknown entries are metadata that buildx attaches by default (SLSA provenance attestation). It's information that proves "where and how this image was built" — security-wise it's a useful thing.

But it doesn't get along with the Jenkins Docker Plugin. Jenkins sees the unknown/unknown entries, gets confused, and can no longer handle the image correctly. This is a known compatibility issue between buildx and Jenkins.

The fix was simple.

docker buildx build \
    --platform linux/amd64,linux/arm64 \
    --provenance=false \
    --sbom=false \
    --push \
    -t your-registry/your-image:latest \
    .

Just adding those two flags. The manifest becomes a clean two-entry "amd64 + arm64" structure, and Jenkins handles it without issue.

Other decisions logged

Bumped up the node size

QEMU wasn't the only reason builds got slower. The lightweight node we were using (roughly t-family micro) had only 1 GB of memory.

npm install, the TypeScript compile, and QEMU itself all eat memory. Once you start swapping under memory pressure, things get dramatically slower than the perceived overhead suggests.

Comparing node classes: vCPU count was the same across them, the only difference was memory (1 GB / 2 GB / 4 GB). Switching to the medium class (4 GB) brought the build down to 22 minutes. That number includes the QEMU overhead, so I think it's reasonable.

The ECR build only runs once a day, so bumping the node size has negligible cost impact.

Switched the cron to weekly

Originally it ran every night automatically. The purpose was to automatically pick up base image security patches.

But running a build every day, including days with no source changes, is pure waste. We can run it manually when needed, so I changed it to weekly (Sunday night). There's at most a one-week delay, but I judged that acceptable in practice.

Summary

Putting it all together:

Problem: built on arm64 -> doesn't run on x86_64
Fix:     docker buildx --platform linux/amd64,linux/arm64 --push

Trap:    still doesn't work in Jenkins
Cause:   buildx defaults attach attestation metadata
Fix:     add --provenance=false --sbom=false

There's plenty of documentation on how to use docker buildx itself. But information about "if you use this in Jenkins, you'll hit the attestation problem" is hard to find. Honestly, this part was an unexpected obstacle.

If you're looking to integrate buildx multi-arch builds into a CI environment, I'd recommend making it a habit to inspect the manifest with docker manifest inspect. It directly helps with isolating "I pulled it, so why won't it start?"

Further reading

If you want to learn how to use Docker systematically, the following book is a useful reference (Japanese).

For learning how to design containerized workloads on AWS (ECR, ECS, etc.), this one is recommended.

If you want to dig further into Jenkins configuration and how it interacts with the Docker Plugin, this one is also worth a look.