Symlinking Only node_modules From an ECR Image — A Jenkins Optimization Story With Three Wrong Turns

In the previous post, I wrote about getting thoroughly stuck on Docker agent user permissions.

This time it's a different problem from the same project. The theme: "we have pre-built artifacts in our ECR image, but we're running npm install every time" — and how to fix that waste.

The short answer: "symlink only node_modules, rebuild dist every time." But getting there involved three wrong turns in verification logic, so here's the record.

Background: the ECR image has artifacts, but we rebuild every time

In this project, Jenkins Docker agents pull a pre-built ECR image. The image contains /workspace/node_modules and /workspace/dist, so in theory we could skip the build step when a job starts.

Yet setupNodeEnvironment(), the shared setup function, was running this every time:

sh '''
    npm ci --include=dev    // 12s + network every time
    npm run build           // tsc every time
'''

The image wasn't really being leveraged.

The problems:

So I filed Issue #833 to switch to "reuse artifacts via symlink."

Context: the environment

First attempt: symlink both node_modules and dist

The idea was straightforward.

// Symlink ECR image artifacts into the Jenkins workspace
if ([ -d /workspace/node_modules ] && [ ! -e node_modules ]); then
    ln -s /workspace/node_modules node_modules
fi
if ([ -d /workspace/dist ] && [ ! -e dist ]); then
    ln -s /workspace/dist dist
fi

But just because the symlink works doesn't mean the artifacts are usable. If the ECR image is stale, dist/index.js could be out of sync with the source. So I added a safety-net verification:

// Verify artifact integrity
if node -e "require('./dist/index.js')" 2>/dev/null; then
    echo "ECR image artifacts verified successfully. Skipping npm install & build."
else
    echo "WARNING: ECR image artifacts verification failed. Falling back..."
    rm -f node_modules dist
    npm ci --include=dev
    npm run build
fi

Result: always falls back

When I ran it in Jenkins:

+ node -e require('./dist/index.js')
+ echo WARNING: ECR image artifacts verification failed. Falling back to full install & build...
WARNING: ECR image artifacts verification failed. Falling back to full install & build...
+ rm -f node_modules dist
+ npm ci --include=dev
...(npm install runs)

The symlink succeeded, but verification always failed.

Cause: dist/index.js executes runCli() the moment it's require()'d. Commander runs with no arguments, finds no matching command, and exits with an error. Since 2>/dev/null was swallowing the output, it took a while to notice.

Fix 1: use the --help flag

Switched to a side-effect-free verification.

if node dist/index.js --help >/dev/null 2>&1; then
    echo "ECR image artifacts verified successfully."
else
    ...

--help doesn't execute commands and returns exit code 0. This was the same verification pattern used in the ecr-verify job.

Result: still falls back

+ node dist/index.js --help
+ echo WARNING: ECR image artifacts verification failed...

Cause: node dist/index.js is hitting the symlinked /workspace/dist/index.js — the pre-built artifact from the ECR image. That image was built by a weekly cron, from before the --help subcommand was added.

The ECR image is too old to have the check capability. The fallback is working correctly, but the goal (skipping npm install) isn't being achieved.

Fix 2: add a check subcommand

"The --help flag could gain side effects in the future — let's make a dedicated CI verification subcommand."

// src/main.ts
program
  .command('check')
  .description('Verify that build artifacts are loadable (used by CI setup)')
  .action(() => {
    process.stdout.write('ok\n');
  });

Simply outputs ok and exits.

if node dist/index.js check >/dev/null 2>&1; then
    echo "ECR image artifacts verified successfully. Skipping npm install & build."
else
    ...

Built locally and confirmed:

$ node dist/index.js check
ok

Wrote tests and submitted a PR.

Result: falls back again (third time)

+ node dist/index.js check
+ echo WARNING: ECR image artifacts verification failed. Falling back to full install & build...

Cause: The code that added the check command won't be baked into the ECR image until the next build cycle. The currently running image doesn't know about check, so it exits as an unknown command.

I'd walked into a chicken-and-egg problem: "verifying ECR image integrity using a feature that requires the ECR image to be updated first."

Rethinking from the root: node_modules and dist are fundamentally different

At this point I stepped back.

The dilemma was "keep the ECR image current while also reducing build CPU." But node_modules and dist have completely different characteristics:

node_modules:
  - Identical as long as package.json doesn't change
  - Change frequency: low (only on dependency add/update)
  - Regeneration cost: high (npm install → 12s + network + CPU)

dist:
  - Changes every time source changes
  - Change frequency: high (every code change)
  - Regeneration cost: low (tsc → a few seconds)

So: symlink only node_modules, and rebuild dist every time.

Minimize dependency on the ECR image while skipping only the expensive part.

Final implementation

// node_modules: reuse from ECR image via symlink
if [ -d /workspace/node_modules ] && [ ! -e node_modules ]; then
    # Verify that package.json dependencies match
    if [ -f /workspace/package.json ] && \
       node -e "
         const a = require('/workspace/package.json');
         const b = require('./package.json');
         const eq = JSON.stringify(a.dependencies) === JSON.stringify(b.dependencies) &&
                    JSON.stringify(a.devDependencies) === JSON.stringify(b.devDependencies);
         process.exit(eq ? 0 : 1);
       " 2>/dev/null; then
        echo "Linking pre-built node_modules from ECR image..."
        ln -s /workspace/node_modules node_modules
    else
        echo "WARNING: package.json dependencies mismatch. Running npm ci..."
        npm ci --include=dev
    fi
elif [ ! -e node_modules ]; then
    echo "WARNING: ECR image node_modules not found. Running npm install..."
    npm install --include=dev
fi

# dist: always build from latest source
echo "Building TypeScript sources..."
npm run build

The verification logic changed. From "execute dist/index.js and check" to "compare package.json contents."

This approach:

The log that actually worked

+ [ -d /workspace/node_modules ]
+ [ ! -e node_modules ]
+ [ -f /workspace/package.json ]
+ node -e ...                        ← dependencies comparison → match (exit 0)
+ echo Linking pre-built node_modules from ECR image...
  Linking pre-built node_modules from ECR image...
+ ln -s /workspace/node_modules node_modules
+ echo Building TypeScript sources...
  Building TypeScript sources...
+ npm run build

> project@0.2.0 build
> tsc -p tsconfig.json && node ./scripts/copy-static-assets.mjs

[OK] Copied ...

npm install is skipped. Only npm run build runs. Exactly as intended.

Summary

The trial-and-error path:

"node_modules and dist have different change frequencies and regeneration costs" — it's obvious in hindsight, but it took a few detours to get there.

Further reading

If you want to go deeper on Jenkins pipeline design and practical Docker container operations, these books are good starting points.