Why env.PULUMI_BACKEND_URL = ... Returns null: A Silent Trap in Jenkins Declarative Pipeline's environment{} Block

I was trying to wire up a Jenkins Pipeline that resolves the Pulumi backend configuration dynamically from the AWS account ID.

"After auth passes, use env.AWS_ACCOUNT_ID as the key to look up the bucket name and passphrase from YAML, then set them into env." In my head, this was a 30-minute job.

But when I actually wrote it and ran it, the values showed up correctly in the logs inside the function, yet came back as null when read in the next stage. Sandbox approval issues, CPS (Continuation Passing Style) transformations, the difference between vars/ and src/ in the shared library — I piled up hypotheses and knocked them down one by one, and the real culprit was what the title says: just two empty-string placeholder lines written into the environment{} block.

To save you from stepping into the same hole, here is the walk from background to root cause.

Background: dev/prod each ended up spanning multiple AWS accounts

The original problem was a gentle one.

We manage infrastructure with Pulumi, and the S3 buckets for state management live in each AWS account. The existing Jenkins Pipeline was built on the assumption of one account each for dev and prod, and the bucket name and passphrase credential ID simply branched on the ENVIRONMENT parameter (dev or prod).

// Original Jenkinsfile (excerpt)
environment {
    PULUMI_BACKEND_URL = (ENVIRONMENT == 'prod' ? 's3://project-pulumi-state-prod' : 's3://project-pulumi-state-dev')
    PULUMI_CONFIG_PASSPHRASE_ID = 'pulumi-config-passphrase-' + ENVIRONMENT
}

Once the setup evolved into multiple AWS accounts lined up under each of dev and prod, that assumption broke. Dev alone had several accounts, each with its own state bucket. ENVIRONMENT=dev alone no longer determined which bucket to look at.

You enter Account A's credentials, but the pipeline goes looking at Account B's bucket — that kind of accident was now possible.

First design: "put an account key in the project definition"

My first idea was to add an account identifier on the project config side.

# First attempt
pulumi-projects:
  some-repo:
    projects:
      some-project:
        aws_account_key: main-account   # ← added
aws-accounts:
  main-account:
    dev:
      backend_url: s3://...
      passphrase_credential_id: ...
    prod:
      backend_url: s3://...
      passphrase_credential_id: ...

Projects and accounts are basically 1-to-1 (deploying the same Pulumi project in parallel across multiple accounts is not currently a use case), so this seemed like a simple YAGNI-friendly fit.

But while reviewing the design before starting to code, something caught my attention.

"By the time the Initialize and Validate stage in the Jenkinsfile runs verifyAwsCredentials, we already know exactly which AWS account we're operating on."

If I use env.AWS_ACCOUNT_ID obtained from aws sts get-caller-identity, I can know at runtime, with certainty, which account we're talking to — without adding anything to the project config files. All the tagging of aws_account_key across 25+ projects, all the new aws-accounts section — none of it would be necessary.

Shifting the center of gravity from "decide at build time" to "resolve at runtime" meant not touching existing project configs, and it was the more YAGNI-aligned choice.

I changed direction.

AWS credentials entered
  ↓ verifyAwsCredentials (existing)
env.AWS_ACCOUNT_ID is fixed
  ↓ NEW: resolve backend config using the AWS account ID as the key
env.PULUMI_BACKEND_URL / env.PULUMI_CONFIG_PASSPHRASE_ID are fixed
  ↓
The Pulumi execution stage's environment{} reads them and runs

Implementation detour (1): sandbox approval

At first, I added a method to the existing AwsGeneralUtils class in the shared library.

// jenkins/jobs/shared/src/jp/co/example/aws/AwsGeneralUtils.groovy
class AwsGeneralUtils {
    Map resolvePulumiBackend(String accountId) {
        def yamlText = script.libraryResource('aws/pulumi-backends.yaml')
        def config = script.readYaml(text: yamlText)
        // ...
        return [backendUrl: ..., passphraseCredentialId: ...]
    }
}

On the Jenkinsfile side I called it as awsUtils().general.resolvePulumiBackend(env.AWS_ACCOUNT_ID) — in line with the existing pattern.

When I ran it, Jenkins' sandbox blocked it.

Scripts not permitted to use method ... (jp.co.example.aws.AwsGeneralUtils resolvePulumiBackend java.lang.String)

I could have approved it from the admin UI, but what was being requested for approval was groovy.lang.GroovyObject invokeMethod java.lang.String java.lang.Object, and Jenkins itself explicitly warns "approving this may invite security vulnerabilities and is not recommended." This is approval for Groovy's dynamic dispatch mechanism itself — not a specific method — and approving it effectively lets any method be called on any object, which guts the sandbox.

Hard no. I needed a different approach.

I laid out the options:

B was the best. No mixing of Pulumi-specific logic into a general-AWS utility, clean separation of responsibility, reusable from other Pulumi-related jobs in one line, and no admin intervention required.

One thing I had wrong here: I thought "if I call src/ class methods from a vars/ function, I can bypass the sandbox." Not true. CPS-transformed code inside the same Shared Library goes through the same sandbox checks no matter where it's called from. The real condition for no-approval is "being a top-level function recognized as a pipeline step" — if you call out to a class from inside, you're back under the same scrutiny. It's an easy boundary to misread, so worth writing down.

I ended up creating a new vars/pulumiUtils.groovy and putting everything — YAML loading, resolution logic — in it.

// jenkins/jobs/shared/vars/pulumiUtils.groovy (first version)
def resolveBackend(String accountId) {
    def yamlText = libraryResource('aws/pulumi-backends.yaml')
    def config = readYaml(text: yamlText)
    def accounts = config?.accounts
    def backend = accounts[accountId.trim()]
    // ...validation
    return [
        backendUrl: backend.backend_url,
        passphraseCredentialId: backend.passphrase_credential_id
    ]
}

On the Jenkinsfile side:

def resolvePulumiBackend() {
    def backend = pulumiUtils.resolveBackend(env.AWS_ACCOUNT_ID)
    env.PULUMI_BACKEND_URL = backend.backendUrl
    env.PULUMI_CONFIG_PASSPHRASE_ID = backend.passphraseCredentialId
}

The sandbox issue was resolved. When I ran it, the echo inside the function showed the correct values.

ℹ️ Resolved Pulumi backend: accountId=XXXXXXXXXXXX, 
   backendUrl=s3://project-pulumi-state-dev, 
   passphraseCredentialId=pulumi-config-passphrase-dev

But then, the next echo:

✅ Pulumi backend resolution complete
AWS Account ID: XXXXXXXXXXXX
Pulumi Backend: null      ← ?
Passphrase Credential ID: null ← ?

Inside the function the values are there, but immediately after returning, the caller sees null.

Implementation detour (2): wandering between return values and side effects

From here I took the long way around by misreading the cause.

My first suspicion was "returning a Map from a vars/ top-level function can end up as null inside due to CPS transformations — a well-known Jenkins Pipeline quirk." In fact, the existing vars/gitUtils and vars/jenkinsCliUtils don't return values; they finish via side effects (executing git clone and so on).

So I rewrote resolveBackend to a side-effect style.

// switched to side-effect style
def resolveBackend(String accountId) {
    // ...
    env.PULUMI_BACKEND_URL = backendUrl
    env.PULUMI_CONFIG_PASSPHRASE_ID = passphraseCredentialId
}

The Jenkinsfile side just calls:

pulumiUtils.resolveBackend(env.AWS_ACCOUNT_ID)

This should work — or so I thought. Same result. The values written into env inside the function still came out as null when read from the caller.

At this point I got pretty rattled. "If Map return AND env writes both fail, then maybe the boundary of 'top-level function in vars/' itself has a fundamental CPS incompatibility" — I was nearly ready to abandon vars/ and inline everything into the Jenkinsfile.

But I stopped and reread the logs once more.

"Wait — the echo inside the function shows the values correctly. That means the contents of the Map are being built fine. If 'Map return breaks under CPS' were really the cause, the in-function echo should already be broken. The fact that it isn't means that in the return-value version too, it wasn't CPS — something else was eating the values."

Honestly, at that moment I had convinced myself that "both the side-effect and the return-value styles failed," and that belief narrowed my view. The moment I decided "Jenkins CPS is the villain," I briefly lost the energy to generate other hypotheses. I remember that clearly.

I reset and went back to the proper return-value style (returning a Map).

The real culprit: empty-string placeholders in the environment{} block

Back on return-value style, same result. Inside the function the values print, but after assigning them to env.X = backend.xxx in the Jenkinsfile, reading env.X in the next stage gives null.

At this point I finally pointed my suspicion elsewhere.

"What if the write is succeeding, but something on the read side is returning a different value?"

Back to the top of the Jenkinsfile. I reopened the environment {} block.

pipeline {
    environment {
        PULUMI_STACK_NAME = "${params.ENVIRONMENT}"
        PULUMI_BACKEND_URL = ''          // ← this
        PULUMI_CONFIG_PASSPHRASE_ID = '' // ← this
        // ...
    }
    // ...
}

At the start of the implementation I had casually written "I'll overwrite them later via env.X = value, so let me declare them empty here" — just two placeholder lines. I removed them. That alone made it work.

✅ Pulumi backend resolution complete
AWS Account ID: XXXXXXXXXXXX
Pulumi Backend: s3://project-pulumi-state-dev
Passphrase Credential ID: pulumi-config-passphrase-dev

…That was the culprit.

What was actually happening

The environment{} block in Jenkins Declarative Pipeline gets special treatment that differs from ordinary env.X = value assignments.

Roughly speaking, variables declared in environment{} are registered into a separate internal binding inside Jenkins as a declaration saying "this variable has this value across the entire pipeline." This is not just initial-value setting — at stage boundaries and on env property read paths, the declared value may take priority over (or overwrite) values assigned at runtime.

In this case:

When declaration and runtime assignment compete, the runtime side loses and the declaration side wins. "Once you declare it, Jenkins owns it" — that's the behavior.

Why it surfaced as null rather than '' is speculation, but I suspect the declarative env's internal state management treats an empty-string declaration as something close to "uninitialized," and once it crosses the CPS serialization boundary, it appears as null on the other side (I didn't chase the internal implementation to the bottom).

Here's a conceptual picture of where the value propagation was breaking:

The lesson

In Declarative Pipeline, variables declared in the environment{} block should not be overwritten at runtime via env.X = value — that's the biggest takeaway.

Sorted out, the split is:

→ declare it in environment{}

→ do not declare it in environment{}. Just do env.X = value in a script block, and later stages' environment{} can still reference it as env.X.

This time, the two "I'll overwrite these later, so let me declare them empty for now" lines that I wrote unconsciously silently swallowed the runtime assignments. No comment, no lint warning — the completely silent breakage kind of trap.

Side note: is the side-effect style an anti-pattern?

During the back-and-forth, the question came up — "isn't rewriting env from inside a function a side-effect anti-pattern?" Let me add a note here.

My conclusion: the side effect itself is not the anti-pattern; the problem is hidden side effects.

Real programs don't work without side effects — file writes, DB updates, logging, env setting, screen rendering. The "avoid side effects" guidance from functional programming concretely means:

A vars/ function that rewrites env.X is hard to intuit from the name (resolveBackend) and argument (accountId) — "it rewrites env" doesn't jump out, which makes it prone to being a hidden side effect. JSDoc and documentation ("sets this in env") mitigate it somewhat, but returning a value and having the caller set it keeps the blast radius scoped to the caller and easier to follow.

The reason this ended up in return-value style isn't "side effect vs return is right or wrong"; I temporarily switched to the side-effect style to isolate the cause. In the end, the return-value style was the more honest design — it makes intent and scope clearer.

The final design

While tidying up the requirements, I learned that "one AWS account may host both dev and prod" is possible (though the S3 buckets are still split per-environment), so I nested the YAML one more level.

# resources/aws/pulumi-backends.yaml
accounts:
  "XXXXXXXXXXX1":
    dev:
      backend_url: s3://project-pulumi-state-dev
      passphrase_credential_id: pulumi-config-passphrase-dev
  "XXXXXXXXXXX2":
    prod:
      backend_url: s3://project-pulumi-state-prod
      passphrase_credential_id: pulumi-config-passphrase-prod
  # If one account has both dev and prod, put both entries
  # "XXXXXXXXXXX3":
  #   dev:
  #     backend_url: s3://...
  #     passphrase_credential_id: ...
  #   prod:
  #     backend_url: s3://...
  #     passphrase_credential_id: ...

The shared library function now takes two arguments (accountId × environment) and returns a Map.

// vars/pulumiUtils.groovy
def resolveBackend(String accountId, String environment) {
    if (!accountId?.trim()) {
        error 'Account ID not specified.'
    }
    if (!environment?.trim()) {
        error 'Environment name (dev/prod etc.) not specified.'
    }

    def yamlText = libraryResource('aws/pulumi-backends.yaml')
    def config = readYaml(text: yamlText)
    def accounts = config?.accounts

    def accountEntry = accounts[accountId.trim()]
    if (!(accountEntry instanceof Map)) {
        error "Unregistered account ID: ${accountId}. Registered: ${accounts.keySet().sort().join(', ')}"
    }

    def backend = accountEntry[environment.trim()]
    if (!(backend instanceof Map)) {
        error "Account ${accountId} has no configuration for environment '${environment}'. Registered environments: ${accountEntry.keySet().sort().join(', ')}"
    }

    String backendUrl = backend.backend_url?.toString()?.trim()
    String passphraseCredentialId = backend.passphrase_credential_id?.toString()?.trim()
    if (!backendUrl || !passphraseCredentialId) {
        error "Account ${accountId} / environment ${environment} configuration is incomplete."
    }

    return [
        backendUrl: backendUrl,
        passphraseCredentialId: passphraseCredentialId
    ]
}

The Jenkinsfile side (with the empty-string placeholders in environment{} removed):

def resolvePulumiBackend() {
    def backend = pulumiUtils.resolveBackend(env.AWS_ACCOUNT_ID, params.ENVIRONMENT)
    env.PULUMI_BACKEND_URL = backend.backendUrl
    env.PULUMI_CONFIG_PASSPHRASE_ID = backend.passphraseCredentialId
}

The overall flow:

Reference: the relevant parts of the actual Jenkinsfile

Words alone are hard to picture, so here is an excerpt from the final working Jenkinsfile. Three points:

Top level (no empty-string placeholders in environment{})

@Library('jenkins-shared-lib') _

pipeline {
    agent {
        label 'ec2-fleet'
    }

    environment {
        // Directory/path constants (fixed at build start)
        SOURCE_CODE_DIR  = 'source-code'
        JENKINS_REPO_DIR = 'jenkins-repo'

        // Timestamp
        TIME_STAMP = sh(script: 'TZ="Asia/Tokyo" date "+%Y/%m/%d %H:%M:%S"', returnStdout: true).trim()

        // Jenkins repository config
        JENKINS_REPO_URL           = 'git@github.com:your-org/your-repo'
        JENKINS_REPO_BRANCH        = 'main'
        JENKINS_GIT_CREDENTIALS_ID = 'github-deploy-key-your-repo'

        // Environment-specific (decided at build start from the ENVIRONMENT parameter)
        PULUMI_STACK_NAME = "${params.ENVIRONMENT}"

        // Teams Webhook URL
        TEAMS_WEBHOOK_URL = '<your-teams-webhook-url>'

        // ★ Do NOT place PULUMI_BACKEND_URL = '' or PULUMI_CONFIG_PASSPHRASE_ID = '' here.
        // The moment you do, runtime assignments get swallowed by the declarative shadow.
    }

    stages {
        // ...
    }
}

As the comment says, the exact thing you're tempted to do ("I'll overwrite it later, so declare it empty now") is what you must not do — that's the core of this design.

Initialize and Validate stage (call verifyAwsCredentials → resolvePulumiBackend)

stage('Initialize and Validate') {
    steps {
        script {
            currentBuild.displayName = "#${env.BUILD_NUMBER} - ${params.ENVIRONMENT} ${params.ACTION}"

            validateParameters()
            prepareWorkspace()
            checkoutRepositories()
            verifyAwsCredentials()        // ← env.AWS_ACCOUNT_ID is fixed
            resolvePulumiBackend()        // ← env.PULUMI_BACKEND_URL / PASSPHRASE_ID are fixed
            dir("${SOURCE_CODE_DIR}/${params.PULUMI_PROJECT_PATH}") {
                handlePulumiConfigFile()
            }
        }
    }
}

The call order matters: resolvePulumiBackend must be called after verifyAwsCredentials has set env.AWS_ACCOUNT_ID.

Top-level function resolvePulumiBackend()

/**
 * Resolve Pulumi backend config from the AWS account ID × environment.
 */
def resolvePulumiBackend() {
    try {
        def backend = pulumiUtils.resolveBackend(env.AWS_ACCOUNT_ID, params.ENVIRONMENT)
        env.PULUMI_BACKEND_URL          = backend.backendUrl
        env.PULUMI_CONFIG_PASSPHRASE_ID = backend.passphraseCredentialId

        echo """
            ✅ Pulumi backend resolution complete
            AWS Account ID: ${env.AWS_ACCOUNT_ID}
            Environment: ${params.ENVIRONMENT}
            Pulumi Backend: ${env.PULUMI_BACKEND_URL}
            Passphrase Credential ID: ${env.PULUMI_CONFIG_PASSPHRASE_ID}
        """.stripIndent()
    } catch (Exception e) {
        error "Failed to resolve Pulumi backend configuration: ${e.message}"
    }
}

Receive the Map from pulumiUtils.resolveBackend in the shared library, and do the env.X = value assignment in this Jenkinsfile's scope — that's the return-value style we landed on. Now both the write and the read are in the same scope, and values don't slip through.

Execute Pulumi Operations stage (referenced from a nested environment{})

stage('Execute Pulumi Operations') {
    agent {
        docker {
            label 'ec2-fleet'
            image 'pulumi/pulumi:latest'
            args "--entrypoint='' -v ${WORKSPACE}:/workspace -w /workspace -u root"
            reuseNode true
        }
    }
    environment {
        // AWS credentials
        AWS_ACCESS_KEY_ID     = "${params.AWS_ACCESS_KEY_ID}"
        AWS_SECRET_ACCESS_KEY = "${params.AWS_SECRET_ACCESS_KEY}"
        AWS_SESSION_TOKEN     = "${params.AWS_SESSION_TOKEN}"
        AWS_DEFAULT_REGION    = "${params.AWS_REGION}"

        // Pulumi config — env.PULUMI_BACKEND_URL was set by resolvePulumiBackend() above
        PULUMI_BACKEND_URL       = "${env.PULUMI_BACKEND_URL}"
        PULUMI_CONFIG_PASSPHRASE = credentials("${env.PULUMI_CONFIG_PASSPHRASE_ID}")
        PULUMI_SKIP_UPDATE_CHECK = 'true'
    }
    stages {
        // ... Pulumi execution sub-stages
    }
}

The key is that env.PULUMI_BACKEND_URL is referenced here via ${...} expansion. The value set earlier by resolvePulumiBackend() reads back cleanly, and the Pulumi CLI picks up this environment variable to access the backend.

Don't place it in the top-level environment{}; reference it via env inside a child stage's environment{} — that's the honest way to handle "variables whose values are determined at runtime."

Further reading

When you fall into a behavioral trap like this one in Declarative Pipeline, going back once and re-grounding yourself in how Jenkins actually works tends to sharpen your nose for similar traps later. Having a practical book that covers the fundamentals through real-world Pipeline / Shared Library usage on your desk gives investigation a place to stand.

And if you want to shore up the "how should I design the CI/CD pipeline itself?" angle — the kind of thinking that makes the distinction between environment{} and runtime assignment feel natural — the classic Continuous Delivery is still very much worth reading. Internalizing its principles for how state flows through build, test, and deployment changes how quickly you can triangulate when a trap like this appears.

Wrap-up

Looking back at the detour, the original hypotheses — "returning a Map from vars/ becomes null due to CPS" and "writes into env from vars/ don't cross the CPS boundary" — were phenomena I did observe, but not the root cause. The real cause was the empty-string placeholders in environment{} silently overriding runtime assignments, and the surface symptom (values exist inside the function but not outside) happened to align neatly with a completely different hypothesis (CPS / vars boundary problem). That alignment is what made this hard.

It's a silent-type trap with no error message, easy to miss even in code review. If someone else walks into the same maze, and this article helps them say "ah, those two lines" 30 minutes earlier, it will have been worth writing.

The key points again: