Passing variables between Azure Pipelines stages, jobs and deployment jobs

Azure Pipelines, the continuous integration and continuous deployment feature of Azure DevOps, has the concept of variables. For scripts and tasks, they behave just like environment variables. There's a bunch that are predefined that you might have used before, like System.AccessToken, System.DefaultWorkingDirectory and Build.ArtifactStagingDirectory.

In YAML pipelines, you can define your own variables in:

a variable block in YAML
```
  variables:
    one: initialValue 
```
This variable block can also reference previously defined variable groups. Variable groups are managed in the Library tab under the Pipelines menu of your Azure DevOps project.
```
  variables:
  - group: "Contoso Variable Group"
  - name: anothervariable
    value: 'Hi there'
```
The UI for the YAML pipeline.

A script using task.setvariable

  - bash: |
      echo "##vso[task.setvariable variable=myVar;]foo"
  - bash: |
      echo "You can use macro syntax for variables: $(myVar)"

It's this last case (using task.setvariable) that can be interesting, as by default that variable is only available to subsequent tasks in the same job. To access the variable from other jobs or stages, you need to add isoutput=true and give the step a name. eg.

  - bash: |
     echo "##vso[task.setvariable variable=myOutputJobVar;isoutput=true]this is the same job too"
    name: setOutput

And here's where it gets interesting. Depending on whether the task that is setting the variable (often referred to as an 'output variable') is in a regular job or a deployment job, and whether you're wanting to reference the variable from another job in the same stage or a job in a subsequent stage, the syntax varies.

If you're not familiar with them, a deployment job is the YAML equivalent of the GUI-based 'Classic' Release pipelines. They are similar to regular pipeline jobs, but there are some differences too.

Most of the existing examples for referencing output variables are listed in the documentation Set variables in scripts, but as deployment jobs have a different syntax, there's another section under Deployment jobs. Unfortunately, between these two pages, not all the possibilities seem to be covered.

So I came up with a pipeline workflow that correctly demonstrates these options:

Deployment Job to another Job in the same stage
Deployment Job to another Job in a different stage
Deployment Job to another Deployment Job in a different stage
Job to another Job in a different stage

Here's the entire YAML pipeline definition (source on GitHub)

trigger:
 - master

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Stage1
  displayName: Stage 1
  jobs:
  - deployment: Stage1DeploymentJob1
    displayName: Stage 1 Deployment Job 1
    environment: Environment1
    strategy:
      runOnce:
        deploy:
          steps:
          - bash: echo "##vso[task.setvariable variable=my_Stage1DeploymentJob1_OutputVar;isOutput=true]Variable from $(Agent.JobName)"
            name: stepVar_Stage1DeploymentJob1
            displayName: Set my_Stage1DeploymentJob1_OutputVar

- stage: Stage2
  displayName: Stage 2
  dependsOn: Stage1
  jobs:
  - deployment: Stage2DeploymentJob1
    displayName: Stage 2 Deployment Job 1
    environment: Environment2
    strategy:
      runOnce:
        deploy:
          steps:
          - bash: echo "##vso[task.setvariable variable=my_Stage2DeploymentJob1_OutputVar;isOutput=true]Variable from $(Agent.JobName)"
            name: stepVar_Stage2DeploymentJob1
            displayName: Set my_Stage2DeploymentJob1_OutputVar

  - job: Stage2Job2
    displayName: Stage 2 Job 2
    dependsOn: Stage2DeploymentJob1
    variables:
      varFrom_Stage2DeploymentJob1: $[ dependencies.Stage2DeploymentJob1.outputs['Stage2DeploymentJob1.stepVar_Stage2DeploymentJob1.my_Stage2DeploymentJob1_OutputVar'] ]
    steps:
      - checkout: none
      - bash: echo $(varFrom_Stage2DeploymentJob1)
        displayName: Display varFrom_Stage2DeploymentJob1

- stage: Stage3
  displayName: Stage 3
  dependsOn: Stage1
  variables:
    varFrom_Stage1DeploymentJob1: $[ stageDependencies.Stage1.Stage1DeploymentJob1.outputs['Stage1DeploymentJob1.stepVar_Stage1DeploymentJob1.my_Stage1DeploymentJob1_OutputVar'] ]
  jobs:
  - deployment: Stage3DeploymentJob1
    displayName: Stage 3 Deployment Job 1
    environment: Environment3
    strategy:
      runOnce:
        deploy:
          steps:
          - bash: echo $(varFrom_Stage1DeploymentJob1)
            displayName: Display varFrom_Stage1DeploymentJob1
          - bash: printenv
            displayName: printenv

  - job: Stage3Job2
    displayName: Stage 3 Job 2
    steps:
      - checkout: none
      - bash: echo "##vso[task.setvariable variable=my_Stage3Job2_OutputVar;isOutput=true]Variable from $(Agent.JobName)"
        name: stepVar_Stage3Job2
        displayName: Set my_Stage3Job2_OutputVar

- stage: Stage4
  displayName: Stage 4
  dependsOn:
    # Need to mention stage here to be able to reference variables from it, even though the dependency is implied by Stage2 and Stage3
    - Stage1
    - Stage3
    - Stage2
  jobs:
  - job: Stage4Job1
    displayName: Stage 4 Job 1
    variables:
      varFrom_Stage1DeploymentJob1: $[ stageDependencies.Stage1.Stage1DeploymentJob1.outputs['Stage1DeploymentJob1.stepVar_Stage1DeploymentJob1.my_Stage1DeploymentJob1_OutputVar'] ]
      varFrom_Stage2DeploymentJob1: $[ stageDependencies.Stage2.Stage2DeploymentJob1.outputs['Stage2DeploymentJob1.stepVar_Stage2DeploymentJob1.my_Stage2DeploymentJob1_OutputVar'] ]
      varFrom_Stage3Job2: $[ stageDependencies.Stage3.Stage3Job2.outputs['stepVar_Stage3Job2.my_Stage3Job2_OutputVar'] ]
    
    steps:
    - checkout: none
    - bash: |
        echo "varFrom_Stage1DeploymentJob1: $(varFrom_Stage1DeploymentJob1)"
        echo "varFrom_Stage2DeploymentJob1: $(varFrom_Stage2DeploymentJob1)"
        echo "varFrom_Stage3Job2: $(varFrom_Stage3Job2)"
      displayName: Display variables
    - bash: printenv
      displayName: printenv

The stage dependencies result in the following flow:

Azure Pipeline stage workflow

Here's the output from the Stage4Job1's Display variables step:

Starting: Display variables
==============================================================================
Task         : Bash
Description  : Run a Bash script on macOS, Linux, or Windows
Version      : 3.201.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
========================== Starting Command Output ===========================
/usr/bin/bash /home/vsts/work/_temp/8ce87e72-545d-4804-8461-3910a1412c90.sh
varFrom_Stage1DeploymentJob1: Variable from Stage 1 Deployment Job 1
varFrom_Stage2DeploymentJob1: Variable from Stage 2 Deployment Job 1
varFrom_Stage3Job2: Variable from Stage 3 Job 2
Finishing: Display variables

Some things to note:

The format for referencing a deployment job from a different stage is $[ stageDependencies.<StageName>.<DeploymentJobName>.outputs['<DeploymentJobName>.<StepName>.<VariableName>'] ] - I've seen some online posts suggesting the environment name be used instead of the second DeploymentJobName, but I think that is incorrect - this format works for me.
As the comment in the YAML mentions, if you want to reference a variable from a job or stage, then you have to add that to your dependsOn list. The fact that Stage 1 is 'upstream' from Stage 4 (because Stage 4 explicitly depends on Stage 2 and Stage 3) isn't enough.

Summary

In this post, we saw how to reference variables from previous Jobs or Deployment Jobs in either the same or different stages of an Azure Pipeline.