Passing variables between Azure Pipelines stages, jobs and deployment jobs

Azure Pipelines, the continuous integration and continuous deployment feature of Azure DevOps, has the concept of variables. For scripts and tasks, they behave just like environment variables. There's a bunch that are predefined that you might have used before, like System.AccessToken, System.DefaultWorkingDirectory and Build.ArtifactStagingDirectory.

In YAML pipelines, you can define your own variables in:

• a variable block in YAML

    variables:
      one: initialValue 
  

  This variable block can also reference previously defined variable groups. Variable groups are managed in the Library tab under the Pipelines menu of your Azure DevOps project.

    variables:
    - group: "Contoso Variable Group"
    - name: anothervariable
      value: 'Hi there'
  

• The UI for the YAML pipeline.

  

• A script using task.setvariable

    - bash: |
        echo "##vso[task.setvariable variable=myVar;]foo"
    - bash: |
        echo "You can use macro syntax for variables: $(myVar)"
  

It's this last case (using task.setvariable) that can be interesting, as by default that variable is only available to subsequent tasks in the same job. To access the variable from other jobs or stages, you need to add isoutput=true and give the step a name. eg.

  - bash: |
     echo "##vso[task.setvariable variable=myOutputJobVar;isoutput=true]this is the same job too"
    name: setOutput

And here's where it gets interesting. Depending on whether the task that is setting the variable (often referred to as an 'output variable') is in a regular job or a deployment job, and whether you're wanting to reference the variable from another job in the same stage or a job in a subsequent stage, the syntax varies.

If you're not familiar with them, a deployment job is the YAML equivalent of the GUI-based 'Classic' Release pipelines. They are similar to regular pipeline jobs, but there are some differences too.

Most of the existing examples for referencing output variables are listed in the documentation Set variables in scripts, but as deployment jobs have a different syntax, there's another section under Deployment jobs. Unfortunately, between these two pages, not all the possibilities seem to be covered.

So I came up with a pipeline workflow that correctly demonstrates these options:

• Deployment Job to another Job in the same stage
• Deployment Job to another Job in a different stage
• Deployment Job to another Deployment Job in a different stage
• Job to another Job in a different stage

Here's the entire YAML pipeline definition (source on GitHub)

trigger:
 - master

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Stage1
  displayName: Stage 1
  jobs:
  - deployment: Stage1DeploymentJob1
    displayName: Stage 1 Deployment Job 1
    environment: Environment1
    strategy:
      runOnce:
        deploy:
          steps:
          - bash: echo "##vso[task.setvariable variable=my_Stage1DeploymentJob1_OutputVar;isOutput=true]Variable from $(Agent.JobName)"
            name: stepVar_Stage1DeploymentJob1
            displayName: Set my_Stage1DeploymentJob1_OutputVar

- stage: Stage2
  displayName: Stage 2
  dependsOn: Stage1
  jobs:
  - deployment: Stage2DeploymentJob1
    displayName: Stage 2 Deployment Job 1
    environment: Environment2
    strategy:
      runOnce:
        deploy:
          steps:
          - bash: echo "##vso[task.setvariable variable=my_Stage2DeploymentJob1_OutputVar;isOutput=true]Variable from $(Agent.JobName)"
            name: stepVar_Stage2DeploymentJob1
            displayName: Set my_Stage2DeploymentJob1_OutputVar

  - job: Stage2Job2
    displayName: Stage 2 Job 2
    dependsOn: Stage2DeploymentJob1
    variables:
      varFrom_Stage2DeploymentJob1: $[ dependencies.Stage2DeploymentJob1.outputs['Stage2DeploymentJob1.stepVar_Stage2DeploymentJob1.my_Stage2DeploymentJob1_OutputVar'] ]
    steps:
      - checkout: none
      - bash: echo $(varFrom_Stage2DeploymentJob1)
        displayName: Display varFrom_Stage2DeploymentJob1

- stage: Stage3
  displayName: Stage 3
  dependsOn: Stage1
  variables:
    varFrom_Stage1DeploymentJob1: $[ stageDependencies.Stage1.Stage1DeploymentJob1.outputs['Stage1DeploymentJob1.stepVar_Stage1DeploymentJob1.my_Stage1DeploymentJob1_OutputVar'] ]
  jobs:
  - deployment: Stage3DeploymentJob1
    displayName: Stage 3 Deployment Job 1
    environment: Environment3
    strategy:
      runOnce:
        deploy:
          steps:
          - bash: echo $(varFrom_Stage1DeploymentJob1)
            displayName: Display varFrom_Stage1DeploymentJob1
          - bash: printenv
            displayName: printenv

  - job: Stage3Job2
    displayName: Stage 3 Job 2
    steps:
      - checkout: none
      - bash: echo "##vso[task.setvariable variable=my_Stage3Job2_OutputVar;isOutput=true]Variable from $(Agent.JobName)"
        name: stepVar_Stage3Job2
        displayName: Set my_Stage3Job2_OutputVar

- stage: Stage4
  displayName: Stage 4
  dependsOn:
    # Need to mention stage here to be able to reference variables from it, even though the dependency is implied by Stage2 and Stage3
    - Stage1
    - Stage3
    - Stage2
  jobs:
  - job: Stage4Job1
    displayName: Stage 4 Job 1
    variables:
      varFrom_Stage1DeploymentJob1: $[ stageDependencies.Stage1.Stage1DeploymentJob1.outputs['Stage1DeploymentJob1.stepVar_Stage1DeploymentJob1.my_Stage1DeploymentJob1_OutputVar'] ]
      varFrom_Stage2DeploymentJob1: $[ stageDependencies.Stage2.Stage2DeploymentJob1.outputs['Stage2DeploymentJob1.stepVar_Stage2DeploymentJob1.my_Stage2DeploymentJob1_OutputVar'] ]
      varFrom_Stage3Job2: $[ stageDependencies.Stage3.Stage3Job2.outputs['stepVar_Stage3Job2.my_Stage3Job2_OutputVar'] ]
    
    steps:
    - checkout: none
    - bash: |
        echo "varFrom_Stage1DeploymentJob1: $(varFrom_Stage1DeploymentJob1)"
        echo "varFrom_Stage2DeploymentJob1: $(varFrom_Stage2DeploymentJob1)"
        echo "varFrom_Stage3Job2: $(varFrom_Stage3Job2)"
      displayName: Display variables
    - bash: printenv
      displayName: printenv

The stage dependencies result in the following flow:

Here's the output from the Stage4Job1's Display variables step:

Starting: Display variables
==============================================================================
Task         : Bash
Description  : Run a Bash script on macOS, Linux, or Windows
Version      : 3.201.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
========================== Starting Command Output ===========================
/usr/bin/bash /home/vsts/work/_temp/8ce87e72-545d-4804-8461-3910a1412c90.sh
varFrom_Stage1DeploymentJob1: Variable from Stage 1 Deployment Job 1
varFrom_Stage2DeploymentJob1: Variable from Stage 2 Deployment Job 1
varFrom_Stage3Job2: Variable from Stage 3 Job 2
Finishing: Display variables

Some things to note:

• The format for referencing a deployment job from a different stage is $[ stageDependencies.<StageName>.<DeploymentJobName>.outputs['<DeploymentJobName>.<StepName>.<VariableName>'] ] - I've seen some online posts suggesting the environment name be used instead of the second DeploymentJobName, but I think that is incorrect - this format works for me.
• As the comment in the YAML mentions, if you want to reference a variable from a job or stage, then you have to add that to your dependsOn list. The fact that Stage 1 is 'upstream' from Stage 4 (because Stage 4 explicitly depends on Stage 2 and Stage 3) isn't enough.

Summary

In this post, we saw how to reference variables from previous Jobs or Deployment Jobs in either the same or different stages of an Azure Pipeline.

UniFi USW Flex Mini 5-port switch

My work from home desk has a great outlook, but unfortunately, it's on the opposite side to the room's network wall sockets. For this reason, I run a single cable around the room perimeter (mostly hidden behind sofas).

In my work with SixPivot, clients sometimes provide a laptop for me to use. In that case, I was having to use wireless, but I'll always prefer wired over wireless if possible. I could run a second cable, but that's getting messy, so the tidier option was to install a small switch on my desk to allow adding extra devices there. 5 port switches are pretty cheap, but often you pay a premium if you want a 'managed' switch. My colleague Shaw pointed out that Ubiquiti has a managed 5-port switch for around $AU50, and even better, it can be powered by Power over Ethernet (PoE).

The UniFi USW Flex Mini 5-port switch is (no surprise) a Gigabit switch with 5 ports. It can be powered either via a USB-C adapter, or 802.3af PoE (via port 1).

You can buy it from resellers on Amazon AU or Amazon US, but I had to hunt around to find a reseller with stock in Australia. I ended up going with UBWH Australia. The web page specifically calls out that the product does not ship with a power supply (which was fine as I was intending to use PoE).

It took a few days to arrive (I was starting to wonder if it was walking across the Nullabor. Despite the previous warnings, I was surprised to find that it did indeed include a power supply, but curiously it appears to have a US-style plug (so not that useful).

I made sure that the network cable coming to the UniFI switch had PoE and then plugged the network cable in. The device status LEDs then illuminated, giving me confidence that it was working. Adding extra cables to connect the two laptops on my desk and moments later they were both online.

It's a managed switch, and as I already run the UniFi Network Application (formerly UniFi Controller) software in a Docker container on my Synology server, I was able to add the device to the management application.

I followed these steps to manage the switch:

1. Open UniFi Controller
2. Navigate to 'UniFi Devices' tab
3. Click on Click to Adopt
4. In the details tab for device, click on Adopt Device
5. Wait for device status to change to Online.
6. You may see an optional firmware update available. You can apply the update by clicking on Click to Update in the device list, or on Apply Update in the device details panel

Here's the Overview panel for the device after it has been in operation for a few days:

So far, so good!

Podcasts 2022

Is it really 12 years since I last posted about what's in my podcast collection? Apparently so!

My current podcast app of choice on iOS is Pocket Casts. It works well, and it's pretty cool that it was originally developed here in Adelaide!

Most of my podcast listening is now done on my morning walk before I start work (now that I work from home). That's usually up to an hour walking. Previously I used to listen on the bus trip into the city (which was about an hour each way), so now with a bit less time I tend to be more selective about what I listen to for the "second half".

I've roughly sorted the podcasts by how regularly I listen to episodes. This is mostly how useful or interesting I find the episodes, but may also be because the podcast doesn't publish episodes that often. Links are to the RSS feeds for each podcast.

• Daily Audio Bible". You might say I listen to this religiously! It's the first thing I put on when heading out for my walk. Episodes take from 20-30 mins, which usually means I have time to listen to something else before I get home.
• RunAs Radio - Interesting IT topics hosted by Richard Campbell.
• Windows Weekly - Weekly episodes with Leo, Paul and Mary Jo, that I don't always get through and sometimes miss altogether, but often interesting.
• The Unhandled Exception Podcast - A newer addition to my list. Some good shows on .NET. Listening to Dan Clarke's interviews has inspired a few of our recent speakers for ADNUG (Kendra Havens and Simon Cropp).
• .NET Rocks! - Carl and Richard. Usually an hour per episode, and if the topic isn't interesting I'll give it a miss.
• no dogma podcast - Bryan Hogan (wonderful accent!) and some great .NET interviews recently.
• Hanselminutes with Scott Hanselman - Scott covers a wide variety of software dev topics.
• Dr. Karl Podcast - Always interesting science talk-back.
• Herding Code - Episodes are sporadic, and even less often since the passing of K Scott Allen. But when they do come out they can be good.
• Software Engineering Radio - the podcast for professional software developers - a wide variety of software topics.
• SQL Down Under - Greg Low's SQL podcast
• Seeds Uniting Church - Good to catch up on sermons. Used to use this more often when I missed out when I was helping with kids' church (aka Sunday School).
• One Dev Minute - Selective listening to topics from Raymond Chen and Larry Osterman (whose blogs I've been following for ages). No new episodes since last year though.
• The Idealcast with Gene Kim by IT Revolution. Some really interesting interviews, hosted by Gene Kim. Some shows are recaps of older conference talks. If you're interested in DevOps, then can be interesting. Episodes can be quite long. I wish they'd didn't break them down into shorter episodes.
• Coding After Work Podcast - Not sure how I got onto this one, not publishing that often now, but a few of the earlier episodes had some interesting guests.
• Weekly Dev Tips - From Steve Smith (@ardalis), has gone quiet and I think he's stopped publishing?
• A special mention should also go to Developer On Fire, which I listened to for the first few hundred episodes a few years back. It burned brightly for a while there. I gather Dave Rael wound it up after episode 452.

Finally, sometimes after listening to the latest Daily Audio Bible episode, I decide to "unplug" and just take in the sounds around me. It's ok to not be plugged in all the time!

2022-06-24 Updated broken links to 'Seeds Uniting Church' and 'One Dev Minute'