Converting Azure Pipelines Variable Groups to YAML

If you've been using Azure Pipelines for a while, you might have made use of the Variable group feature (under the Library tab). These provide a way of defining a group of variables that can be referenced both by 'Classic' Release pipelines and the newer YAML-based pipelines.

You might get to a point where you realise that a lot of the variables being defined in variable groups would be better off being declared directly in a YAML file. That way you get the benefit of version control history. Obviously, you wouldn't commit any secrets in your source code, but other non-sensitive values should be fine. Secrets are better left in a variable group, or better yet as an Azure Key Vault secret.

I came up with the following PowerShell script that makes use of the Azure CLI commands to convert a variable group into the equivalent YAML syntax.

param (
    [Parameter(Mandatory=$true)]
    [string]
    $GroupName,
    [string]
    $Organisation,
    [string]
    $Project,
    [switch]
    $Array
)

$ErrorActionPreference = 'Stop'

# Find id of group
$groups = (az pipelines variable-group list --organization "https://dev.azure.com/$Organisation" --project "$Project") | ConvertFrom-Json

$groupId = $groups | Where-Object { $_.name -eq $GroupName } | Select-Object -ExpandProperty id -First 1

$group = (az pipelines variable-group show --id $groupId --organization "https://dev.azure.com/$Organisation" --project "$Project") | ConvertFrom-Json

$group.variables | Get-Member -MemberType NoteProperty | ForEach-Object {

    if ($Array.IsPresent) {
        Write-Output "- name: $($_.Name)"
        Write-Output "  value: $($group.variables.$($_.Name).Value)"
    } else {
        Write-Output "  $($_.Name): $($group.variables.$($_.Name).Value)"
    }
}

The script is also in this GitHub Gist.

If I run the script like this:

.\Convert-VariableGroup.ps1 -Organisation gardiner -GroupName "My Variable Group" -Project "GitHub Builds"

Then it produces:

And_More: $(Another.Thing)
Another.Thing: Another value
ASecret:
Something: A value

If I add the -Array switch, then the output changes to:

- name: And_More
  value: $(Another.Thing)
- name: Another.Thing
  value: Another value
- name: ASecret
  value:
- name: Something
  value: A value

Note that the variable which was marked as 'secret' doesn't have a value.

STEPtember 2022

For the second year in a row, along with some of my SixPivot colleagues, I'm taking part in STEPtember. STEPtember raises funds for people with and research into Cerebral Palsy. I'll be aiming to walk at least 10,000 steps each day in September.

To sponsor me, head over to https://www.steptember.org.au/s/157783/383539. In Australia, donations of $2 or more are tax deductible.

Trigger an Azure Pipeline build from a 'Classic' release pipeline

Azure Pipelines YAML pipelines can have pipeline resource triggers (for triggering a pipeline when another YAML pipeline completes), and 'Classic' build pipelines have build completion triggers to do the same. 'Classic' Release pipelines have release triggers, which trigger every time a new build artifact is available.

But what if you've got a mix of 'classic' release and YAML pipelines? Can you trigger a YAML pipeline from a 'classic' release pipeline? There's nothing built in to do this, but we can make it happen via the Azure DevOps APIs.

1. In the Azure DevOps UI, under Pipelines, navigate to the YAML pipeline that you want to trigger.
2. Make note of the definitionId in the URL. You'll need this for the PIPELINE_ID value later.
3. Click on the '…' button in the top right and select Manage Security

4. Select the <Project Name> Build Service (<Org Name>) user, and ensure Queue builds option is set to Allow.

   Note: You can also set this for all pipelines, rather than for an individual pipeline.

   

5. In the release pipeline, select the Agent Job, and ensure the Allow scripts to access the OAuth token option is checked (this will allow access to System.AccessToken)

   

6. Add a script task (I'm using the Bash Task, but you could equally use PowerShell though you'd need to adjust the script slightly)

7. Add the following inline script:

    PIPELINE_ID="15"
   
    url="$(SYSTEM.TEAMFOUNDATIONCOLLECTIONURI)$(SYSTEM.TEAMPROJECTID)/_apis/pipelines/$PIPELINE_ID/runs?api-version=6.0-preview.1"
   
    echo $url
   
    curl -s --request POST \
    -u ":$(System.AccessToken)" \
    --header "Content-Type: application/json" \
    --data '{
      "resources": {
          "repositories": {
              "self": {
                  "refName": "refs/heads/main"
              }
          }
      }
    }' \
    $url
   

8. If you now create a new release, once that completes you should see the YAML pipeline build queued to run. Notice that the summary indicates the build was triggered manually by the service account.

   

Things to note:

• Update the value assigned to PIPELINE_ID with the value you noted from the definitionId earlier.
• If the repo uses a default branch other than main then you'll need to modify that too.
• The script assumes you're triggering in the same project. If you need to trigger a pipeline in a different project you'll need to replace $(SYSTEM.TEAMPROJECTID) with the project name, or more preferably the project id (GUID) for that project.
• Failing to set permission to allow queuing builds won't result in an error, but it will just mean builds aren't queued. If builds aren't being queued, double-check that you have enabled this on the correct pipeline, or it is enabled for all pipelines.

Credits: The curl invocation was inspired by the example at https://cloudaffaire.com/how-to-create-and-execute-azure-pipelines-using-rest-api/.