David Gardiner - Azure Pipelines

Upgrading Azure Database for PostgreSQL flexible server

2026-02-28T13:00:00.000+10:30

I was working on a project recently that made use of Azure Database for PostgreSQL flexible server. The system had been set up a while ago, and so when I was reviewing the resources in the Azure Portal, I noticed a warning banner for the PostreSQL server:

Your server version will lose standard Azure support on March 31, 2026. Upgrade now to avoid extended support charges starting April 1, 2026.

Terraform was being used for Infrastructure as Code, and it looked similar to this:

resource "azurerm_postgresql_flexible_server" "server" {
  name                              = "psql-postgresql-apps-australiaeast"
  resource_group_name               = data.azurerm_resource_group.rg.name
  location                          = data.azurerm_resource_group.rg.location
  version                           = "11"
  delegated_subnet_id               = azurerm_subnet.example.id
  private_dns_zone_id               = azurerm_private_dns_zone.example.id
  public_network_access_enabled     = false
  administrator_login               = "psqladmin"
  administrator_password_wo         = ephemeral.random_password.postgresql_password.result
  administrator_password_wo_version = 1
  zone                              = "1"

  storage_mb   = 32768
  storage_tier = "P4"

  sku_name   = "B_Standard_B1ms"
  depends_on = [azurerm_private_dns_zone_virtual_network_link.example]
}

As you can see from the code and screenshot above, the PostgreSQL version in use was 11. Doing a bit of research, I found version 11 was first released back in 2018, and the the final minor update 11.22 was released in 2023.

Azure provides standard support for PostgreSQL versions (documented at Azure Database for PostgreSQL version policy). There is also the option of paying for extended support, though in the case of v11 that only gets you to November this year, so just a few extra months.

In my case, I wanted to do a test of the upgrade process first, so I restored a backup of the existing server to a new resource. This essentially creates an exact copy of the server at the same version.

While we are using Infrastructure as Code, I decided to use the Azure Portal to test the upgrade, as I figured if there were any problems, they might be easier to understand, rather than try and interpret weird Terraform/AzureRM errors.

Following the upgrade documentation, I clicked on the Upgrade in the Portal.

This initiates a deployment, which depending on how much data you have and the particular SKU you're running on (eg. how fast the VM you're using is), this may take quite a while. One time it took over an hour (which was important as that may be longer than the default Terraform lifecycle, and also the pipeline job timeouts).

If that succeeds, then you should be good to try the real thing with IaC.

Upgrading with Terraform

To upgrade a major version with Terraform, you need to make a couple of changes to your azurerm_postgresql_flexible_server resource:

The version property should be updated to the desired version
The create_mode property should be set to Update (if it wasn't specified then the default is 'Default')

resource "azurerm_postgresql_flexible_server" "server" {
  name                              = "psql-postgresql-apps-australiaeast"
  resource_group_name               = data.azurerm_resource_group.rg.name
  location                          = data.azurerm_resource_group.rg.location
  version                           = "17"
  delegated_subnet_id               = azurerm_subnet.example.id
  private_dns_zone_id               = azurerm_private_dns_zone.example.id
  public_network_access_enabled     = false
  administrator_login               = "psqladmin"
  administrator_password_wo         = ephemeral.random_password.postgresql_password.result
  administrator_password_wo_version = 1
  zone                              = "1"
  create_mode                       = "Update"

  storage_mb   = 32768
  storage_tier = "P4"

  sku_name   = "B_Standard_B1ms"
  depends_on = [azurerm_private_dns_zone_virtual_network_link.example]
}

The weird thing (which I assume is a side-effect of Terraform state) is that even after you've completed the upgrade, you can't change create_mode back to Default - Terraform will throw an error if you try that. Instead you just need to leave it set to Update, but as long as the version property doesn't change then Terraform will leave it at the same version.

Adjust your timeouts

I was using Azure Pipelines, so I added a timeoutInMinutes property to the job and set it to 90 minutes. Be aware that there are different default and maximum timeouts depending on what kind of build agent you use.

Likewise the Terraform azurerm_postgresql_flexible_server resource has default timeouts. You may want to specify a timeout block to extend those values if necessary.

Gotchas

I hit some compatibility issues with the PostgreSQL instance I was attempting to upgrade. The Portal displayed the following error(s):

The major version upgrade failed precheck. Upgrading shared_preload_libraries library pg_failover_slots from source version 11 to target version 17 is not supported.;
Upgrading shared_preload_libraries library pg_failover_slots from source version 11 to target version 17 is not supported.;
Upgrading shared_preload_libraries library pg_failover_slots from source version 11 to target version 17 is not supported.;
Upgrading shared_preload_libraries library pg_failover_slots from source version 11 to target version 17 is not supported.;
Upgrading shared_preload_libraries library pg_failover_slots from source version 11 to target version 17 is not supported.;
Upgrading with password authentication mode enabled is not allowed from source version MajorVersion11. Please enable SCRAM and reset the passwords prior to retrying the upgrade.

There's two issues here:

The pg_failover_slots shared preloaded library is not supported for upgrading
Legacy MD5 passwords are deprecated in newer versions, and "SCRAM" needs to be enabled

How do we resolve these with Infrastructure as Code? In this case as we're using Terraform, we need to map/import those settings and then we can modify them. We make use of the azurerm_postgresql_flexible_server_configuration resource for this.

The value properties should initially match the existing values (eg. Make sure that Terraform thinks they are unchanged). A trick to get the existing values is to run the Terraform in 'plan' mode and take note of what values it can see from and then copy those into your code.

import {
  to = azurerm_postgresql_flexible_server_configuration.accepted_pasword_auth_method
  id = "${azurerm_resource_group.group.id}/providers/Microsoft.DBforPostgreSQL/flexibleServers/psql-postgresql-apps-australiaeast/configurations/azure.accepted_password_auth_method"
}

resource "azurerm_postgresql_flexible_server_configuration" "accepted_pasword_auth_method" {
  name      = "azure.accepted_password_auth_method"
  server_id = azurerm_postgresql_flexible_server.server.id
  value     = "md5"
}

import {
  to = azurerm_postgresql_flexible_server_configuration.password_encryption
  id = "${azurerm_resource_group.group.id}/providers/Microsoft.DBforPostgreSQL/flexibleServers/psql-postgresql-apps-australiaeast/configurations/password_encryption"
}

resource "azurerm_postgresql_flexible_server_configuration" "password_encryption" {
  name      = "password_encryption"
  server_id = azurerm_postgresql_flexible_server.server.id
  value     = "md5"
}

import {
  to = azurerm_postgresql_flexible_server_configuration.shared_preload_libraries
  id = "${azurerm_resource_group.group.id}/providers/Microsoft.DBforPostgreSQL/flexibleServers/psql-postgresql-apps-australiaeast/configurations/shared_preload_libraries"
}

resource "azurerm_postgresql_flexible_server_configuration" "shared_preload_libraries" {
  name      = "shared_preload_libraries"
  server_id = azurerm_postgresql_flexible_server.server.id
  value     = "anon,auto_explain,pg_cron,pg_failover_slots,pg_hint_plan,pg_partman_bgw,pg_prewarm,pg_stat_statements,pgaudit,pglogical,timescaledb,wal2json"
}

Once you've got those in place then you can make the changes to remove the upgrade block:

resource "azurerm_postgresql_flexible_server_configuration" "accepted_pasword_auth_method" {
  name      = "azure.accepted_password_auth_method"
  server_id = azurerm_postgresql_flexible_server.server.id
  value     = "md5,SCRAM-SHA-256"
}

resource "azurerm_postgresql_flexible_server_configuration" "password_encryption" {
  name      = "password_encryption"
  server_id = azurerm_postgresql_flexible_server.server.id
  value     = "SCRAM-SHA-256"
}

resource "azurerm_postgresql_flexible_server_configuration" "shared_preload_libraries" {
  name      = "shared_preload_libraries"
  server_id = azurerm_postgresql_flexible_server.server.id
  value     = "anon,auto_explain,pg_cron,pg_hint_plan,pg_partman_bgw,pg_prewarm,pg_stat_statements,pgaudit,pglogical,timescaledb,wal2json"
}

This will allow any existing MD5 passwords to continue to work, but any new passwords will use the more modern SCRAM-SHA-256.

For the shared_preload_libraries, we've removed the offending pg_failover_slots from the list.

Tips

Temporarily upgrade your server SKU to beefier hardware so the upgrade goes faster. If you're using IaC then make sure you use that to make the change.
Note that if you change the separate storage performance tier (IOPS), you will need to wait 12 hours before downgrading again.

Completion

If everything goes to plan, you should end up with your PostgreSQL resource upgraded to the version that you specified. Here's my resource upgraded to 17.7. v18 is actually available but I wasn't offered it due to 'regional capacity constraints', which explains why the 'Upgrade' button is now disabled.

I've published source code for a working example of Azure Database for PostgreSQL flexible server with an Azure Container app and using a VNet at https://github.com/flcdrg/terraform-azure-postgresql-containerapps

Azure Pipelines template expressions

2025-07-14T08:00:00.000+09:30

In my last post I wrote about using custom conditions in Azure Pipelines to evaluate whether to skip a step, job or stage at runtime.

Sometimes we can do better though. With template expressions we can not just skip something, we can remove it entirely. We can also use it to optionally insert values in a pipeline (something you can't do with runtime custom conditions).

The important thing to remember is that template expression are a "compile time" feature. They can only operate on things that are available at compile time. Variables set by scripts, and task output variables are two examples of things that are not available at compile time.

Compare these two Azure Pipeline runs. The first uses custom conditions to decided if the 'Publish Artifact' step is executed or not. Notice the 'Publish Artifact' step is listed, but the icon shown is a white arrow (rather than green tick)

If we use a template expression, then if it evaluates to false then the step is not even included in the job!

Template expressions use the syntax ${{ }}

You can reference parameters and variables in template expressions. The latter are only variables that are defined in the YAML file and most of the predefined variables. (That page does list which variables can be used in template expressions, but you may need to scroll the page to the right to see that column!)

You can't reference variables that are created by scripts or anything else that is only available at runtime.

You can use general functions (the same ones we used previously with runtime Custom Conditions) in template expressions, as well as two special Template expression functions.

Common patterns

You can see a complete pipeline demonstrating all the following patterns at https://github.com/flcdrg/azure-pipelines-template-expressions/blob/main/azure-pipelines.yml.

Conditionally include stage, job or step

The official documentation calls this Conditional Insertion.

Here's an example where we only want to publish build artifacts if we're building the main branch:

- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
    - publish: $(Build.ArtifactStagingDirectory)
      artifact: drop
      displayName: "Publish Artifact"

Conditionally set variable

Using template expressions to conditionally set the values of variables is a common use case.

variables:
  ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
    Environment: "Production"
  ${{ else }}:
    Environment: "Not-Production"

Note that YAML formatting rules apply - each property must be unique, so you can't repeat the expression line more than once. Instead you would just group additional variable declarations together.

Conditionally set stage or job dependency

DependsOn applies to stages and jobs. We can conditionally include the entire dependency:

${{ if ne(parameters.DependsOn, '') }}:
  dependsOn:
    - Version

Or when we have multiple dependencies, we can conditionally include an additional dependency. Because dependsOn in this case is referring to an array, we need to use the array syntax for our template expression.

dependsOn:
  - Version
  - ${{ if ne(parameters.DependsOn, '') }}:
      - ${{ parameters.DependsOn }}

Looping

The official name for this in the documentation is Iterative insertion. Loop expressions are a powerful technique that can be used to reduce duplication in your pipelines.

If you need to define an array, the only way I'm aware of doing that is by declaring a parameter, as one of the types supported by parameters is object. As an object, you can use YAML to define it as an array and add sub-properties etc to that as required.

parameters:
  - name: Environments
    type: object
    default:
      - name: Dev
        displayName: "Development"
      - name: Test
        displayName: "Testing"
      - name: Prod
        displayName: "Production"

I tend to add a displayName on those parameters just to make it clear they're just there to store the array data (and you probably shouldn't be altering the values in the Azure Pipelines web UI if you run the pipeline manually)

- ${{ each env in parameters.Environments }}:
    - stage: DeployTo${{ env.name }}
      jobs:
        - job: DeployTo${{ env.name }}
          displayName: "Deploy to ${{ env.name }}"
          steps:
            - script: echo "Deploying to ${{ env.name }} environment..."
              displayName: "Deploy to ${{ env.name }}"

Conclusion

Template expressions are a powerful feature. Curiously, despite GitHub Actions having many similarities to Azure Pipelines, this is one aspect that they didn't port over.

Often I see custom conditions being used where template expressions would be a better fit. It's worth considering if using them more could simplify and improve your pipelines.

Azure Pipeline conditionals

2025-06-30T07:00:00.000+09:30

There are two main ways to make parts of your Azure Pipeline conditional:

Add a custom condition to the step, job or stage.
Use conditional insertion with template expressions.

In this post we'll look at custom conditions.

Custom conditions

Custom conditions are evaluated at runtime.

Here's an example of a step with a custom condition which causes it to be skipped if the pipeline runs on a branch other than 'main':

- script: echo "hello world"
  condition: eq(variables['Build.SourceBranchName'], 'main')

You can use any of the Job status check functions for the condition expression, or to form part of it:

always() - evaluates to True
canceled() - evaluates to True if the pipeline was cancelled
failed() - evaluates to True if any previous dependent job failed.
failed(JOBNAME) - evaluates to True if the named job failed.
succeeded() - evaluates to True all previous dependent jobs succeeded or partially succeeded
succeeded(JOBNAME) - evaluates to True if the named job succeeded
succeededOrFailed() - evalutes to True regardless of any dependent jobs succeeding or failing
succeededOrFailed(JOBNAME) - evalates to True if the job succeeded or failed

Often you'll combine these with the not() function. For example:

not(cancelled()) - evaluates to True if no dependent jobs were cancelled. This is often the best choice where there's a chance one of the dependent jobs may have been skipped (which means it has neither succeeded or failed)
not(always()) - evaluates to False. Useful if you wish to ensure a step, job or stage is always skipped, for example as a temporary measure while you're debugging a problem with a pipeline.

You can reference predefined and custom pipeline variables in the expression. In addition to the not() function we've just seen, the other functions I most commonly use are:

eq() - Evaluates to True if the two parameters are equal (string comparisons are case-insensitive)
ne() - Evaluates to True if the two parameters are not equal
and() - Evaluates to True if all the parameters (2 or more) are True
or() - Evaluates to True if any of the parameters (2 or more) are True

There are more functions documented that may be useful in more unique scenarios.

Be aware that as soon as you add a custom condition then the evaluation of the expression will determine whether that step, job or stage is executed. This can mean it ignores any previous failures or cancellations (which may not be what you intended!)

eg. This step will always be executed when the current branch is 'main', even if previous steps have failed.

- script: echo "hello world"
  condition: eq(variables['Build.SourceBranchName'], 'main')

To preserve the more common behaviour of skipping the step if any previous steps have failed you need to use this approach:

- script: echo "hello world"
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'main'))

This also means that the condition in the next example is effectively redundant. If you see code like this then I'd recommend deleting the condition - it's just noise!

- script: echo "hello world"
  condition: succeeded()

Another common scenario is when a task creates output variables that you can then use to determine if subsequent tasks need to be run. The Terraform tasks are a good example - if the 'Plan' task does not identify any required changes, then you can safely skip the 'Apply' task.

eg.


- task: TerraformCLI@2
  displayName: "Terraform: plan"
  inputs:
    command: plan
    workingDirectory: "$(TerraformSourceDirectory)"
    commandOptions: -no-color -input=false -detailed-exitcode
    environmentServiceName: Azure MSDN - rg-tfupgrade-australiasoutheast
    publishPlanResults: Plan
    allowTelemetryCollection: false

- task: TerraformCLI@2
  displayName: "Terraform: apply"
  condition: and(succeeded(), eq(variables['TERRAFORM_PLAN_HAS_CHANGES'], 'true'))
  inputs:
    command: apply
    workingDirectory: "$(TerraformSourceDirectory)"
    commandOptions: -no-color -input=false -auto-approve
    allowTelemetryCollection: false

If you want to use a condition where the expression needs to reference an output variable from a previous job or stage, then you will need to first declare that variable in the current job or stage's variable block. You can then reference it in the condition expression.

eg. For jobs:


      - job: Job1
        steps:
          - bash: echo "##vso[task.setvariable variable=my_Job1_OutputVar;isOutput=true]Variable set in stepVar_Job1"
            name: stepVar_Job1

      - job: Job2
        dependsOn: Job1
        condition: and(succeeded(), eq( variables.varFrom_Job1, 'Variable set in stepVar_Job1'))
        variables:
          varFrom_Job1: $[ dependencies.Job1.outputs['stepVar_Job1.my_Job1_OutputVar'] ]

and for stages (note the use of stageDependencies):

  - stage: Stage1
    jobs:
      - job: Stage1Job1
        steps:
          - bash: echo "##vso[task.setvariable variable=my_Stage1Job1_OutputVar;isOutput=true]Variable set in stepVar_Stage1Job1"
            name: stepVar_Stage1Job1

  - stage: Stage3
    displayName: Stage 3
    dependsOn: Stage1
    condition: and(succeeded(), eq(variables.varFrom_Stage1DeploymentJob1, 'Variable set in stepVar_Stage1Job1'))
    variables:
      varFrom_Stage1DeploymentJob1: $[ stageDependencies.Stage1.Stage1DeploymentJob1.outputs['Stage1DeploymentJob1.stepVar_Stage1DeploymentJob1.my_Stage1DeploymentJob1_OutputVar'] ]

Take a look at the pipelines defined in my azure-pipelines-variables GitHub repository for more examples of these.

Here's an example of a pipeline run with custom conditions similar to the code excerpts above:

For activities that were skipped, when you select the specific task, job or stage, you can view the conditional expression and the actual parameters that were used in its evaluation to understand why it resulted in a False value. In the screenshot above, notice that while the succeeded() function evaluated to True, the ne() function did not, and because those two were combined with an and() then the final result was also False.

In the next post we'll look at conditional insertion with template expressions and discuss when you'd use that approach over custom conditions.

.NET Code Coverage in Azure DevOps and SonarCloud

2024-12-15T22:00:00.000+10:30

Sonar offer some really useful products for analysing the quality of your application's source code. There's a great mix of free and paid products, including SonarQube Cloud (formerly known as SonarCloud), SonarQube Server (for on-prem), and SonarQube for IDE (formerly SonarLint) static code analysers for IntelliJ, Visual Studio, VS Code and Eclipse.

I was looking to integrate an Azure DevOps project containing a .NET application with SonarQube Cloud, and in particular include code coverage data both for Azure Pipelines (so you can view the coverage in the pipeline run), but also in SonarQube Cloud.

This process is quite similar if you're using the self-hosted SonarQube Server product, though note that there are different Azure Pipeline tasks provided by a different extension for SonarQube Server.

A sample project can be found at https://dev.azure.com/gardiner/SonarCloudDemo

Prerequisites

You have a SonarQube Cloud account.
You've configured it to be integrated with your Azure DevOps organisation.
You've installed the SonarQube Cloud extension (or SonarQube Server extension if you're using SonarQube Server)
You've created a service connection in the Azure DevOps project pointing to SonarQube Cloud.

I've created a .NET solution which contains a simple ASP.NET web application and an xUnit test project.

By default, when you add a new xUnit test project, it includes a reference to the coverlet.collector NuGet package. This implements a 'Data Collector' for the VSTest platform. Normally you'd run this via:

dotnet test --collect:"XPlat Code Coverage"

You would then end up with a TestResults subdirectory which contains a coverage.cobertura.xml file. But the problem here is that the xml file is one level deeper - VSTest creates GUID-named subdirectory under TestResults. So you will need to go searching for the file, there's no way to ensure it gets created in a known location.

It turns out that's a problem for Sonar, as the SonarCloudPrepare task needs to be told where the code coverage file is located, and unfortunately that property doesn't support wildcards!

We can solve that problem by removing the reference to coverlet.collector, and instead adding a package reference to coverlet.msbuild.

dotnet remove package coverlet.collector
dotnet add package coverlet.msbuild

To collect code coverage information with this package, you run it like this:

dotnet test /p:CollectCoverage=true

But more importantly, it supports additional parameters so we can now fix the location of output files. The CoverletOutput property lets us define the directory (relative to the test project) where output files will be written.

dotnet test /p:CollectCoverage=true /p:CoverletOutput='./results/coverage' /p:CoverletOutputFormat=cobertura

Notice that I've not just set CoverletOutput to the directory (results), but also the first part of the coverage filename (coverage).

In the pipeline task, you can let SonarQube know where the file is by setting sonar.cs.opencover.reportsPaths like this:

  - task: SonarCloudPrepare@3
    inputs:
      SonarQube: "SonarCloud"
      organization: "gardiner"
      scannerMode: "dotnet"
      projectKey: "Gardiner_SonarCloudDemo"
      projectName: "SonarCloudDemo"
      extraProperties: |
        sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/Tests/results/coverage.opencover.xml

SonarQube and Azure Pipelines coverage

So now we've solved the problem of where the coverage file will be saved. Can we also deliver the coverage data to both SonarQube and Azure Pipelines?

Let's review what we need to make that happen.

According to the docs for coverlet.msbuild, it supports generating the following formats:

json (default)
lcov
opencover
cobertura
teamcity*

(The TeamCity format just generates special service messages in the standard output that TeamCity will recognise, it doesn't create a file)

According to the docs for SonarCloud, it supports the following formats for .NET code coverage:

Visual Studio Code Coverage
dotnet-coverage Code Coverage
dotCover
OpenCover
Coverlet (OpenCover format)
Generic test data

The docs for the Azure Pipelines PublishCodeCoverageResults@2 task don't actually mention which formats are supported (hopefully this will be fixed soon). But in the blog post that announced the availability of the v2 task the following formats were mentioned (including ones from the v1 task):

Cobertura
JaCoCo
.coverage
.covx

So unfortunately there isn't a single format that all three components understand. Instead we will have to ask coverlet.msbuild to generate two output files - OpenCover for SonarQube, and Cobertura for Azure Pipelines.

We want to generate two outputs, but there is a known problem with trying to pass in parameters to dotnet test on Linux. The workaround is to set properties in the csproj file instead.


  opencover,cobertura

Our Azure Pipeline should look something like this:

steps:
  - checkout: self
    fetchDepth: 0
    
  - task: SonarCloudPrepare@3
    inputs:
      SonarQube: "SonarCloud"
      organization: "gardiner"
      scannerMode: "dotnet"
      projectKey: "Gardiner_SonarCloudDemo"
      projectName: "SonarCloudDemo"
      extraProperties: |
        # Additional properties that will be passed to the scanner, put one key=value per line

        # Disable Multi-Language analysis
        sonar.scanner.scanAll=false

        # Configure location of the OpenCover report
        sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/Tests/results/coverage.opencover.xml

  - task: DotNetCoreCLI@2
    inputs:
      command: build

  - task: DotNetCoreCLI@2
    inputs:
      command: test
      projects: "Tests/Tests.csproj"
      arguments: "/p:CollectCoverage=true /p:CoverletOutput=results/coverage"

  - task: SonarCloudAnalyze@3
    inputs:
      jdkversion: "JAVA_HOME_17_X64"

  - task: SonarCloudPublish@3
    inputs:
      pollingTimeoutSec: "300"

  - task: PublishCodeCoverageResults@2
    inputs:
      summaryFileLocation: "$(Build.SourcesDirectory)/Tests/results/coverage.cobertura.xml"
      failIfCoverageEmpty: true

A few things to point out:

We're doing a full Git clone (not shallow) so that SonarQube can do a proper analysis. This avoids you seeing warnings like this:
- [INFO] SonarQube Cloud: Analysis succeeded with warning: Could not find ref 'main' in refs/heads, refs/remotes/upstream or refs/remotes/origin. You may see unexpected issues and changes. Please make sure to fetch this ref before pull request analysis.
- [INFO] SonarQube Cloud: Analysis succeeded with warning: Shallow clone detected during the analysis. Some files will miss SCM information. This will affect features like auto-assignment of issues. Please configure your build to disable shallow clone.
Set sonar.scanner.scanAll=false to avoid this warning:
- [INFO] SonarQube Cloud: Analysis succeeded with warning: Multi-Language analysis is enabled. If this was not intended and you have issues such as hitting your LOC limit or analyzing unwanted files, please set "/d:sonar.scanner.scanAll=false" in the begin step.

And now we can view our code coverage in SonarQube:

And in Azure Pipelines!

Check out the example project at https://dev.azure.com/gardiner/_git/SonarCloudDemo, and you can view the SonarQube analysis at https://sonarcloud.io/project/overview?id=Gardiner_SonarCloudDemo

Docker run from an Azure Pipeline Container Job with a volume mount

2024-05-10T17:30:00.000+09:30

This caught me out today. I was trying to run a Docker container directly from a script task, where the pipeline job was already running in a container (as a Container Job), similar to this:

  - job: MyJob
    container:
      image: my-container-job-image:latest

    steps:
      - script: |
          docker run --mount type=bind,source="$(pwd)",target=/home/src --rm -w /home/src my-container:latest

The bit that was failing was the --mount, with the following error message:

docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /__w/3/s.

Eventually, I realised the problem - By default, when a job is running as a Container Job, all the tasks are also running in the context of that container. So $(pwd) was resolving to /__w/3/s. That happens to be the default directory, and also where your source code is mapped to (via a volume mount that you can see by viewing the output of the "Initialize containers" step).

But when you invoke docker run, Docker doesn't try and run the new container inside the existing Container Job container, rather it will run alongside it! So any paths you pass to Docker need to be relative to the host machine, not relative to inside the container job.

In my case, the solution was to add a target: host property to the script task, so that the entire script is now run in the context of the host, rather than the container. eg.

      - script: |
          docker run --mount type=bind,source="$(pwd)",target=/home/src --rm -w /home/src my-container:latest
        target: host

Now when the pipeline runs, $(pwd) will resolve to /agent/_work/3/s (which is the actual directory on the host machine), and the mount will work correctly!

Migrating deprecated Terraform resources (part 2)

2023-12-26T16:30:00.000+10:30

In my previous post, I showed how to migrate deprecated Terraform resources to a supported resource type. But I hinted that there are some gotchas to be aware of. Here are some issues that I have encountered and how to work around them.

Deploying to a new environment

The import statement will fail if you try to deploy to a brand new empty environment. It makes sense when you think about it - how can you import a resource that doesn't exist yet? Unfortunately, there's no way to make the import statement conditional.

An example of this would be if your application has been in development for a while, and to assist in testing you now want to create a new UAT environment. There are no resources in the UAT environment yet, so the import statement will fail.

Initializing plugins and modules...
data.azurerm_resource_group.group: Refreshing...
data.azurerm_client_config.client: Refreshing...
data.azurerm_client_config.client: Refresh complete after 0s [id=xxxxxxxxxxxxx=]
data.azurerm_resource_group.group: Refresh complete after 0s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast]
azurerm_service_plan.plan: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast]
╷
│ Error: Cannot import non-existent remote object
│
│ While attempting to import an existing object to
│ "azurerm_service_plan.plan", the provider detected that no object exists
│ with the given id. Only pre-existing objects can be imported; check that
│ the id is correct and that it is associated with the provider's configured
│ region or endpoint, or use "terraform apply" to create a new remote object
│ for this resource.
╵
Operation failed: failed running terraform plan (exit 1)

Deploying to an environment where resources may have been deleted

This is similar to the previous scenario. It's one I've encountered where we have non-production environments where you want to "mothball" when they're not being actively used to save money. We selectively delete resources, and then when the environment is needed we re-provision them. Again, the import statement will fail if it is referencing a deleted resource.

A Solution

The workaround to support these scenarios is to not use the import statement in your HCL code, but instead use the terraform import command. Because we're calling the command from the command line, we can make it conditional. e.g.

if az appservice plan show --name plan-tfupgrade-australiasoutheast --resource-group $ARM_RESOURCE_GROUP --query id --output tsv > /dev/null 2>&1; then
  terraform import azurerm_service_plan.plan /subscriptions/$ARM_SUBSCRIPTION_ID/resourceGroups/$ARM_RESOURCE_GROUP/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast
else
  echo "Resource plan-tfupgrade-australiasoutheast does not exist in Azure"
fi

Repeat this pattern for all resources that need to be imported. Also, take care when referencing the Azure resource names. They need to be correct for the detection code to work as expected!

It's not quite as elegant as using the import statement in your HCL code, but it does the job.

Example output

Here's an example of the output from the terraform import command:

data.azurerm_resource_group.group: Reading...
data.azurerm_client_config.client: Reading...
data.azurerm_client_config.client: Read complete after 0s [id=xxxxxxxxxxxx=]
data.azurerm_resource_group.group: Read complete after 0s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast]
azurerm_service_plan.plan: Importing from ID "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast"...
azurerm_service_plan.plan: Import prepared!
  Prepared azurerm_service_plan for import
azurerm_service_plan.plan: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast]

Import successful!

And the resultant plan shows no additional changes are necessary, which is just what we like to see!

data.azurerm_resource_group.group: Reading...
data.azurerm_client_config.client: Reading...
data.azurerm_client_config.client: Read complete after 0s [id=xxxxxxxxxxxxx=]
data.azurerm_resource_group.group: Read complete after 0s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast]
azurerm_service_plan.plan: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast]
azurerm_linux_web_app.appservice: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/sites/appservice-tfupgrade-australiasoutheast]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

And if we apply this to an empty environment, it also runs successfully!

azurerm_service_plan.plan: Creating...
azurerm_service_plan.plan: Still creating... [10s elapsed]
azurerm_service_plan.plan: Creation complete after 10s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast]
azurerm_linux_web_app.appservice: Creating...
azurerm_linux_web_app.appservice: Still creating... [10s elapsed]
azurerm_linux_web_app.appservice: Still creating... [20s elapsed]
azurerm_linux_web_app.appservice: Still creating... [30s elapsed]
azurerm_linux_web_app.appservice: Still creating... [40s elapsed]
azurerm_linux_web_app.appservice: Still creating... [50s elapsed]
azurerm_linux_web_app.appservice: Creation complete after 56s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/sites/appservice-tfupgrade-australiasoutheast]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Post-migration clean up

Whether you use the import statement or the terraform import command, once you've migrated your Terraform resources in all existing environments, I recommend you remove the migration code from HCL and pipeline YAML files.

Remove the import statements as they have no further purpose, and you only risk encountering the issue mentioned above if you ever want to deploy to a new environment.

There's less risk of leaving the conditional terraform import commands in the pipeline YAML files, but as long as they remain they are slowing the pipeline down. Remove them to keep your pipelines running as fast as possible and to keep your YAML files as simple as possible. If you ever need to do another migration in the future, you'll have the commands in your source control history to refer to.

Migrating deprecated Terraform resources

2023-12-04T08:00:00.000+10:30

One of the challenges with using Terraform for your infrastructure as code is that the providers (that interact with cloud providers like Azure) are updated very frequently, and especially with major version releases this includes deprecating specific resource types. For example, when the Azure provider (AzureRM) version 3.0 was released, it deprecated many resource types and data sources. Some of these still exist in version 3, but are deprecated, will not receive any updates, and will be removed in version 4. Others have already been removed entirely.

I've created an example repo that demonstrates the migration process outlined here. Find it at https://github.com/flcdrg/terraform-azure-upgrade-resources.

While this post uses Azure and Azure Pipelines, the same principles should apply for other cloud providers and CI/CD systems.

To set the scene, here's some Terraform code that creates an Azure App Service Plan and an App Service (src). The resource types are from v2.x AzureRM provider. Bear in mind, the last release of v2 was v2.99.0 in March 2022.

# https://registry.terraform.io/providers/hashicorp/azurerm/2.99.0/docs/resources/app_service_plan
resource "azurerm_app_service_plan" "plan" {
  name                = "plan-tfupgrade-australiasoutheast"
  resource_group_name = data.azurerm_resource_group.group.name
  location            = data.azurerm_resource_group.group.location
  kind                = "Linux"
  reserved            = true
  sku {
    tier = "Basic"
    size = "B1"
  }
}

# https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service
resource "azurerm_app_service" "appservice" {
  app_service_plan_id = azurerm_app_service_plan.plan.id
  name                = "appservice-tfupgrade-australiasoutheast"
  location            = data.azurerm_resource_group.group.location
  resource_group_name = data.azurerm_resource_group.group.name
  https_only          = true

  app_settings = {
    "TEST" = "TEST"
  }

  site_config {
    always_on                 = true
    ftps_state                = "Disabled"
    http2_enabled             = true
    linux_fx_version          = "DOTNETCORE|6.0"
    min_tls_version           = "1.2"
    use_32_bit_worker_process = false
  }
  identity {
    type = "SystemAssigned"
  }
}

The documentation for AzureRM v3.x shows that these resource types are deprecated and will be completely removed in v4.x. In addition, as these resource types are not being updated, they don't support the latest features of Azure App Services, such as the new .NET 8 runtime.

So how can we switch to the azurerm_service_plan and azurerm_linux_web_app resource types?

You might think it's just a matter of just changing the resource types and updating a few properties. But if you tried that you'll discover that Terraform will try to delete the existing resources and create new ones. This is because the resource types are different, and Terraform doesn't know that they are actually the same thing because the state representation of those resources is different.

Instead, we need to let Terraform know that the Azure resources that have already been created map to the new Terraform resource types we've defined in our configuration. In addition, we want to do this in a testable way using a pull request to verify that our changes look correct before we merge them into the main branch.

The approach we'll take is to make use of the relatively new import block language feature. (In a future blog post I'll cover when you might consider using the terraform import CLI command instead).

By using the import block, we can tell Terraform that the existing resources in Azure should be mapped to the new resource types we've defined in Terraform configuration. This means that Terraform will not try to delete the existing resources and create new ones. Instead, it will update the existing resources to match the Terraform configuration.

In the following example, we're indicating that the Azure resource with the resource ID /subscriptions/.../resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast should be mapped to the azurerm_service_plan resource type. Note the use of the data block reference to insert the subscription ID, rather than hard-coding it.

import {
  id = "/subscriptions/${data.azurerm_client_config.client.subscription_id}/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast"
  to = azurerm_service_plan.plan
}

resource "azurerm_service_plan" "plan" {
  name                = "plan-tfupgrade-australiasoutheast"
  resource_group_name = data.azurerm_resource_group.group.name
  location            = data.azurerm_resource_group.group.location
  sku_name            = "B1"
  os_type             = "Linux"
}

Often when you're adding the new resource, the property names and 'shape' will change. Sometimes it's pretty easy to figure out the equivalent, but sometimes you might need some help. One option you can utilise is generating the configuration.

In this case, you add the import block, but don't add the resource block. If you then run terraform plan -generate-config-out=generated_resources.tf. Terraform will create a new file generated_resources.tf which will contain generated resources. You can then copy/paste those over into your regular .tf files. You'll almost certainly want to edit them to remove redundant settings and replace hard-coded values with variable references where applicable. If you're doing this as part of a pipeline, publish the generated file as a build artifact, so you can download it and incorporate the changes. You could make this an optional part of the pipeline that is enabled by setting a pipeline parameter to true.

There's still one problem to solve though. While we've mapped the new resource types to the existing resources, Terraform state still knows about the old resource types, and will try to delete them now that they are no longer defined in the Terraform configuration. To solve this, we can use the terraform state rm command to remove the old resources from state. If they're not in state, then Terraform doesn't know about them and won't try to delete them.

The following script will remove the old resources from state if they exist. Note that the terraform state rm command will fail if the resource doesn't exist in state, so we need to check for the existence of the resource first.

# Remove state of old resources from Terraform
mapfile -t RESOURCES < <( terraform state list )

if [[ " ${RESOURCES[@]} " =~ "azurerm_app_service_plan.plan" ]]; then
  terraform state rm azurerm_app_service_plan.plan
fi

if [[ " ${RESOURCES[@]} " =~ "azurerm_app_service.appservice" ]]; then
  terraform state rm azurerm_app_service.appservice
fi

You will need to add an entry in this script for each Terraform resource type that you are removing.

Testing

Ok, so we have a strategy for upgrading our Terraform resources. But how do we test it? We don't want to just merge the changes into the main branch and hope for the best. We want to test it first, and do this in isolation from other changes that might be happening in the main branch (or other branches). To test our changes need to update the Terraform state. But if we update the state used by everyone else then we won't be popular when their builds start failing because Terraform will be trying to recreate resources that we've just deleted from state. Except those resources still exist in Azure!

What we want is a local copy of the Terraform state that we can try out our changes in without affecting anyone else. One way to do this is to copy the remote state to a local file, then reinitialise Terraform to use the 'local' backend. Obviously we won't do a real deployment using this, but it is perfect for running terraform plan against.

Here's an example script that will copy the remote state to a local file, then reinitialise Terraform to use the local backend. It assumes that your backend configuration is defined separately in a backend.tf file. Normally this would be pointing to a remote backend (e.g. Terraform Cloud or an Azure Storage account), within the pipeline run we replace this file with configuration to use a local backend.

terraform state pull > $(Build.ArtifactStagingDirectory)/pull.tfstate

cat > backend.tf <


We now should have all the pieces in place to test our changes on PR build, and then once we're happy with the plan, merge the changes and run the migration for real. If you have multiple environments (dev/test/prod) then you can roll this out to each environment as part of the normal release process.
For Azure Pipelines, we make use of conditional expressions, so that on PR builds we test the migration using local state, but on the main branch we modify the remote state and actually apply the changes.
Here's the Azure Pipeline in full (src):
trigger: none

pr:
  branches:
    include:
      - main

pool:
  vmImage: ubuntu-latest

variables:
  - group: Terraform-Token

jobs:
  - job: build
    displayName: "Test Terraform Upgrade"

    variables:
      TerraformSourceDirectory: $(System.DefaultWorkingDirectory)/v3

    steps:
      - script: echo "##vso[task.setvariable variable=TF_TOKEN_app_terraform_io]$(TF_TOKEN)"
        displayName: "Terraform Token"

      - task: TerraformInstaller@2
        displayName: "Terraform: Installer"
        inputs:
          terraformVersion: "latest"

      - task: TerraformCLI@2
        displayName: "Terraform: init"
        inputs:
          command: init
          workingDirectory: "$(TerraformSourceDirectory)"
          backendType: selfConfigured
          commandOptions: -no-color -input=false
          allowTelemetryCollection: false

      - ${{ if ne(variables['Build.SourceBranch'], 'refs/heads/main') }}:
          # Copy state from Terraform Cloud to local, so we can modify it without affecting the remote state
          - script: |
              terraform state pull > $(Build.ArtifactStagingDirectory)/pull.tfstate

              # Write multiple lines of text to local file using bash
              cat > backend.tf <

Using it in practise
Ideally when you migrate the resource types, there will be no changes to the properties (and Terraform will report that no changes need to be made). Often the new resource type provides additional properties that you can take advantage of. Whether you set this initially or in a subsequent PR is up to you.
Here's an example output from terraform plan:
Terraform v1.6.4
on linux_amd64
Initializing plugins and modules...
data.azurerm_resource_group.group: Refreshing...
data.azurerm_client_config.client: Refreshing...
data.azurerm_client_config.client: Refresh complete after 0s [id=xxxxxxxxxxx=]
data.azurerm_resource_group.group: Refresh complete after 0s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast]
azurerm_service_plan.plan: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast]
azurerm_linux_web_app.appservice: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/sites/appservice-tfupgrade-australiasoutheast]

Terraform will perform the following actions:

  # azurerm_linux_web_app.appservice will be imported
    resource "azurerm_linux_web_app" "appservice" {
        app_settings                                   = {
            "TEST" = "TEST"
        }
        client_affinity_enabled                        = false
        client_certificate_enabled                     = false
        client_certificate_mode                        = "Required"
        custom_domain_verification_id                  = (sensitive value)
        default_hostname                               = "appservice-tfupgrade-australiasoutheast.azurewebsites.net"
        enabled                                        = true
        ftp_publish_basic_authentication_enabled       = true
        https_only                                     = true
        id                                             = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/sites/appservice-tfupgrade-australiasoutheast"
        key_vault_reference_identity_id                = "SystemAssigned"
        kind                                           = "app,linux"
        location                                       = "australiasoutheast"
        name                                           = "appservice-tfupgrade-australiasoutheast"
        outbound_ip_address_list                       = [
            "52.189.223.107",
            "13.77.42.25",
            "13.77.46.217",
            "52.189.221.141",
            "13.77.50.99",
        ]
        outbound_ip_addresses                          = "52.189.223.107,13.77.42.25,13.77.46.217,52.189.221.141,13.77.50.99"
        possible_outbound_ip_address_list              = [
            "52.189.223.107",
            "13.77.42.25",
            "13.77.46.217",
            "52.189.221.141",
            "52.243.85.201",
            "52.243.85.94",
            "52.189.234.152",
            "13.77.56.61",
            "52.189.214.112",
            "20.11.210.198",
            "20.211.233.197",
            "20.211.238.191",
            "20.11.210.187",
            "20.11.211.1",
            "20.11.211.80",
            "4.198.70.38",
            "20.92.41.250",
            "4.198.68.27",
            "4.198.68.42",
            "20.92.47.59",
            "20.92.42.78",
            "13.77.50.99",
        ]
        possible_outbound_ip_addresses                 = "52.189.223.107,13.77.42.25,13.77.46.217,52.189.221.141,52.243.85.201,52.243.85.94,52.189.234.152,13.77.56.61,52.189.214.112,20.11.210.198,20.211.233.197,20.211.238.191,20.11.210.187,20.11.211.1,20.11.211.80,4.198.70.38,20.92.41.250,4.198.68.27,4.198.68.42,20.92.47.59,20.92.42.78,13.77.50.99"
        public_network_access_enabled                  = true
        resource_group_name                            = "rg-tfupgrade-australiasoutheast"
        service_plan_id                                = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast"
        site_credential                                = (sensitive value)
        tags                                           = {}
        webdeploy_publish_basic_authentication_enabled = true

        identity {
            identity_ids = []
            principal_id = "a71e1fd5-e61b-4591-a439-98bad90cc837"
            tenant_id    = "59b0934d-4f35-4bff-a2b7-a451fe5f8bd6"
            type         = "SystemAssigned"
        }

        site_config {
            always_on                               = true
            auto_heal_enabled                       = false
            container_registry_use_managed_identity = false
            default_documents                       = []
            detailed_error_logging_enabled          = false
            ftps_state                              = "Disabled"
            health_check_eviction_time_in_min       = 0
            http2_enabled                           = true
            linux_fx_version                        = "DOTNETCORE|6.0"
            load_balancing_mode                     = "LeastRequests"
            local_mysql_enabled                     = false
            managed_pipeline_mode                   = "Integrated"
            minimum_tls_version                     = "1.2"
            remote_debugging_enabled                = false
            remote_debugging_version                = "VS2019"
            scm_minimum_tls_version                 = "1.2"
            scm_type                                = "VSTSRM"
            scm_use_main_ip_restriction             = false
            use_32_bit_worker                       = false
            vnet_route_all_enabled                  = false
            websockets_enabled                      = false
            worker_count                            = 1

            application_stack {
                docker_registry_password = (sensitive value)
                dotnet_version           = "6.0"
            }
        }
    }

  # azurerm_service_plan.plan will be imported
    resource "azurerm_service_plan" "plan" {
        id                           = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-tfupgrade-australiasoutheast/providers/Microsoft.Web/serverfarms/plan-tfupgrade-australiasoutheast"
        kind                         = "linux"
        location                     = "australiasoutheast"
        maximum_elastic_worker_count = 1
        name                         = "plan-tfupgrade-australiasoutheast"
        os_type                      = "Linux"
        per_site_scaling_enabled     = false
        reserved                     = true
        resource_group_name          = "rg-tfupgrade-australiasoutheast"
        sku_name                     = "B1"
        tags                         = {}
        worker_count                 = 1
        zone_balancing_enabled       = false
    }

Plan: 2 to import, 0 to add, 0 to change, 0 to destroy.

In the next post, I'll cover a few things to watch out for, and some post-migration clean up steps.



Terraform command 'init' failed with exit code '1'
2023-11-27T09:00:00.000+10:30
I'm setting up a new GitHub repo to demonstrate using Terraform with Azure Pipelines via the CLI and I hit a weird error right at the start. I was using Jason Johnson's Azure Pipelines Terraform Tasks extension like this:
- task: TerraformCLI@2
  inputs:
    command: init
    workingDirectory: "$(System.DefaultWorkingDirectory)/v2"
    backendType: selfConfigured
    commandOptions: -no-color -input=false
    allowTelemetryCollection: false

and it kept failing with the error:
/opt/hostedtoolcache/terraform/1.6.4/x64/terraform version
Terraform v1.6.4
on linux_amd64
+ provider registry.terraform.io/hashicorp/azurerm v2.99.0
+ provider registry.terraform.io/hashicorp/random v3.5.1
/opt/hostedtoolcache/terraform/1.6.4/x64/terraform init --input=false --no-color
Usage: terraform [global options] init [options]
...
Terraform command 'init' failed with exit code '1'.

I tried all sorts of things. It looked identical to other pipelines I had working (and ones I'd seen online). Was there a weird invisible character being passed on the command line? Out of desperation I copied the commandOptions line from another pipeline (which looked identical except that the arguments were in a different order).
But when I looked at the diff, not only were the arguments different, I realised that the dashes were different too! Terraform CLI arguments use a single dash, not a double dash. So the correct line is
commandOptions: -no-color -input=false

In hindsight, I realised that this is the first pipeline I've written on GitHub using the Terraform CLI (e.g. not in a work context) and so I did it manually, rather than copy/pasting from an existing (working) pipeline. A pity Terraform doesn't support double dashes, but there you go.



Azure DevOps API PropertiesCollections
2023-06-20T22:30:00.000+09:30
I was looking at some of the Azure DevOps API documentation and noticed that some of the endpoints mention a properties object of type PropertiesCollection. Unfortunately, the details for that data structure are not particularly helpful, and I couldn't figure out how to use it. Some pages include examples, but none that I could find included an expanded properties object.
To figure out how to use it, I created a simple .NET console application. I added references to the following NuGet packages:

Microsoft.TeamFoundationServer.Client
Microsoft.VisualStudio.Services.InteractiveClient
Microsoft.VisualStudio.Services.Release.Client

using Microsoft.VisualStudio.Services.Common;
using Microsoft.VisualStudio.Services.ReleaseManagement.WebApi.Clients;
using Microsoft.VisualStudio.Services.WebApi;

const string collectionUri = "https://dev.azure.com/organisation";
const string projectName = "MyProject";
const string pat = "YOUR-PAT-HERE";
const int releaseId = 20;

var creds = new VssBasicCredential(string.Empty, pat);

// Connect to Azure DevOps Services
var connection = new VssConnection(new Uri(collectionUri), creds);

using var client = connection.GetClient();

// Get data about a specific release
var release = await client.GetReleaseAsync(projectName, releaseId);

release.Properties.Add("Thing", "hey");

// Send the updated release back to Azure DevOps Services
var result = await client.UpdateReleaseAsync(release!, projectName, releaseId);

Console.WriteLine();

This allowed me to create a property key and value, that I could then examine by querying the item (in this case a 'classic' release), by calling the GET endpoint. eg.
https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/releases/{releaseId}?propertyFilters=Thing&api-version=7.0

Note that you need to specify the propertyFilters parameter. Otherwise the `properties`` object will not be included in the response.
And in doing that, we can see the JSON data structure!
    "properties": {
        "Thing": {
            "$type": "System.String",
            "$value": "hey"
        }
    }

So, to add a property, you need to add a new key/value pair to the properties object, where the key is the name of the property, and the value is an object with two properties: $type and $value. The $type property is the type of the value, and the $value property is the value itself.
The documentation clarifies the types supported:

Values of type Byte[], Int32, Double, DateType and String preserve their type, other primitives are retuned as a String. Byte[] expected as base64 encoded string.

(I think 'DateType' is a typo, and should be 'DateTime')
Now that we know the shape of the data, I can jump back to PowerShell and use that to add a new property:
$uri = "https://vsrm.dev.azure.com/$($organisation)/$($project)/_apis/release/releases/$($releaseId)?api-version=7.0&propertyFilters=Extra"

$result = Invoke-RestMethod -Uri $uri -Method Get -Headers $headers

if (-not ($result.properties.Extra)) {
    $result.properties | Add-Member -MemberType NoteProperty -Name "Extra" -Value @{
        "`$type" = "System.String"
        "`$value" = "haaaa"
    }
}
$body = $result | ConvertTo-Json -Depth 20

"Updating via PUT"

Invoke-RestMethod -Uri $uri -Method Put -Headers $headers -Body $body -ContentType "application/json"




Get a list of Azure Pipelines and YAML files
2023-02-17T12:00:00.000+10:30
I wanted to document the pipelines in a particular Azure DevOps project. Rather than manually write up the name of each pipeline and the corresponding YAML file, I figured there must be a way to query that data.
I've done something similar in the past using the Azure DevOps REST API, but this time I'm using the Azure CLI.
Make sure you have the devops extension installed (az extension add --name azure-devops if you don't have it already). The commands provided by this extension use the same REST API under the hood that we used directly last time.
I can get a list of pipelines for the current project with az pipelines list.
This command returns a list of objects corresponding to the BuildDefinitionReference data structure. While it has the pipeline name, I noticed that doesn't include any information about the YAML file. To get that you need query an individual pipeline using:
az pipelines show --name PipelineName

This produces a BuildDefinition object, which happens to include a process property. While it isn't documented in the BuildProcess data structure, if you look at the actual data you'll see not only the type property but a yamlFilename property, which is just what we want.
"process": {
    "type": 2,
    "yamlFilename": "release.yaml"
  }

Putting it all together, and taking advantage of the JMESPath query to limit which fields we get back, I can produce a comma-separated list of pipeline names and their corresponding YAML files with the following:
(az pipelines list --query "[].name" --query-order NameAsc -o tsv) | % { (az pipelines show --name $_ --query "[name, process.yamlFilename]" | ConvertFrom-Json) -join "," }

So for this project:

You get this:
Alternate,alternate.yml
ReleaseTest,release.yaml

This will be more useful in a project with many more pipelines. If the project has multiple repositories you could also include the repository name as well.
e.g.
(az pipelines list --query "[].name" --query-order NameAsc -o tsv) | % { (az pipelines show --name $_ --query "[name, process.yamlFilename, repository.name]" | ConvertFrom-Json) -join "," }

Such a project would produce something similar to this:
Custom Git,azure-pipelines.yml,Repro
Repro,azure-pipelines.yml,Repro
task-test,azure-pipelines.yml,task-test

You can include extra columns of data as needed.