David Gardiner - Azure DevOps

.NET Code Coverage in Azure DevOps and SonarCloud

2024-12-15T22:00:00.000+10:30

Sonar offer some really useful products for analysing the quality of your application's source code. There's a great mix of free and paid products, including SonarQube Cloud (formerly known as SonarCloud), SonarQube Server (for on-prem), and SonarQube for IDE (formerly SonarLint) static code analysers for IntelliJ, Visual Studio, VS Code and Eclipse.

I was looking to integrate an Azure DevOps project containing a .NET application with SonarQube Cloud, and in particular include code coverage data both for Azure Pipelines (so you can view the coverage in the pipeline run), but also in SonarQube Cloud.

This process is quite similar if you're using the self-hosted SonarQube Server product, though note that there are different Azure Pipeline tasks provided by a different extension for SonarQube Server.

A sample project can be found at https://dev.azure.com/gardiner/SonarCloudDemo

Prerequisites

You have a SonarQube Cloud account.
You've configured it to be integrated with your Azure DevOps organisation.
You've installed the SonarQube Cloud extension (or SonarQube Server extension if you're using SonarQube Server)
You've created a service connection in the Azure DevOps project pointing to SonarQube Cloud.

I've created a .NET solution which contains a simple ASP.NET web application and an xUnit test project.

By default, when you add a new xUnit test project, it includes a reference to the coverlet.collector NuGet package. This implements a 'Data Collector' for the VSTest platform. Normally you'd run this via:

dotnet test --collect:"XPlat Code Coverage"

You would then end up with a TestResults subdirectory which contains a coverage.cobertura.xml file. But the problem here is that the xml file is one level deeper - VSTest creates GUID-named subdirectory under TestResults. So you will need to go searching for the file, there's no way to ensure it gets created in a known location.

It turns out that's a problem for Sonar, as the SonarCloudPrepare task needs to be told where the code coverage file is located, and unfortunately that property doesn't support wildcards!

We can solve that problem by removing the reference to coverlet.collector, and instead adding a package reference to coverlet.msbuild.

dotnet remove package coverlet.collector
dotnet add package coverlet.msbuild

To collect code coverage information with this package, you run it like this:

dotnet test /p:CollectCoverage=true

But more importantly, it supports additional parameters so we can now fix the location of output files. The CoverletOutput property lets us define the directory (relative to the test project) where output files will be written.

dotnet test /p:CollectCoverage=true /p:CoverletOutput='./results/coverage' /p:CoverletOutputFormat=cobertura

Notice that I've not just set CoverletOutput to the directory (results), but also the first part of the coverage filename (coverage).

In the pipeline task, you can let SonarQube know where the file is by setting sonar.cs.opencover.reportsPaths like this:

  - task: SonarCloudPrepare@3
    inputs:
      SonarQube: "SonarCloud"
      organization: "gardiner"
      scannerMode: "dotnet"
      projectKey: "Gardiner_SonarCloudDemo"
      projectName: "SonarCloudDemo"
      extraProperties: |
        sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/Tests/results/coverage.opencover.xml

SonarQube and Azure Pipelines coverage

So now we've solved the problem of where the coverage file will be saved. Can we also deliver the coverage data to both SonarQube and Azure Pipelines?

Let's review what we need to make that happen.

According to the docs for coverlet.msbuild, it supports generating the following formats:

json (default)
lcov
opencover
cobertura
teamcity*

(The TeamCity format just generates special service messages in the standard output that TeamCity will recognise, it doesn't create a file)

According to the docs for SonarCloud, it supports the following formats for .NET code coverage:

Visual Studio Code Coverage
dotnet-coverage Code Coverage
dotCover
OpenCover
Coverlet (OpenCover format)
Generic test data

The docs for the Azure Pipelines PublishCodeCoverageResults@2 task don't actually mention which formats are supported (hopefully this will be fixed soon). But in the blog post that announced the availability of the v2 task the following formats were mentioned (including ones from the v1 task):

Cobertura
JaCoCo
.coverage
.covx

So unfortunately there isn't a single format that all three components understand. Instead we will have to ask coverlet.msbuild to generate two output files - OpenCover for SonarQube, and Cobertura for Azure Pipelines.

We want to generate two outputs, but there is a known problem with trying to pass in parameters to dotnet test on Linux. The workaround is to set properties in the csproj file instead.

<PropertyGroup>
  <CoverletOutputFormat>opencover,cobertura</CoverletOutputFormat>
</PropertyGroup>

Our Azure Pipeline should look something like this:

steps:
  - checkout: self
    fetchDepth: 0
    
  - task: SonarCloudPrepare@3
    inputs:
      SonarQube: "SonarCloud"
      organization: "gardiner"
      scannerMode: "dotnet"
      projectKey: "Gardiner_SonarCloudDemo"
      projectName: "SonarCloudDemo"
      extraProperties: |
        # Additional properties that will be passed to the scanner, put one key=value per line

        # Disable Multi-Language analysis
        sonar.scanner.scanAll=false

        # Configure location of the OpenCover report
        sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/Tests/results/coverage.opencover.xml

  - task: DotNetCoreCLI@2
    inputs:
      command: build

  - task: DotNetCoreCLI@2
    inputs:
      command: test
      projects: "Tests/Tests.csproj"
      arguments: "/p:CollectCoverage=true /p:CoverletOutput=results/coverage"

  - task: SonarCloudAnalyze@3
    inputs:
      jdkversion: "JAVA_HOME_17_X64"

  - task: SonarCloudPublish@3
    inputs:
      pollingTimeoutSec: "300"

  - task: PublishCodeCoverageResults@2
    inputs:
      summaryFileLocation: "$(Build.SourcesDirectory)/Tests/results/coverage.cobertura.xml"
      failIfCoverageEmpty: true

A few things to point out:

We're doing a full Git clone (not shallow) so that SonarQube can do a proper analysis. This avoids you seeing warnings like this:
- [INFO] SonarQube Cloud: Analysis succeeded with warning: Could not find ref 'main' in refs/heads, refs/remotes/upstream or refs/remotes/origin. You may see unexpected issues and changes. Please make sure to fetch this ref before pull request analysis.
- [INFO] SonarQube Cloud: Analysis succeeded with warning: Shallow clone detected during the analysis. Some files will miss SCM information. This will affect features like auto-assignment of issues. Please configure your build to disable shallow clone.
Set sonar.scanner.scanAll=false to avoid this warning:
- [INFO] SonarQube Cloud: Analysis succeeded with warning: Multi-Language analysis is enabled. If this was not intended and you have issues such as hitting your LOC limit or analyzing unwanted files, please set "/d:sonar.scanner.scanAll=false" in the begin step.

And now we can view our code coverage in SonarQube:

And in Azure Pipelines!

Check out the example project at https://dev.azure.com/gardiner/_git/SonarCloudDemo, and you can view the SonarQube analysis at https://sonarcloud.io/project/overview?id=Gardiner_SonarCloudDemo

Docker run from an Azure Pipeline Container Job with a volume mount

2024-05-10T17:30:00.000+09:30

This caught me out today. I was trying to run a Docker container directly from a script task, where the pipeline job was already running in a container (as a Container Job), similar to this:

  - job: MyJob
    container:
      image: my-container-job-image:latest

    steps:
      - script: |
          docker run --mount type=bind,source="$(pwd)",target=/home/src --rm -w /home/src my-container:latest

The bit that was failing was the --mount, with the following error message:

docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /__w/3/s.

Eventually, I realised the problem - By default, when a job is running as a Container Job, all the tasks are also running in the context of that container. So $(pwd) was resolving to /__w/3/s. That happens to be the default directory, and also where your source code is mapped to (via a volume mount that you can see by viewing the output of the "Initialize containers" step).

But when you invoke docker run, Docker doesn't try and run the new container inside the existing Container Job container, rather it will run alongside it! So any paths you pass to Docker need to be relative to the host machine, not relative to inside the container job.

In my case, the solution was to add a target: host property to the script task, so that the entire script is now run in the context of the host, rather than the container. eg.

      - script: |
          docker run --mount type=bind,source="$(pwd)",target=/home/src --rm -w /home/src my-container:latest
        target: host

Now when the pipeline runs, $(pwd) will resolve to /agent/_work/3/s (which is the actual directory on the host machine), and the mount will work correctly!

Azure DevOps API PropertiesCollections

2023-06-20T22:30:00.000+09:30

I was looking at some of the Azure DevOps API documentation and noticed that some of the endpoints mention a properties object of type PropertiesCollection. Unfortunately, the details for that data structure are not particularly helpful, and I couldn't figure out how to use it. Some pages include examples, but none that I could find included an expanded properties object.

To figure out how to use it, I created a simple .NET console application. I added references to the following NuGet packages:

Microsoft.TeamFoundationServer.Client
Microsoft.VisualStudio.Services.InteractiveClient
Microsoft.VisualStudio.Services.Release.Client

using Microsoft.VisualStudio.Services.Common;
using Microsoft.VisualStudio.Services.ReleaseManagement.WebApi.Clients;
using Microsoft.VisualStudio.Services.WebApi;

const string collectionUri = "https://dev.azure.com/organisation";
const string projectName = "MyProject";
const string pat = "YOUR-PAT-HERE";
const int releaseId = 20;

var creds = new VssBasicCredential(string.Empty, pat);

// Connect to Azure DevOps Services
var connection = new VssConnection(new Uri(collectionUri), creds);

using var client = connection.GetClient<ReleaseHttpClient>();

// Get data about a specific release
var release = await client.GetReleaseAsync(projectName, releaseId);

release.Properties.Add("Thing", "hey");

// Send the updated release back to Azure DevOps Services
var result = await client.UpdateReleaseAsync(release!, projectName, releaseId);

Console.WriteLine();

This allowed me to create a property key and value, that I could then examine by querying the item (in this case a 'classic' release), by calling the GET endpoint. eg.

https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/releases/{releaseId}?propertyFilters=Thing&api-version=7.0

Note that you need to specify the propertyFilters parameter. Otherwise the `properties`` object will not be included in the response.

And in doing that, we can see the JSON data structure!

    "properties": {
        "Thing": {
            "$type": "System.String",
            "$value": "hey"
        }
    }

So, to add a property, you need to add a new key/value pair to the properties object, where the key is the name of the property, and the value is an object with two properties: $type and $value. The $type property is the type of the value, and the $value property is the value itself.

The documentation clarifies the types supported:

Values of type Byte[], Int32, Double, DateType and String preserve their type, other primitives are retuned as a String. Byte[] expected as base64 encoded string.

(I think 'DateType' is a typo, and should be 'DateTime')

Now that we know the shape of the data, I can jump back to PowerShell and use that to add a new property:

$uri = "https://vsrm.dev.azure.com/$($organisation)/$($project)/_apis/release/releases/$($releaseId)?api-version=7.0&propertyFilters=Extra"

$result = Invoke-RestMethod -Uri $uri -Method Get -Headers $headers

if (-not ($result.properties.Extra)) {
    $result.properties | Add-Member -MemberType NoteProperty -Name "Extra" -Value @{
        "`$type" = "System.String"
        "`$value" = "haaaa"
    }
}
$body = $result | ConvertTo-Json -Depth 20

"Updating via PUT"

Invoke-RestMethod -Uri $uri -Method Put -Headers $headers -Body $body -ContentType "application/json"

Get a list of Azure Pipelines and YAML files

2023-02-17T12:00:00.000+10:30

I wanted to document the pipelines in a particular Azure DevOps project. Rather than manually write up the name of each pipeline and the corresponding YAML file, I figured there must be a way to query that data.

I've done something similar in the past using the Azure DevOps REST API, but this time I'm using the Azure CLI.

Make sure you have the devops extension installed (az extension add --name azure-devops if you don't have it already). The commands provided by this extension use the same REST API under the hood that we used directly last time.

I can get a list of pipelines for the current project with az pipelines list.

This command returns a list of objects corresponding to the BuildDefinitionReference data structure. While it has the pipeline name, I noticed that doesn't include any information about the YAML file. To get that you need query an individual pipeline using:

az pipelines show --name PipelineName

This produces a BuildDefinition object, which happens to include a process property. While it isn't documented in the BuildProcess data structure, if you look at the actual data you'll see not only the type property but a yamlFilename property, which is just what we want.

"process": {
    "type": 2,
    "yamlFilename": "release.yaml"
  }

Putting it all together, and taking advantage of the JMESPath query to limit which fields we get back, I can produce a comma-separated list of pipeline names and their corresponding YAML files with the following:

(az pipelines list --query "[].name" --query-order NameAsc -o tsv) | % { (az pipelines show --name $_ --query "[name, process.yamlFilename]" | ConvertFrom-Json) -join "," }

So for this project:

You get this:

Alternate,alternate.yml
ReleaseTest,release.yaml

This will be more useful in a project with many more pipelines. If the project has multiple repositories you could also include the repository name as well.

e.g.

(az pipelines list --query "[].name" --query-order NameAsc -o tsv) | % { (az pipelines show --name $_ --query "[name, process.yamlFilename, repository.name]" | ConvertFrom-Json) -join "," }

Such a project would produce something similar to this:

Custom Git,azure-pipelines.yml,Repro
Repro,azure-pipelines.yml,Repro
task-test,azure-pipelines.yml,task-test

You can include extra columns of data as needed.

Passed AZ-400

2022-02-23T19:30:00.000+10:30

I'm pleased to report that today I passed Microsoft exam AZ-400: Designing and Implementing Microsoft DevOps Solutions, which combined with AZ-201 that I took last year, now qualifies me for the Microsoft Certified: DevOps Engineer Expert certification.

View my verified achievement from Microsoft

The exam is quite broad in the content it covers:

Develop an instrumentation strategy (5-10%)
Develop a Site Reliability Engineering (SRE) strategy (5-10%)
Develop a security and compliance plan (10-15%)
Manage source control (10-15%)
Facilitate communication and collaboration (10-15%)
Define and implement continuous integration (20-25%)
Define and implement a continuous delivery and release management strategy (10-15%)

Some areas I'd been working with for quite a few years, but others were new to me. To help prepare I used a couple of resources:

Nice to get that one dusted.

Azure DevOps PowerShell Scripts - List all release definitions and dependant builds

2021-06-17T20:00:00.000+09:30

It's easy to open a specific release definition in the Azure DevOps web UI to find out what builds it references, but you can't do the opposite - open a build to see which release definitions rely on it.

Last time we got a list of Azure Pipelines Release Definitions. Let's go the next step and match up the builds that are being consumed by each release definition. With this list, we can now filter down to a build and find which releases use it.

See Personal access tokens for instructions on how to create the personal access token.

param (
    [string] $organisation,
    [string] $personalAccessToken
)

$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($personalAccessToken)"))
$headers = @{Authorization=("Basic {0}" -f $base64AuthInfo)}

$result = Invoke-RestMethod -Uri "https://dev.azure.com/$organisation/_apis/projects?api-version=6.0" -Method Get -Headers $headers

$projectNames = $result.value.name

$projectNames | ForEach-Object {
    $project = $_

    $result = Invoke-RestMethod -Uri "https://vsrm.dev.azure.com/$organisation/$project/_apis/release/definitions?api-version=6.1-preview.4&`$expand=artifacts" -Method Get -Headers $headers

    $result.value | Select-Object name, 
        @{ Name = "Url"; Expression = { $_._links.web.href }},
        @{ Name="Artifacts"; Expression= { $_.artifacts.alias }}, 
        @{ Name="artifactSourceDefinitionUrl"; Expression= { $_.artifacts.definitionReference.artifactSourceDefinitionUrl.id }}
}

This script also makes use of the Definitions - List REST API. This time we're requesting the artifacts property to be included in the results. It is this property that contains information about builds and their artifacts that are being consumed by the release definition.

Azure DevOps PowerShell Scripts - List all release definitions

2021-06-11T09:00:00.000+09:30

If you want to list all the Azure Pipelines Release Definitions for all projects in an Azure DevOps organisation, this script will return a list of their names, the date of the latest release and a link to view the definition within the web UI.

Note that Release definitions are part of the "classic" release pipelines. If you're using YAML-based deployments then those will be viewable via the Pipelines script.

See Personal access tokens for instructions on how to create the personal access token.

param (
    [string] $organisation,
    [string] $personalAccessToken
)

$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($personalAccessToken)"))
$headers = @{Authorization=("Basic {0}" -f $base64AuthInfo)}

$result = Invoke-RestMethod -Uri "https://dev.azure.com/$organisation/_apis/projects?api-version=6.0" -Method Get -Headers $headers

$projectNames = $result.value.name

$projectNames | ForEach-Object {
    $project = $_

    $result = Invoke-RestMethod -Uri "https://vsrm.dev.azure.com/$organisation/$project/_apis/release/definitions?api-version=6.1-preview.4&`$expand=lastRelease" -Method Get -Headers $headers

    $result.value | Select-Object name, @{ Name="CreatedOn"; Expression= { $_.lastRelease.createdOn }}, @{ Name = "Url"; Expression = { $_._links.web.href }}
} | Sort-Object

It makes use of the Definitions - List REST API. Note that the documentation for that API is slightly misleading in the examples. You do need to pass in $expand=lastRelease to get the lastRelease property populated.

Azure DevOps PowerShell Scripts - List all Azure Pipelines

2021-06-10T09:00:00.000+09:30

If you want to list all the Azure Pipelines for all projects in an Azure DevOps organisation, this script will return a list of their names.

See Personal access tokens for instructions on how to create the personal access token.

param (
    [string] $organisation,
    [string] $personalAccessToken
)

$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($personalAccessToken)"))
$headers = @{Authorization=("Basic {0}" -f $base64AuthInfo)}

$result = Invoke-RestMethod -Uri "https://dev.azure.com/$organisation/_apis/projects?api-version=6.0" -Method Get -Headers $headers

$projectNames = $result.value.name

$projectNames | ForEach-Object {
    $project = $_

    $result = Invoke-RestMethod -Uri "https://dev.azure.com/$organisation/$project/_apis/pipelines?api-version=6.0-preview.1" -Method Get -Headers $headers

    $result.value.name
} | Sort-Object

It makes use of the Pipelines - List REST API, so you could ask for any of the other properties instead of or in addition to name as well.

Azure DevOps PowerShell Scripts - List all Git repositories

2021-06-09T12:30:00.000+09:30

If you want to list all the Git repositories for all projects in an Azure DevOps organisation, this script will return all the remote URLs.

See Personal access tokens for instructions on how to create the personal access token.

param (
    [string] $organisation,
    [string] $personalAccessToken
)

$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($personalAccessToken)"))
$headers = @{Authorization=("Basic {0}" -f $base64AuthInfo)}

$result = Invoke-RestMethod -Uri "https://dev.azure.com/$organisation/_apis/projects?api-version=6.0" -Method Get -Headers $headers

$projectNames = $result.value.name

$projectNames | ForEach-Object {
    $project = $_

    $result = Invoke-RestMethod -Uri "https://dev.azure.com/$organisation/$project/_apis/git/repositories?api-version=6.0" -Method Get -Headers $headers

    $result.value.remoteUrl
} | Sort-Object

It makes use of the Repositories - List REST API, so you could ask for any of the other properties instead of or in addition to remoteUrl as well.

Errors from PowerShell 7 in Azure Pipelines

2020-04-30T15:00:00.000+09:30

I'm aiming to use PowerShell 7 as much as possible, including in Azure Pipelines tasks. I was getting a bit frustrated recently though where I had a task failing, but the log gave absolutely no clue as to why.

eg.

Starting: MyTask
==============================================================================
Task         : PowerShell
Description  : Run a PowerShell script on Linux, macOS, or Windows
Version      : 2.165.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
==============================================================================
Generating script.
========================== Starting Command Output ===========================
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a\_temp\be69a030-2e55-4cb5-9f80-98968e90f3b2.ps1'"
##[error]PowerShell exited with code '1'.
Finishing: MyTask

What's going on? Turns out it's because PowerShell 7 now defaults to the 'ConciseView' for errors and so the error ends up being so concise it isn't actually logged. I'm not the first to experience this. The fact that the concise output is not logged at all does look like a bug though.

The solution

Set $ErrorView to 'NormalView' as the first line of your script - then you'll see useful error details that will help with diagnosing what the problem is.

$ErrorView = 'NormalView'

Hopefully in the future, they'll fix the problem with no errors being logged, or enhance the pwsh task so you can set the ErrorView as a task property outside of the script.

Update 15th May

As Justin mentions in the comments, this problem has been fixed, so no need to prepend your scripts with $ErrorView = 'NormalView' now!