.NET Code Coverage in Azure DevOps and SonarCloud

Sonar offer some really useful products for analysing the quality of your application's source code. There's a great mix of free and paid products, including SonarQube Cloud (formerly known as SonarCloud), SonarQube Server (for on-prem), and SonarQube for IDE (formerly SonarLint) static code analysers for IntelliJ, Visual Studio, VS Code and Eclipse.

I was looking to integrate an Azure DevOps project containing a .NET application with SonarQube Cloud, and in particular include code coverage data both for Azure Pipelines (so you can view the coverage in the pipeline run), but also in SonarQube Cloud.

This process is quite similar if you're using the self-hosted SonarQube Server product, though note that there are different Azure Pipeline tasks provided by a different extension for SonarQube Server.

A sample project can be found at https://dev.azure.com/gardiner/SonarCloudDemo

Prerequisites

• You have a SonarQube Cloud account.
• You've configured it to be integrated with your Azure DevOps organisation.
• You've installed the SonarQube Cloud extension (or SonarQube Server extension if you're using SonarQube Server)
• You've created a service connection in the Azure DevOps project pointing to SonarQube Cloud.

I've created a .NET solution which contains a simple ASP.NET web application and an xUnit test project.

By default, when you add a new xUnit test project, it includes a reference to the coverlet.collector NuGet package. This implements a 'Data Collector' for the VSTest platform. Normally you'd run this via:

dotnet test --collect:"XPlat Code Coverage"

You would then end up with a TestResults subdirectory which contains a coverage.cobertura.xml file. But the problem here is that the xml file is one level deeper - VSTest creates GUID-named subdirectory under TestResults. So you will need to go searching for the file, there's no way to ensure it gets created in a known location.

It turns out that's a problem for Sonar, as the SonarCloudPrepare task needs to be told where the code coverage file is located, and unfortunately that property doesn't support wildcards!

We can solve that problem by removing the reference to coverlet.collector, and instead adding a package reference to coverlet.msbuild.

dotnet remove package coverlet.collector
dotnet add package coverlet.msbuild

To collect code coverage information with this package, you run it like this:

dotnet test /p:CollectCoverage=true

But more importantly, it supports additional parameters so we can now fix the location of output files. The CoverletOutput property lets us define the directory (relative to the test project) where output files will be written.

dotnet test /p:CollectCoverage=true /p:CoverletOutput='./results/coverage' /p:CoverletOutputFormat=cobertura

Notice that I've not just set CoverletOutput to the directory (results), but also the first part of the coverage filename (coverage).

In the pipeline task, you can let SonarQube know where the file is by setting sonar.cs.opencover.reportsPaths like this:

  - task: SonarCloudPrepare@3
    inputs:
      SonarQube: "SonarCloud"
      organization: "gardiner"
      scannerMode: "dotnet"
      projectKey: "Gardiner_SonarCloudDemo"
      projectName: "SonarCloudDemo"
      extraProperties: |
        sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/Tests/results/coverage.opencover.xml

SonarQube and Azure Pipelines coverage

So now we've solved the problem of where the coverage file will be saved. Can we also deliver the coverage data to both SonarQube and Azure Pipelines?

Let's review what we need to make that happen.

According to the docs for coverlet.msbuild, it supports generating the following formats:

• json (default)
• lcov
• opencover
• cobertura
• teamcity*

(The TeamCity format just generates special service messages in the standard output that TeamCity will recognise, it doesn't create a file)

According to the docs for SonarCloud, it supports the following formats for .NET code coverage:

• Visual Studio Code Coverage
• dotnet-coverage Code Coverage
• dotCover
• OpenCover
• Coverlet (OpenCover format)
• Generic test data

The docs for the Azure Pipelines PublishCodeCoverageResults@2 task don't actually mention which formats are supported (hopefully this will be fixed soon). But in the blog post that announced the availability of the v2 task the following formats were mentioned (including ones from the v1 task):

• Cobertura
• JaCoCo
• .coverage
• .covx

So unfortunately there isn't a single format that all three components understand. Instead we will have to ask coverlet.msbuild to generate two output files - OpenCover for SonarQube, and Cobertura for Azure Pipelines.

We want to generate two outputs, but there is a known problem with trying to pass in parameters to dotnet test on Linux. The workaround is to set properties in the csproj file instead.

<PropertyGroup>
  <CoverletOutputFormat>opencover,cobertura</CoverletOutputFormat>
</PropertyGroup>          

Our Azure Pipeline should look something like this:

steps:
  - checkout: self
    fetchDepth: 0
    
  - task: SonarCloudPrepare@3
    inputs:
      SonarQube: "SonarCloud"
      organization: "gardiner"
      scannerMode: "dotnet"
      projectKey: "Gardiner_SonarCloudDemo"
      projectName: "SonarCloudDemo"
      extraProperties: |
        # Additional properties that will be passed to the scanner, put one key=value per line

        # Disable Multi-Language analysis
        sonar.scanner.scanAll=false

        # Configure location of the OpenCover report
        sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/Tests/results/coverage.opencover.xml

  - task: DotNetCoreCLI@2
    inputs:
      command: build

  - task: DotNetCoreCLI@2
    inputs:
      command: test
      projects: "Tests/Tests.csproj"
      arguments: "/p:CollectCoverage=true /p:CoverletOutput=results/coverage"

  - task: SonarCloudAnalyze@3
    inputs:
      jdkversion: "JAVA_HOME_17_X64"

  - task: SonarCloudPublish@3
    inputs:
      pollingTimeoutSec: "300"

  - task: PublishCodeCoverageResults@2
    inputs:
      summaryFileLocation: "$(Build.SourcesDirectory)/Tests/results/coverage.cobertura.xml"
      failIfCoverageEmpty: true

A few things to point out:

• We're doing a full Git clone (not shallow) so that SonarQube can do a proper analysis. This avoids you seeing warnings like this:
  • [INFO] SonarQube Cloud: Analysis succeeded with warning: Could not find ref 'main' in refs/heads, refs/remotes/upstream or refs/remotes/origin. You may see unexpected issues and changes. Please make sure to fetch this ref before pull request analysis.
  • [INFO] SonarQube Cloud: Analysis succeeded with warning: Shallow clone detected during the analysis. Some files will miss SCM information. This will affect features like auto-assignment of issues. Please configure your build to disable shallow clone.
• Set sonar.scanner.scanAll=false to avoid this warning:
  • [INFO] SonarQube Cloud: Analysis succeeded with warning: Multi-Language analysis is enabled. If this was not intended and you have issues such as hitting your LOC limit or analyzing unwanted files, please set "/d:sonar.scanner.scanAll=false" in the begin step.

And now we can view our code coverage in SonarQube:

And in Azure Pipelines!

Check out the example project at https://dev.azure.com/gardiner/_git/SonarCloudDemo, and you can view the SonarQube analysis at https://sonarcloud.io/project/overview?id=Gardiner_SonarCloudDemo

DDD Brisbane 2024

Well this was an early Christmas present. I had submitted a talk for DDD Brisbane earlier this year, and it got voted in! So my work (SixPivot) kindly flew me up to Brisbane for the weekend. I'd never been to DDD Brisbane before, so I was really looking forward to being there.

It was only two weeks ago that we ran DDD Adelaide, but it's quite a different experience being an organiser compared to just being an attendee. Chris Gilbert (one of the DDD Brisbane organisers) also works at SixPivot so when I knew I'd be going I mentioned to him I'd be happy to help out on the day. So as well as speaking I was also a DDD Brisbane volunteer (complete with a bright red t-shirt!). It was a great way to support the event, as well as giving me an inside perspective on how the Brisbane team do DDD and hopefully be able to bring home some new ideas for Adelaide.

I stayed at the Vine Apartments, which is conveniently a short walk from Brisbane State High School (where DDD Brisbane is held). Just before 7am Saturday morning, Bronwen Zande (the other DDD Brisbane 2024 organiser) and John O'Brien (who was also the DDD Brisbane photographer) picked me up on their way through, and we began setting up.

A bunch of other volunteers quickly appeared and a very organised preparation began. They have obviously done this before as they knew what needed to be done, and what needed to go where.

Around 8am registrations began. It's a similar process to Adelaide, with attendees being 'checked in' via an app, and then given lanyards and really nifty bags (made by Boomerang Bags).

The keynote speaker was Joel Pobar. I remember seeing him speak at a few Microsoft TechEd conferences many years ago, so it was interesting to hear what he's been up to since those days. He now works for Anthropic, and his talk on AI was a) pretty deep, with a few maths equations that were a bit over my head and b) left me feeling a bit uneasy about the future as he shared his own concerns about where things might be heading. A personal quibble, I would have preferred he kept his language G rated. That was a distraction I didn't think was necessary.

Then I was off to act as host for the 'PAC 2' session room (they ran 4 breakout sessions in parallel for most of the day).

I heard Daniel Fang present "LEGO + Python + GPT = The Journey from Dad to Superdad with OpenAI and Robotics!"

A short break for morning tea with fresh fruit and some nice biscuits.

Then Jason Taylor with "Practical Clean Architecture with ASP.NET Core 9". I hope we can get Jason down to Adelaide to speak at Adelaide .NET User Group sometime.

Followed by Stephen Rees-Carter with "Th1nk Lik3 a H4cker". This was really interesting hands-on session with the audience invited to try and hack along with Stephen.

Lunch was served outside. Roast beef, chicken, and hot vegetables - yum.

I handed over my room hosting duties for PAC 2 to my SixPivot colleague Dylan (who was also volunteering for the day), so I was now free to check out some sessions in different rooms for the afternoon (apart from when it was time for me to present back in PAC 2).

I caught a bit of Bryden Oliver & Luke Parker presenting "Feel the Mayhem - When Deployments go Wrong". This was less about the technical details of failed deployments (which I've previously done talks on myself) and more about the "people and process" part of recovering from an incident.

I left a few minutes early (to get back to the PAC 2 room to start preparing for my presentation) and caught the end of Aaron Powell's "I've burnt out, now what".

Then it was my turn to present "10 tips and tricks for GitHub Actions and Azure DevOps".

I'd previously presented this talk at Adelaide .NET User Group a few months ago and it went over an hour. I knew I only had 45 minutes at DDD, so when I did a test run for my work colleagues earlier this week I tried to tighten it up, but over compensated by finishing it in about 25 minutes! A few more practice runs helped to find the middle ground, such that on the day I finished right about the 40 minute mark, giving 5 minutes for a few questions at the end. I also was able to give out a couple of pairs of DDD Adelaide socks.

After a spot of afternoon tea (actually I was chatting to people so much it was all gone by the time I realised!), it was off the last session of the day - Joe Patterson with "Simplicity Driven Design".

DDD Brisbane have a locknote to finish off the day. This year it was Dr Jenine Beekhuyzen OAM. I have not met her before, but I do have an interesting connection. A number of years ago Bronwen Zande was promoting the Tech Girls are Superheroes books (written by Dr Beekhuyzen) on Twitter and I was able to get a copy for my eldest daughter.

Sometime later, I'd arranged for Bronwen to speak to the Adelaide .NET User Group and while she was visiting I also arranged for her to visit the high school my kids attended and she gave a talk about her experience as a software developer (and more excitingly for the students) working with Microsoft's HoloLens.

So now it was great to hear Dr Beekhuyzen in person. Bronwen had asked me a few weeks ago if I might ask my daughter if she had any thoughts on the impact reading the book and having relevant role models might have had.

Near the end of Jenine's talk, Bronwen invited me up on stage to share her thoughts. I was able to thank both Jenine and Bronwen for their positive influence and read out the following quote from my daughter:

It was encouraging to read stories about women in tech at a young age when I was making decisions about what I wanted to do in the future

After the prize draw (which cleverly required that you had submitted feedback at least once to be eligible to win, as well as be present in the room), there were thank yous and a closing farewell, and also an intriguing hint of a possible 'DDD Outback 2025' satellite event.

Then it was time to pack down (which again didn't take too long with the volunteers all pitching in), and a final farewell to my DDD Brisbane friends.

It was such a privilege to be able to attend and I can't thank enough the organisers, volunteers and also the sponsors for all they've done to put on another successful event. I hope I get an opportunity to visit again in the future.

DDD Adelaide 2024

Last Saturday we ran DDD Adelaide 2024 - Adelaide's largest community run conference for the tech community. Around 350 software developers and folks from related disciplines gathered at the University of Adelaide for a day of great speakers, food, coffee and conversations.

This follows on from the 2023 conference, so you'd think it would be all routine now. But there were a few organisational challenges on the morning. Thankfully, most attendees would have been blissfully unaware, other than we had a last minute room change. While the new room was a little further away, we did hear back that the room itself had great atmosphere.

Again, Cargo Catering and B3 Coffee did an awesome job keeping everyone fed and watered throughout the day. Because we increased the size of the event we switched to 'lunch bowls' for lunch, and they worked really well (with no big queues of people waiting ages compared to last year).

By the end of the day, I was starting to finally relax and enjoy the last session (Tony Albrecht on human biases in programming - excellent!) and then wrap up the day with a bit of fun with Andrew Best, thanking all those who helped make the day happen, and the all important prize draw.

This coming weekend I'll be attending our sister event DDD Brisbane for the first time. I'll be giving my 10 tips and tricks for GitHub Actions and Azure DevOps talk, and also helping out on the day as a volunteer. I'm really looking forward to catching up with friends and colleagues, as well as observing how Brisbane run their event. Maybe I'll see you there?